20% time is a beguiling idea. Productivity is a function of passion; and nothing fuels passion like ownership. The problem with 20% time stems from the blurring of organisational focus and the diffusion of collective action that results from it.
So ... the problem remains: How to harness the passion of self-ownership, and steer it so that it is directed in line with the driving force and focus of the entire company ... so the group retains it's focus, and the individual retains his (or her) sense of ownership.
I can well imagine that the solution to this strategic challenge is going to be idiosyncratic to the business or industry in question ... but for many types of numerical software development, there is a real need for good tools and automation, so why not make a grand bargain: give employees some (limited) freedom to choose what they work on, but mandate that it should be in support of (well advertised) organisational strategic objectives.
Friday, 22 November 2013
Friday, 15 November 2013
Continuous Collaboration: Using Git to meld continuous integration with collaborative editing.
Written in response to:
http://www.grahamlea.com/2013/11/git-mercurial-anti-agile-continuous-integration/
I strongly agree with the fundamental point that the author is making.
However, there are nuances. A lot of this depends on the type of development that you are doing.
For example, most of my day-to-day work is done in very small increments. Minor bug-fixes, incremental classifier performance improvements, parameter changes, and so on. Only rarely will I work on a feature that is so significant in its' impact that the work-in-progress causes the branch to spend several days in a broken / non-working state. I also work in fairly small teams, so the rate of pushes to Gerrit is quite low: only around a dozen pushes per day or so. This means that integration is pretty easy, and that our CI server gives us value & helps with our quality gating. We can follow a single-branch development path with little to no pain, and because both our software and the division of labour in the team are fairly well organised, conflicts very very seldom occur when merging (even when using suboptimal tools to perform the merges).
This state of affairs probably does not hold for all developers, but it holds for me, and for most of the people that I work with. As a result, we can happily work without feature branches (most of the time), and lean on the CI process to keep ourselves in sync & to measure the performance of our classifiers & other algorithms.
Now, don't get me wrong, I think that Git is great. I am the nominated Git expert in my team, and spend a lot of time helping other team members navigate the nuances of using Git with Gerrit, but for most people it is yet another tool to learn in an already over-complex development environment. Git gives us the flexibility to do what we need to in the environment that we have; but it is anything but effortless and transparent, which is what it really needs to be.
Software development is about developing software. Making systems that work. Not wrangling branches in Git.
My ideal tool would be the bastard son of Git and a real-time collaborative editor. My unit tests should be able to report when my local working copy is in a good state. Likewise, my unit tests should be able to report whether a merge or rebase has succeeded or failed. Why can I not then fully automate the process of integrating my work with that of my colleagues? Indeed, my work should be integrated & shared whenever the following two conditions are met: 1) My unit tests pass on my local working copy, and 2) My unit tests pass on the fully integrated copy. These are the same criteria that I would use when doing the process manually ... so why do it manually? Why not automate it? Triggered by every save, the resulting process would create the appearance of an almost-real-time collaborative working environment, opening up the possibility for new forms of close collaboration and team-working that are simply not possible with current tools. A source file would be a shared document that updates almost in real time. (If it is only comments that are being edited, then there is no reason why the updating could not actually be in real time). This means that you could discuss a change with a colleague, IRC-style, in the comments of a source document, and make the change in the source file *at the same time*, keeping a record not only of the logic change, but also of the reasoning that led to it. (OK, this might cause too much noise, but with comment-folding, that might not matter too much).
Having said all of that, branches are still useful, as are commit messages, so we would still want something like Git to keep a record of significant changes, and to isolate incompatible works-in-progress in separate branches; but there is no reason why we cannot separate out the "integration" use case and the "collaboration" use case from the "version control" and "record keeping" use cases.
http://www.grahamlea.com/2013/11/git-mercurial-anti-agile-continuous-integration/
I strongly agree with the fundamental point that the author is making.
However, there are nuances. A lot of this depends on the type of development that you are doing.
For example, most of my day-to-day work is done in very small increments. Minor bug-fixes, incremental classifier performance improvements, parameter changes, and so on. Only rarely will I work on a feature that is so significant in its' impact that the work-in-progress causes the branch to spend several days in a broken / non-working state. I also work in fairly small teams, so the rate of pushes to Gerrit is quite low: only around a dozen pushes per day or so. This means that integration is pretty easy, and that our CI server gives us value & helps with our quality gating. We can follow a single-branch development path with little to no pain, and because both our software and the division of labour in the team are fairly well organised, conflicts very very seldom occur when merging (even when using suboptimal tools to perform the merges).
This state of affairs probably does not hold for all developers, but it holds for me, and for most of the people that I work with. As a result, we can happily work without feature branches (most of the time), and lean on the CI process to keep ourselves in sync & to measure the performance of our classifiers & other algorithms.
Now, don't get me wrong, I think that Git is great. I am the nominated Git expert in my team, and spend a lot of time helping other team members navigate the nuances of using Git with Gerrit, but for most people it is yet another tool to learn in an already over-complex development environment. Git gives us the flexibility to do what we need to in the environment that we have; but it is anything but effortless and transparent, which is what it really needs to be.
Software development is about developing software. Making systems that work. Not wrangling branches in Git.
My ideal tool would be the bastard son of Git and a real-time collaborative editor. My unit tests should be able to report when my local working copy is in a good state. Likewise, my unit tests should be able to report whether a merge or rebase has succeeded or failed. Why can I not then fully automate the process of integrating my work with that of my colleagues? Indeed, my work should be integrated & shared whenever the following two conditions are met: 1) My unit tests pass on my local working copy, and 2) My unit tests pass on the fully integrated copy. These are the same criteria that I would use when doing the process manually ... so why do it manually? Why not automate it? Triggered by every save, the resulting process would create the appearance of an almost-real-time collaborative working environment, opening up the possibility for new forms of close collaboration and team-working that are simply not possible with current tools. A source file would be a shared document that updates almost in real time. (If it is only comments that are being edited, then there is no reason why the updating could not actually be in real time). This means that you could discuss a change with a colleague, IRC-style, in the comments of a source document, and make the change in the source file *at the same time*, keeping a record not only of the logic change, but also of the reasoning that led to it. (OK, this might cause too much noise, but with comment-folding, that might not matter too much).
Having said all of that, branches are still useful, as are commit messages, so we would still want something like Git to keep a record of significant changes, and to isolate incompatible works-in-progress in separate branches; but there is no reason why we cannot separate out the "integration" use case and the "collaboration" use case from the "version control" and "record keeping" use cases.
Wednesday, 13 November 2013
Specification and Risk
When using a general purpose library for a specific application, we generally only use a tiny subset of the functionality that the library provides. As a result, it is often reasonable to wrap that library in a simplified, special-purpose API, tailored to the specific needs of the application under development. This simplifies the interface, and in reducing the number of ways that it can be used, we also reduce the number of ways that it can go wrong, reducing the cognitive burden on the developer, simplifying the system, and reducing both cost and risk.
In this way, a restriction in what we expect the system to do results in a beneficial simplification; reduced costs and reduced risk.
It is possible to go too far with this, though, and there are two deleterious effects that immediately spring to mind:
Firstly, there is the obvious risk of a bug in the specification - that proposed restrictions may be inappropriate or incompatible with the primary task of the system, or with the political needs of various stakeholders.
Secondly, and more insidiously, there is the risk that excessive restrictions become positive specification items; moving from "the system does not have to handle X" to "the system must check for X and explicitly error in this way". Whilst this seems ridiculous, it is a surprisingly easy psychological trap to fall into, and, of course, it increases system complexity and cost rather than reducing it.
Consistently being able to strike the right balance is primarily a function of development organisation culture; and another reason why businesses need to pay attention to this critical (but frequently overlooked) aspect of their internal organisation.
In this way, a restriction in what we expect the system to do results in a beneficial simplification; reduced costs and reduced risk.
It is possible to go too far with this, though, and there are two deleterious effects that immediately spring to mind:
Firstly, there is the obvious risk of a bug in the specification - that proposed restrictions may be inappropriate or incompatible with the primary task of the system, or with the political needs of various stakeholders.
Secondly, and more insidiously, there is the risk that excessive restrictions become positive specification items; moving from "the system does not have to handle X" to "the system must check for X and explicitly error in this way". Whilst this seems ridiculous, it is a surprisingly easy psychological trap to fall into, and, of course, it increases system complexity and cost rather than reducing it.
Consistently being able to strike the right balance is primarily a function of development organisation culture; and another reason why businesses need to pay attention to this critical (but frequently overlooked) aspect of their internal organisation.
Tuesday, 12 November 2013
Qualitative changes in privacy induced by quantitative changes in social-communications network topology
How do we form opinions and make judgements of other people?
Can we understand this process in terms of the flow of information through a network/graph?
Can we use this model to understand the impact of changes in the network? (Both structural/topological changes in connectivity and quantitative changes to flow along existing graph edges).
Can we use this approach to predict what will happen as our privacy is increasingly eroded by technology?
I.e.: Do we see some relationship between the structure of a social/communication graph and the notion of "privacy". (Temporal aspects might also be important?)
If the graph changes, does the quality of "privacy" change in different ways, and how does that impact the way that we make judgements about other people, and the way that other people make judgements about us.
What does that mean for the nature of society going forwards; particularly as developments in personal digital technologies mean that increasing amounts of highly intimate, personal information are captured, stored and disseminated in an increasingly uncontrolled, (albeit sparsely distributed) manner.
The sparsity of the distribution might be important - in terms of the creation of novel/disruptive power/influence networks.
Wednesday, 6 November 2013
The software conspiracy: Maintaining a software-developer biased imbalance in human-machine labour arbitrage.
Have you ever wondered why software engineering tools are so terrible?
Perhaps there is an implicit/unspoken conspiracy across our profession?
After all, we software developers are working away at our jobs to automate various economic activities; the inevitable result of which is to force workers in other industries out of their jobs and their livelihoods.
A claim can be made that technological developments create new, less rote and routine roles, with more intellectual challenge and greater responsibility -- but there is no real evidence that this outcome will necessarily always hold; indeed, there is some empirical evidence to suggest that this pattern is even now beginning to fail.
We are not stupid. Indeed, we are well aware of the effects of our actions on their individual welfare and security, so why should we bring the same calamity upon ourselves? Perhaps we should keep our software tools in their present primitive state; to ensure job security for ourselves just as we undermine it for others?
Perhaps there is an implicit/unspoken conspiracy across our profession?
After all, we software developers are working away at our jobs to automate various economic activities; the inevitable result of which is to force workers in other industries out of their jobs and their livelihoods.
A claim can be made that technological developments create new, less rote and routine roles, with more intellectual challenge and greater responsibility -- but there is no real evidence that this outcome will necessarily always hold; indeed, there is some empirical evidence to suggest that this pattern is even now beginning to fail.
We are not stupid. Indeed, we are well aware of the effects of our actions on their individual welfare and security, so why should we bring the same calamity upon ourselves? Perhaps we should keep our software tools in their present primitive state; to ensure job security for ourselves just as we undermine it for others?
Friday, 25 October 2013
The insanity!
People often choose to use C++ because of it's performance, willingly sacrificing programmer productivity to the alter of language; tool-chain and application complexity to do so.
Then they inexplicably decide to use strings everywhere; with bazillions of dynamic allocations and memory copies, recklessly discarding any shred of hope that they might have had for, y'know, performance.
If you want to do a ton of unnecessary string manipulations & memcpys your application is going to be slower than it could be, regardless of the language it is written in. If you want performance, the programming language that you use is literally the least significant choice that you can make. What you chose to do is far more important than the language that you use to express those operations.
Also, the word "fast" is totally and utterly meaningless; to the point of being dangerous. What a database developer means by "blazingly fast" is totally different from what an embedded engineer, or a VHDL developer means by the same term.
Then they inexplicably decide to use strings everywhere; with bazillions of dynamic allocations and memory copies, recklessly discarding any shred of hope that they might have had for, y'know, performance.
If you want to do a ton of unnecessary string manipulations & memcpys your application is going to be slower than it could be, regardless of the language it is written in. If you want performance, the programming language that you use is literally the least significant choice that you can make. What you chose to do is far more important than the language that you use to express those operations.
Also, the word "fast" is totally and utterly meaningless; to the point of being dangerous. What a database developer means by "blazingly fast" is totally different from what an embedded engineer, or a VHDL developer means by the same term.
Biological opposition
http://www.theatlantic.com/politics/archive/2013/10/can-your-genes-predict-whether-youll-be-a-conservative-or-a-liberal/280677/
Is political orientation influenced by personality traits? Seems plausible.
Are personality traits influenced by genetic factors? Seems plausible.
Can we conclude that political affiliation is influenced by genetic factors? I suppose, although the causal link the degree of influence could be very weak.
Should this influence how political campaigns operate? Possibly, although the implications might be a little scary if people took the idea too seriously.
Does this mean that it is intrinsically very difficult (perhaps impossible), to see issues from alternative points of view? Possibly, and this is (I feel) an interesting question to contemplate.
How can we effectively communicate with other people if we face biological barriers when trying to empathise with them?
Does this mean that empathy is sometimes impossible, or merely that it can sometimes be a stressful, energy-intensive mental exercise?
Does this raise other issues?
What do you think?
Is political orientation influenced by personality traits? Seems plausible.
Are personality traits influenced by genetic factors? Seems plausible.
Can we conclude that political affiliation is influenced by genetic factors? I suppose, although the causal link the degree of influence could be very weak.
Should this influence how political campaigns operate? Possibly, although the implications might be a little scary if people took the idea too seriously.
Does this mean that it is intrinsically very difficult (perhaps impossible), to see issues from alternative points of view? Possibly, and this is (I feel) an interesting question to contemplate.
How can we effectively communicate with other people if we face biological barriers when trying to empathise with them?
Does this mean that empathy is sometimes impossible, or merely that it can sometimes be a stressful, energy-intensive mental exercise?
Does this raise other issues?
What do you think?
Thursday, 24 October 2013
Intellectual diversification
Do tailored search force us into an individual intellectual rut? Is this a threat?
To really understand the world through the eyes of others - it is not sufficient to randomise the input that we receive.
The existence of a rut is not necessarily a problem, although an inability or an unwillingness to break out of it may be.
Perhaps, as Google builds an increased understanding of us, it might be willing to share that understanding more widely, so that we in turn may start to understand others; both as individuals and as social groups. (For those that are willing to invest the time).
I also hope that over time, the distribution of human culture and practice will change, with an increase in variance, but a reduction in extreme outliers; so we can enhance our diversity whilst placing an upper bound on the risk of misunderstanding and conflict.
To really understand the world through the eyes of others - it is not sufficient to randomise the input that we receive.
The existence of a rut is not necessarily a problem, although an inability or an unwillingness to break out of it may be.
Perhaps, as Google builds an increased understanding of us, it might be willing to share that understanding more widely, so that we in turn may start to understand others; both as individuals and as social groups. (For those that are willing to invest the time).
I also hope that over time, the distribution of human culture and practice will change, with an increase in variance, but a reduction in extreme outliers; so we can enhance our diversity whilst placing an upper bound on the risk of misunderstanding and conflict.
As complexity increases, phase transitions are observed
Once you get beyond a certain level of complexity; technology stops behaving like a deliberately engineered thing, and starts to take on other, less familiar characteristics.
Wednesday, 23 October 2013
Oil and water: Mixing agility and government IT.
The below is a response to this Washington Post article:
http://www.washingtonpost.com/blogs/wonkblog/wp/2013/10/21/the-way-government-does-tech-is-outdated-and-risky/
Governments have a fiduciary responsibility to look after taxpayer money. When dealing with the private sector, this is normally interpreted to mean an obligation to issue a tender, and to take the lowest (viable) bid that meets the specification. This works fairly well when the scope of the work is known beforehand, when costs can be predicted, and when a schedule of work can be drawn up.
However, as the complexity of a system increases, making any sort of prediction about that system becomes exponentially more difficult. This means that the specification for a complex system must be exponentially longer than a specification for a simple system, with an exponentially greater risk of errors. Making time & cost estimates becomes exponentially more difficult, and the error in those estimates becomes exponentially greater. The number of unknowns that must be researched grows exponentially with complexity also.
The term "Agile" is a bit of a buzzword, and has attracted more than it's fair share of snake-oil salesmen, but what it comes down to is, essentially, throwing the towel in and admitting defeat. When you *cannot* make predictions about your system, what do you do? You need to find another way of managing costs and reducing project risk.
Unfortunately, because of the fiduciary responsibilities named above, these options are not open to Government contract mechanisms. There is a fundamental conflict that cannot be resolved. As a result, government cannot (should not?) attempt to implement anything other than the very simplest of systems in the traditional, mandated manner.
How then, can complex information systems be developed for public benefit? The philanthropy of private individuals & organisations is one solution; whether through crowd-funding, or open source initiatives. Political leadership and coordination of such activities is something that could easily fall into government remit, without the significant legal hurdles that contracting work out imposes.
http://www.washingtonpost.com/blogs/wonkblog/wp/2013/10/21/the-way-government-does-tech-is-outdated-and-risky/
Governments have a fiduciary responsibility to look after taxpayer money. When dealing with the private sector, this is normally interpreted to mean an obligation to issue a tender, and to take the lowest (viable) bid that meets the specification. This works fairly well when the scope of the work is known beforehand, when costs can be predicted, and when a schedule of work can be drawn up.
However, as the complexity of a system increases, making any sort of prediction about that system becomes exponentially more difficult. This means that the specification for a complex system must be exponentially longer than a specification for a simple system, with an exponentially greater risk of errors. Making time & cost estimates becomes exponentially more difficult, and the error in those estimates becomes exponentially greater. The number of unknowns that must be researched grows exponentially with complexity also.
The term "Agile" is a bit of a buzzword, and has attracted more than it's fair share of snake-oil salesmen, but what it comes down to is, essentially, throwing the towel in and admitting defeat. When you *cannot* make predictions about your system, what do you do? You need to find another way of managing costs and reducing project risk.
Unfortunately, because of the fiduciary responsibilities named above, these options are not open to Government contract mechanisms. There is a fundamental conflict that cannot be resolved. As a result, government cannot (should not?) attempt to implement anything other than the very simplest of systems in the traditional, mandated manner.
How then, can complex information systems be developed for public benefit? The philanthropy of private individuals & organisations is one solution; whether through crowd-funding, or open source initiatives. Political leadership and coordination of such activities is something that could easily fall into government remit, without the significant legal hurdles that contracting work out imposes.
Tuesday, 22 October 2013
Learning abstractions
The more restricted and limited you make something, the easier it is to use.
Remove options, reduce flexibility and the device becomes simpler and easier. Good for the novice, bad for everybody else.
The real trick is to gradually surface functionality: provide powerful abstractions & flexible conceptual tools, but then hide them so that they only surface when they are needed, and, more importantly, when the user is able to cope with them.
So, by all means hide files, and other abstractions that serve to complicate the interaction & confuse the novice, but keep them in the background, waiting until they are needed; and provide neat ways of discovering and learning the conceptual tools required to use our devices to their full potential.
User interface design is about planning the user's learning process; guiding them on their journey from novice to expert.
Remove options, reduce flexibility and the device becomes simpler and easier. Good for the novice, bad for everybody else.
The real trick is to gradually surface functionality: provide powerful abstractions & flexible conceptual tools, but then hide them so that they only surface when they are needed, and, more importantly, when the user is able to cope with them.
So, by all means hide files, and other abstractions that serve to complicate the interaction & confuse the novice, but keep them in the background, waiting until they are needed; and provide neat ways of discovering and learning the conceptual tools required to use our devices to their full potential.
User interface design is about planning the user's learning process; guiding them on their journey from novice to expert.
Friday, 18 October 2013
A failure to interoperate; or; the power struggle between tools.
I have spent the morning cursing and swearing as I try to get CMake and Eclipse to play nicely with one another.
This is a frustration that I feel again and again. So many tools try to take control; to impose their own view of how things "should be" in the world - and those views often conflict.
Both CMake and I agree that out-of-source builds are the only sane thing to do. Why take the risk of having the build process screw up the source tree in some subtle way? This also means that we can have easily create multiple builds from the same source - and not just limited to Debug & Release builds, either. We can have builds that just generate documentation, or builds that just perform static analysis & style checking, or builds for different platforms, different targets, different levels of optimisation; all completely configuration-managed and under version control, yet totally independent of the actual content of the source tree.
Yet why do so many IDEs take a totally divergent view on how the build should be organized? Why must the project file live in the root directory of the source tree? Why must I jump through hoops to build my software outside of the source tree?
Why is it that using an IDE automatically means that I have to use a brain-dead and limited concept of project organisation?
Come back make, come back gcc, come back vi. All is forgiven.
I am all for the idea that some tools should be "opinionated" - but you do need to be prepared for pain if you want to use two different "opinionated" tools together.
For that reason, we should think carefully about developing more tools that are less all-encompassing, that are more humble, more secular, and more flexible in their approach.
This is a frustration that I feel again and again. So many tools try to take control; to impose their own view of how things "should be" in the world - and those views often conflict.
Both CMake and I agree that out-of-source builds are the only sane thing to do. Why take the risk of having the build process screw up the source tree in some subtle way? This also means that we can have easily create multiple builds from the same source - and not just limited to Debug & Release builds, either. We can have builds that just generate documentation, or builds that just perform static analysis & style checking, or builds for different platforms, different targets, different levels of optimisation; all completely configuration-managed and under version control, yet totally independent of the actual content of the source tree.
Yet why do so many IDEs take a totally divergent view on how the build should be organized? Why must the project file live in the root directory of the source tree? Why must I jump through hoops to build my software outside of the source tree?
Why is it that using an IDE automatically means that I have to use a brain-dead and limited concept of project organisation?
Come back make, come back gcc, come back vi. All is forgiven.
I am all for the idea that some tools should be "opinionated" - but you do need to be prepared for pain if you want to use two different "opinionated" tools together.
For that reason, we should think carefully about developing more tools that are less all-encompassing, that are more humble, more secular, and more flexible in their approach.
Wednesday, 16 October 2013
Software security and reliability is a public interest issue
In response to complaints that there are not enough computer-security-trained professionals in the market today:
http://www.reuters.com/article/2013/10/14/security-internet-idUSL6N0I30AW20131014
It is not just a question of skills and human resources: We also need better defensive tools and techniques, which need to be made freely available, ideally packaged with common software development tools like GCC or the LLVM compiler framework (and switched on by default)!
It would be a terrible waste if these "cyber warriors" (what a ludicrous title) all sat in isolated silos, tasked with protecting individual organizations, when the (stupendous) cost of this defensive work could be easily amortized across companies (with incredible cost savings as a result).
We need better tools to analyse complex systems, particularly software systems, for security vulnerabilities, so that those vulnerabilities can be closed. This includes static analysis tools; fuzz testing tools and vulnerability analysis tools.
We need better professional licensing and certification processes, so that we can better control the quality & reduce the vulnerability of the systems on which we all rely.
We need security-oriented programming conventions, and software registers, so that security software can do it's job more easily and more effectively.
We ALL have an interest in the reliability and trustworthiness of the complex systems that we rely on to power our infrastructure; our financial system; our workplaces. Nobody wants to gain competitive advantage because a competitor was targeted by a criminal. In a world dominated by unreliability and insecurity, it is only the criminals that win.
There is a HUGE incentive to work together here. In the public interest, let's do so.
http://www.reuters.com/article/2013/10/14/security-internet-idUSL6N0I30AW20131014
It is not just a question of skills and human resources: We also need better defensive tools and techniques, which need to be made freely available, ideally packaged with common software development tools like GCC or the LLVM compiler framework (and switched on by default)!
It would be a terrible waste if these "cyber warriors" (what a ludicrous title) all sat in isolated silos, tasked with protecting individual organizations, when the (stupendous) cost of this defensive work could be easily amortized across companies (with incredible cost savings as a result).
We need better tools to analyse complex systems, particularly software systems, for security vulnerabilities, so that those vulnerabilities can be closed. This includes static analysis tools; fuzz testing tools and vulnerability analysis tools.
We need better professional licensing and certification processes, so that we can better control the quality & reduce the vulnerability of the systems on which we all rely.
We need security-oriented programming conventions, and software registers, so that security software can do it's job more easily and more effectively.
We ALL have an interest in the reliability and trustworthiness of the complex systems that we rely on to power our infrastructure; our financial system; our workplaces. Nobody wants to gain competitive advantage because a competitor was targeted by a criminal. In a world dominated by unreliability and insecurity, it is only the criminals that win.
There is a HUGE incentive to work together here. In the public interest, let's do so.
Thursday, 3 October 2013
Hubris and the complexity trap
I have been blessed with the opportunity to work with some fantastically bright people. A surprisingly large number of them become unstuck for the same reason: Hubris.
Overconfident in their own abilities, they engineer systems of dazzling cleverness; the very complexity of which turns them into snares that confound, befuddle, and ultimately humble their creators.
The true value of a person then becomes apparent. It is not intelligence, per se, but the humility to confront and acknowledge failure.
As Brian Kernighan once said: "Debugging a system is twice as hard as creating it in the first place. Therefore, if you create a system as cleverly as possible, you are, by definition, not smart enough to debug it."
Simplicity is of paramount importance in a world that grows ever more complex, day by day.
Sometimes, you need a self-aware simple-mindedness to achieve the simplest, and therefore best, result.
Overconfident in their own abilities, they engineer systems of dazzling cleverness; the very complexity of which turns them into snares that confound, befuddle, and ultimately humble their creators.
The true value of a person then becomes apparent. It is not intelligence, per se, but the humility to confront and acknowledge failure.
As Brian Kernighan once said: "Debugging a system is twice as hard as creating it in the first place. Therefore, if you create a system as cleverly as possible, you are, by definition, not smart enough to debug it."
Simplicity is of paramount importance in a world that grows ever more complex, day by day.
Sometimes, you need a self-aware simple-mindedness to achieve the simplest, and therefore best, result.
Wednesday, 2 October 2013
All power to the developer
The typing pool has been superseded by the word processor; clerical workers by the spreadsheet and the database. What used to require the management of men, now requires the management of machines.
Two very different tasks indeed.
Man-management is a very interesting, very challenging job. Marshalling and aligning a group of humans to meet a common objective requires the ability to handle the vagaries of human relationships; interpersonal conflicts, political machinations, uncertain and conflicting motivations, forgetfulness, imperfect communication and understanding ... the challenges that must be overcome form a long long list. Indeed, one might be excused if the challenge of getting other people to do what you want them to rather masks the extent to which our objectives are woolly and ill-defined in the first place.
By contrast, as a programmer, our minions are infinitely more compliant; patient, predictable and indefatigable. The orders that we issue will be followed to the minutest detail. Indeed, the orders that we issue must be *specified* in the minutest detail.
Therein lies the rub.
The professions of programming and man-management both require the professional to make decisions and to delegate tasks. For the man-manager, the central challenge is to ensure that the tasks are completed; for the developer, the central challenge is to ensure that the tasks (and their consequences) are well enough understood in the first place.
This is surprisingly hard; especially the consequences bit, and especially in the face of complexity.
Two very different tasks indeed.
Man-management is a very interesting, very challenging job. Marshalling and aligning a group of humans to meet a common objective requires the ability to handle the vagaries of human relationships; interpersonal conflicts, political machinations, uncertain and conflicting motivations, forgetfulness, imperfect communication and understanding ... the challenges that must be overcome form a long long list. Indeed, one might be excused if the challenge of getting other people to do what you want them to rather masks the extent to which our objectives are woolly and ill-defined in the first place.
By contrast, as a programmer, our minions are infinitely more compliant; patient, predictable and indefatigable. The orders that we issue will be followed to the minutest detail. Indeed, the orders that we issue must be *specified* in the minutest detail.
Therein lies the rub.
The professions of programming and man-management both require the professional to make decisions and to delegate tasks. For the man-manager, the central challenge is to ensure that the tasks are completed; for the developer, the central challenge is to ensure that the tasks (and their consequences) are well enough understood in the first place.
This is surprisingly hard; especially the consequences bit, and especially in the face of complexity.
To what extent can our brains accommodate externally-directed actions without recognizing them as such?
The human brain is parallel and decentralized, yet we somehow manage to maintain the illusion that we are a single conscious entity, experiencing life in a sequential stream of thoughts and experiences.
This is clearly a lie that we tell ourselves. Various independent bits of brain make decisions and initiate actions entirely on their own, and somehow we manage to rationalize and confabulate and merrily deceive ourselves (after the fact) that we had some sort of explicit, sequential, conscious plan all along.
Whilst this model of human behavior is disturbing on a number of levels, it does have some interesting consequences when we consider the near-future technology of brain augmentation.
It is plausible that we could embed a bit of electronics into our brains; integrated so tightly that it is able to make decisions for us; to control our actions and influence our perceptions.
Would we feel that we were being controlled? Or would we integrate those decisions into our stream of consciousness: to confabulate a reality in which these decisions really were made by our conscious free will?
Will the perceptual pathways responsible for our self-perception bend to accommodate outside influences; to stretch the notion of self to accommodate the other? To allow us to believe that we initiated acts that were (in reality) initiated by others?
This is clearly a lie that we tell ourselves. Various independent bits of brain make decisions and initiate actions entirely on their own, and somehow we manage to rationalize and confabulate and merrily deceive ourselves (after the fact) that we had some sort of explicit, sequential, conscious plan all along.
Whilst this model of human behavior is disturbing on a number of levels, it does have some interesting consequences when we consider the near-future technology of brain augmentation.
It is plausible that we could embed a bit of electronics into our brains; integrated so tightly that it is able to make decisions for us; to control our actions and influence our perceptions.
Would we feel that we were being controlled? Or would we integrate those decisions into our stream of consciousness: to confabulate a reality in which these decisions really were made by our conscious free will?
Will the perceptual pathways responsible for our self-perception bend to accommodate outside influences; to stretch the notion of self to accommodate the other? To allow us to believe that we initiated acts that were (in reality) initiated by others?
GUI-NG Desiderata
The command line may be powerful and flexible, but it is ugly to look at and difficult to learn. WIMP style graphical user interfaces look nicer, and are easier to pick up and use, but they don't really help you do anything other than the most basic and pedestrian of tasks.
Cm'on guys, we should be able to do better than this. We should be able to design beautiful, powerful tools with a UI concept that combines the best of both worlds.
A beautiful, well designed UI for the power user: The design sense of Apple, the good looks of Lighttable and Sublime text; the power and flexibility of Unix.
The Acme editor from Plan9 shows us how radical we can be. IPython also makes a decent enough effort in something approaching the right direction, but does not go nearly far enough.
The (Unix) command-line points the way: the idea of the OS as a programming and data analysis environment above all else; the ability to perform complex tasks by streaming data from one small program to another; the use of the keyboard as the primary user input mechanism. (Mice are an ergonomic disaster zone).
From WIMP graphical user interfaces, we should prioritize menus above all else. Menus help the motivated novice to discover tools and functionality in a way that the command line simply cannot. In today's time-constrained world, RTFM is a condescending cop-out. Nobody has the time. Discoverability is everything; the requirement for arcane knowledge (and the time required to achieve it) is both obsolescent and wasteful.
Windows, Icons and Pointers are secondary considerations - The command line and menus should take priority. They could (and arguably should) still exist; but they do not have to dominate the interaction paradigm in the way that they have done in the past, and should be there to support other functionality, not to provide a framework within which other functionality must fit.
Speaking of paradigms; the "desktop metaphor" that is so baked in to the conventional user interface -- it is a redundant piece of skeuomorphism that has outlived it's usefulness. Sure, we can keep the notion of "depth", and layer things over the top of one another in the UI, but the flat, 2D relationships between items - connectedness, proximity, above/below, left/right -- these notions need to take priority and need to be given consistent semantic meanings in the "pattern language" that defines the UI that we create.
What does this all mean in practice?
Well, how about this: A primarily command-line driven interface, but with embedded widgets and graphics. I.e. emphatically NOT a tty - or anything resembling one. Something somewhat akin to tmux can act as a "window manager" - although it would have to be tmux on steroids. The edges and corners of the screen; key pieces of real estate all; should harbour hierarchical menus galore: some static, always where you expect them, some dynamic, perhaps responding to a search, perhaps responding to recently run applications.
Above all else, applications need to work together. Textual commands help us to quickly and easily glue functionality together, pipes help us to move data around the systems that we build, and the graphical capabilities of our user interface allows us to diagram and explore the data-flow of our scripts and systems.
Cm'on guys, we should be able to do better than this. We should be able to design beautiful, powerful tools with a UI concept that combines the best of both worlds.
A beautiful, well designed UI for the power user: The design sense of Apple, the good looks of Lighttable and Sublime text; the power and flexibility of Unix.
The Acme editor from Plan9 shows us how radical we can be. IPython also makes a decent enough effort in something approaching the right direction, but does not go nearly far enough.
The (Unix) command-line points the way: the idea of the OS as a programming and data analysis environment above all else; the ability to perform complex tasks by streaming data from one small program to another; the use of the keyboard as the primary user input mechanism. (Mice are an ergonomic disaster zone).
From WIMP graphical user interfaces, we should prioritize menus above all else. Menus help the motivated novice to discover tools and functionality in a way that the command line simply cannot. In today's time-constrained world, RTFM is a condescending cop-out. Nobody has the time. Discoverability is everything; the requirement for arcane knowledge (and the time required to achieve it) is both obsolescent and wasteful.
Windows, Icons and Pointers are secondary considerations - The command line and menus should take priority. They could (and arguably should) still exist; but they do not have to dominate the interaction paradigm in the way that they have done in the past, and should be there to support other functionality, not to provide a framework within which other functionality must fit.
Speaking of paradigms; the "desktop metaphor" that is so baked in to the conventional user interface -- it is a redundant piece of skeuomorphism that has outlived it's usefulness. Sure, we can keep the notion of "depth", and layer things over the top of one another in the UI, but the flat, 2D relationships between items - connectedness, proximity, above/below, left/right -- these notions need to take priority and need to be given consistent semantic meanings in the "pattern language" that defines the UI that we create.
What does this all mean in practice?
Well, how about this: A primarily command-line driven interface, but with embedded widgets and graphics. I.e. emphatically NOT a tty - or anything resembling one. Something somewhat akin to tmux can act as a "window manager" - although it would have to be tmux on steroids. The edges and corners of the screen; key pieces of real estate all; should harbour hierarchical menus galore: some static, always where you expect them, some dynamic, perhaps responding to a search, perhaps responding to recently run applications.
Above all else, applications need to work together. Textual commands help us to quickly and easily glue functionality together, pipes help us to move data around the systems that we build, and the graphical capabilities of our user interface allows us to diagram and explore the data-flow of our scripts and systems.
Friday, 20 September 2013
Software & Hardware - How complexity and risk define engineering practice and culture.
Striking cultural differences exist between companies that specialise in software development and those that specialise in hardware development & manufacturing.
The key to understanding the divergence of these two cultures may be found in the differing approaches that each takes to risk and failure; driven both by a difference in the expected cost of failure and a difference in the cost associated with predicting failure.
One approach seeks to mitigate the impact of adverse events by emphasising flexibility and agility; the other approach seeks to minimise the chances that adverse events occur at all, by emphasising predictability and control.
In other words, do you design so you can fix your system quickly when it breaks (at the expense of having it break often), or do you design your system so that it very rarely breaks (but when it does, it is more expensive to fix)?
The answer to this question not only depends on how safety-critical the system is, but how complex it is too. The prediction-and-control approach rapidly becomes untenable when you have systems that reach a certain level of complexity -- the cost of accurately predicting when failures are going to occur rapidly becomes larger than the cost of the failure itself. As the complexity of the system under development increases, the activity looks less like development and more like research. The predictability of disciplined engineering falls apart in the face of sufficient complexity. Worse; complexity increases in a combinatorial manner, so we can very easily move from a predictable system to an unpredictable one with the addition of only a small number of innocuous looking components.
Most forms of mechanical engineering emphasise the second (predictive) approach, particularly for safety critical equipment, since the systems are simple (compared to a lot of software) and the costs of failure are high. On the other hand, a lot of software development emphasises the first (agile/reactive) approach, because the costs associated with failures are (normally) a lot less than the costs associated with development.
Of course, a lot of pejorative terms get mixed up in this, like: "sloppy engineering", and "cowboy developers", vs "expensive failures" and "moribund bureaucracy" but really, these approaches are just the result of the same cost/benefit analysis producing different answers given different input conditions.
Problems mainly arise when you use the wrong risk-management approach for the wrong application; or for the wrong *part* of the application. Things can get quite subtle quite quickly, and managers really need to be on top of their game to succeed.
One of the challenges in developing automotive ADAS systems is that a lot of the software is safety critical, and therefore very expensive to write, because of all of the (necessary) bureaucratic support that the OEMs require for traceability and accountability.
Equally, a lot of the advanced functionality for machine vision / Radar / Lidar signal processing is very advanced, and (unfortunately) has a lot of necessary complexity. As a result it is very very costly to develop when using the former approach; yet may be involved in safety-critical functions.
This is not by any means a solved problem, and very much requires detailed management on a case-by-case basis.
Certainly testing infrastructure becomes much more important as the sensor systems that we develop become more complex. (Disclaimer: my area of interest). Indeed, my experience indicates that for sophisticated sensor systems well over 80% of the effort (measured by both in hours of development & size of the code-base) is associated with test infrastructure, and less than 20% with the software that ends up in the vehicle.
Perhaps the word "test" is a misnomer here; since the role of this infrastructure is not so much to do V&V on the completed system, but to help to develop the system's requirements -- to do the "Data Science" and analytics that are needed to understand the operating environment well enough that you can correctly specify the behaviour of the application.
The key to understanding the divergence of these two cultures may be found in the differing approaches that each takes to risk and failure; driven both by a difference in the expected cost of failure and a difference in the cost associated with predicting failure.
One approach seeks to mitigate the impact of adverse events by emphasising flexibility and agility; the other approach seeks to minimise the chances that adverse events occur at all, by emphasising predictability and control.
In other words, do you design so you can fix your system quickly when it breaks (at the expense of having it break often), or do you design your system so that it very rarely breaks (but when it does, it is more expensive to fix)?
The answer to this question not only depends on how safety-critical the system is, but how complex it is too. The prediction-and-control approach rapidly becomes untenable when you have systems that reach a certain level of complexity -- the cost of accurately predicting when failures are going to occur rapidly becomes larger than the cost of the failure itself. As the complexity of the system under development increases, the activity looks less like development and more like research. The predictability of disciplined engineering falls apart in the face of sufficient complexity. Worse; complexity increases in a combinatorial manner, so we can very easily move from a predictable system to an unpredictable one with the addition of only a small number of innocuous looking components.
Most forms of mechanical engineering emphasise the second (predictive) approach, particularly for safety critical equipment, since the systems are simple (compared to a lot of software) and the costs of failure are high. On the other hand, a lot of software development emphasises the first (agile/reactive) approach, because the costs associated with failures are (normally) a lot less than the costs associated with development.
Of course, a lot of pejorative terms get mixed up in this, like: "sloppy engineering", and "cowboy developers", vs "expensive failures" and "moribund bureaucracy" but really, these approaches are just the result of the same cost/benefit analysis producing different answers given different input conditions.
Problems mainly arise when you use the wrong risk-management approach for the wrong application; or for the wrong *part* of the application. Things can get quite subtle quite quickly, and managers really need to be on top of their game to succeed.
One of the challenges in developing automotive ADAS systems is that a lot of the software is safety critical, and therefore very expensive to write, because of all of the (necessary) bureaucratic support that the OEMs require for traceability and accountability.
Equally, a lot of the advanced functionality for machine vision / Radar / Lidar signal processing is very advanced, and (unfortunately) has a lot of necessary complexity. As a result it is very very costly to develop when using the former approach; yet may be involved in safety-critical functions.
This is not by any means a solved problem, and very much requires detailed management on a case-by-case basis.
Certainly testing infrastructure becomes much more important as the sensor systems that we develop become more complex. (Disclaimer: my area of interest). Indeed, my experience indicates that for sophisticated sensor systems well over 80% of the effort (measured by both in hours of development & size of the code-base) is associated with test infrastructure, and less than 20% with the software that ends up in the vehicle.
Perhaps the word "test" is a misnomer here; since the role of this infrastructure is not so much to do V&V on the completed system, but to help to develop the system's requirements -- to do the "Data Science" and analytics that are needed to understand the operating environment well enough that you can correctly specify the behaviour of the application.
Monday, 16 September 2013
Dstillery
Media 6 Degrees (the new owner of my former employer, EveryScreen Media), has changed it's name to "Dstillery".
http://www.fastcompany.com/3017495/dstillery-is-picasso-in-the-dark-art-of-digital-advertising
This Fast Company article does a remarkably good job of explaining what the technology delivers. (Disclosure: I implemented the early mobile-phone IP-tracking algorithms when I worked at EveryScreen Media).
Whilst I do believe (from what I saw) that ESM/M6D/Dstillery take privacy very seriously; and will (continue to) behave in a responsible manner, I still feel a sense of unease when I think about the extent to which participants in the advertising industry are able to peer into people's personal lives.
To balance this (sort-of) criticism, I feel I should emphasise the fact that the advertising industry is, in general, a pile 'em high sell 'em cheap kind of affair, where advertising impressions are bought and sold by the million; where performance is measured in statistical terms; and where the idea of paying close attention to any one individual would be laughed off as a ludicrous waste of time.
However, it is possible that not all participants will feel that way, just as it is possible that not all participants will be as motivated to act responsibly as my former employers were.
There has been a lot of debate recently about NSA spying on what we read on-line and using our mobile phones to track where we go in the world. Well, you don't need anything like the NSA's multi-billion dollar budgets and mastery of (de)cryptography to do something that feels (like it could be) similarly invasive. (The advertising industry is an order of magnitude less creepy, but it is facilitated by the same social & technological developments, and is still heading in a very similar direction).
I think that this is something that we should think very carefully about, and just as we seek to find better ways to regulate our security services, so too we should (carefully; deliberately; deliberatively) seek to find ways to regulate the flow of personal information around our advertising ecosystem.
--
Edit: One thing that does bear mentioning -- the link up between M6D & ESM really is a smart move. From a data point of view, it is a marriage made in heaven (a bit of a no-brainer, actually); and I think that the insights that result from combining their respective data-streams will yield genuine benefits for their clients & the brands that they represent.
http://www.fastcompany.com/3017495/dstillery-is-picasso-in-the-dark-art-of-digital-advertising
This Fast Company article does a remarkably good job of explaining what the technology delivers. (Disclosure: I implemented the early mobile-phone IP-tracking algorithms when I worked at EveryScreen Media).
Whilst I do believe (from what I saw) that ESM/M6D/Dstillery take privacy very seriously; and will (continue to) behave in a responsible manner, I still feel a sense of unease when I think about the extent to which participants in the advertising industry are able to peer into people's personal lives.
To balance this (sort-of) criticism, I feel I should emphasise the fact that the advertising industry is, in general, a pile 'em high sell 'em cheap kind of affair, where advertising impressions are bought and sold by the million; where performance is measured in statistical terms; and where the idea of paying close attention to any one individual would be laughed off as a ludicrous waste of time.
However, it is possible that not all participants will feel that way, just as it is possible that not all participants will be as motivated to act responsibly as my former employers were.
There has been a lot of debate recently about NSA spying on what we read on-line and using our mobile phones to track where we go in the world. Well, you don't need anything like the NSA's multi-billion dollar budgets and mastery of (de)cryptography to do something that feels (like it could be) similarly invasive. (The advertising industry is an order of magnitude less creepy, but it is facilitated by the same social & technological developments, and is still heading in a very similar direction).
I think that this is something that we should think very carefully about, and just as we seek to find better ways to regulate our security services, so too we should (carefully; deliberately; deliberatively) seek to find ways to regulate the flow of personal information around our advertising ecosystem.
--
Edit: One thing that does bear mentioning -- the link up between M6D & ESM really is a smart move. From a data point of view, it is a marriage made in heaven (a bit of a no-brainer, actually); and I think that the insights that result from combining their respective data-streams will yield genuine benefits for their clients & the brands that they represent.
Wednesday, 4 September 2013
Antisocial development
Human beings are social creatures; to the extent that our sanity can be undermined by isolation.
Much of our behaviour is guided and controlled by observing others around us. We instinctively imitate our peers, and follow normative standards of behaviour. As a result, I believe that the most effective learning occurs in a social context; with the presence of peers to shape and guide behaviours.
I also believe that learning is at the centre of what we do as software developers. Functioning software is seemingly more a by-product of learning than the converse.
It is therefore a great pity that modern development methods seem tailor-made to encourage developers to work alone; to minimize the need for contact with their peers and colleagues, and to reduce the need for social interaction.
For example, part of the appeal of the modern distributed version control system is that it allows individuals to work independently and alone, without requiring coordination or synchronisation.
It is possible that this has a rational basis; After all Fred Brooks' analysis in: "The Mythical Man-Month" suggests that optimally sized teams are composed of a single solitary individual, Conway's Law also seems to suggest that maximum modularity is achieved by distributed teams.
Perhaps developer solitude really is the global optimum towards which we, as an industry, are headed. However, this clearly neglects the important social aspects of human behaviour, as well as our need to learn as a group.
I wonder if we ever see a resurgence of tools that support and encourage face-to-face social interaction and learning; rather than obviate the need for it.
Subscribe to:
Posts (Atom)