The typing pool has been superseded by the word processor; clerical workers by the spreadsheet and the database. What used to require the management of men, now requires the management of machines.
Two very different tasks indeed.
Man-management is a very interesting, very challenging job. Marshalling and aligning a group of humans to meet a common objective requires the ability to handle the vagaries of human relationships; interpersonal conflicts, political machinations, uncertain and conflicting motivations, forgetfulness, imperfect communication and understanding ... the challenges that must be overcome form a long long list. Indeed, one might be excused if the challenge of getting other people to do what you want them to rather masks the extent to which our objectives are woolly and ill-defined in the first place.
By contrast, as a programmer, our minions are infinitely more compliant; patient, predictable and indefatigable. The orders that we issue will be followed to the minutest detail. Indeed, the orders that we issue must be *specified* in the minutest detail.
Therein lies the rub.
The professions of programming and man-management both require the professional to make decisions and to delegate tasks. For the man-manager, the central challenge is to ensure that the tasks are completed; for the developer, the central challenge is to ensure that the tasks (and their consequences) are well enough understood in the first place.
This is surprisingly hard; especially the consequences bit, and especially in the face of complexity.
Wednesday, 2 October 2013
To what extent can our brains accommodate externally-directed actions without recognizing them as such?
The human brain is parallel and decentralized, yet we somehow manage to maintain the illusion that we are a single conscious entity, experiencing life in a sequential stream of thoughts and experiences.
This is clearly a lie that we tell ourselves. Various independent bits of brain make decisions and initiate actions entirely on their own, and somehow we manage to rationalize and confabulate and merrily deceive ourselves (after the fact) that we had some sort of explicit, sequential, conscious plan all along.
Whilst this model of human behavior is disturbing on a number of levels, it does have some interesting consequences when we consider the near-future technology of brain augmentation.
It is plausible that we could embed a bit of electronics into our brains; integrated so tightly that it is able to make decisions for us; to control our actions and influence our perceptions.
Would we feel that we were being controlled? Or would we integrate those decisions into our stream of consciousness: to confabulate a reality in which these decisions really were made by our conscious free will?
Will the perceptual pathways responsible for our self-perception bend to accommodate outside influences; to stretch the notion of self to accommodate the other? To allow us to believe that we initiated acts that were (in reality) initiated by others?
This is clearly a lie that we tell ourselves. Various independent bits of brain make decisions and initiate actions entirely on their own, and somehow we manage to rationalize and confabulate and merrily deceive ourselves (after the fact) that we had some sort of explicit, sequential, conscious plan all along.
Whilst this model of human behavior is disturbing on a number of levels, it does have some interesting consequences when we consider the near-future technology of brain augmentation.
It is plausible that we could embed a bit of electronics into our brains; integrated so tightly that it is able to make decisions for us; to control our actions and influence our perceptions.
Would we feel that we were being controlled? Or would we integrate those decisions into our stream of consciousness: to confabulate a reality in which these decisions really were made by our conscious free will?
Will the perceptual pathways responsible for our self-perception bend to accommodate outside influences; to stretch the notion of self to accommodate the other? To allow us to believe that we initiated acts that were (in reality) initiated by others?
GUI-NG Desiderata
The command line may be powerful and flexible, but it is ugly to look at and difficult to learn. WIMP style graphical user interfaces look nicer, and are easier to pick up and use, but they don't really help you do anything other than the most basic and pedestrian of tasks.
Cm'on guys, we should be able to do better than this. We should be able to design beautiful, powerful tools with a UI concept that combines the best of both worlds.
A beautiful, well designed UI for the power user: The design sense of Apple, the good looks of Lighttable and Sublime text; the power and flexibility of Unix.
The Acme editor from Plan9 shows us how radical we can be. IPython also makes a decent enough effort in something approaching the right direction, but does not go nearly far enough.
The (Unix) command-line points the way: the idea of the OS as a programming and data analysis environment above all else; the ability to perform complex tasks by streaming data from one small program to another; the use of the keyboard as the primary user input mechanism. (Mice are an ergonomic disaster zone).
From WIMP graphical user interfaces, we should prioritize menus above all else. Menus help the motivated novice to discover tools and functionality in a way that the command line simply cannot. In today's time-constrained world, RTFM is a condescending cop-out. Nobody has the time. Discoverability is everything; the requirement for arcane knowledge (and the time required to achieve it) is both obsolescent and wasteful.
Windows, Icons and Pointers are secondary considerations - The command line and menus should take priority. They could (and arguably should) still exist; but they do not have to dominate the interaction paradigm in the way that they have done in the past, and should be there to support other functionality, not to provide a framework within which other functionality must fit.
Speaking of paradigms; the "desktop metaphor" that is so baked in to the conventional user interface -- it is a redundant piece of skeuomorphism that has outlived it's usefulness. Sure, we can keep the notion of "depth", and layer things over the top of one another in the UI, but the flat, 2D relationships between items - connectedness, proximity, above/below, left/right -- these notions need to take priority and need to be given consistent semantic meanings in the "pattern language" that defines the UI that we create.
What does this all mean in practice?
Well, how about this: A primarily command-line driven interface, but with embedded widgets and graphics. I.e. emphatically NOT a tty - or anything resembling one. Something somewhat akin to tmux can act as a "window manager" - although it would have to be tmux on steroids. The edges and corners of the screen; key pieces of real estate all; should harbour hierarchical menus galore: some static, always where you expect them, some dynamic, perhaps responding to a search, perhaps responding to recently run applications.
Above all else, applications need to work together. Textual commands help us to quickly and easily glue functionality together, pipes help us to move data around the systems that we build, and the graphical capabilities of our user interface allows us to diagram and explore the data-flow of our scripts and systems.
Cm'on guys, we should be able to do better than this. We should be able to design beautiful, powerful tools with a UI concept that combines the best of both worlds.
A beautiful, well designed UI for the power user: The design sense of Apple, the good looks of Lighttable and Sublime text; the power and flexibility of Unix.
The Acme editor from Plan9 shows us how radical we can be. IPython also makes a decent enough effort in something approaching the right direction, but does not go nearly far enough.
The (Unix) command-line points the way: the idea of the OS as a programming and data analysis environment above all else; the ability to perform complex tasks by streaming data from one small program to another; the use of the keyboard as the primary user input mechanism. (Mice are an ergonomic disaster zone).
From WIMP graphical user interfaces, we should prioritize menus above all else. Menus help the motivated novice to discover tools and functionality in a way that the command line simply cannot. In today's time-constrained world, RTFM is a condescending cop-out. Nobody has the time. Discoverability is everything; the requirement for arcane knowledge (and the time required to achieve it) is both obsolescent and wasteful.
Windows, Icons and Pointers are secondary considerations - The command line and menus should take priority. They could (and arguably should) still exist; but they do not have to dominate the interaction paradigm in the way that they have done in the past, and should be there to support other functionality, not to provide a framework within which other functionality must fit.
Speaking of paradigms; the "desktop metaphor" that is so baked in to the conventional user interface -- it is a redundant piece of skeuomorphism that has outlived it's usefulness. Sure, we can keep the notion of "depth", and layer things over the top of one another in the UI, but the flat, 2D relationships between items - connectedness, proximity, above/below, left/right -- these notions need to take priority and need to be given consistent semantic meanings in the "pattern language" that defines the UI that we create.
What does this all mean in practice?
Well, how about this: A primarily command-line driven interface, but with embedded widgets and graphics. I.e. emphatically NOT a tty - or anything resembling one. Something somewhat akin to tmux can act as a "window manager" - although it would have to be tmux on steroids. The edges and corners of the screen; key pieces of real estate all; should harbour hierarchical menus galore: some static, always where you expect them, some dynamic, perhaps responding to a search, perhaps responding to recently run applications.
Above all else, applications need to work together. Textual commands help us to quickly and easily glue functionality together, pipes help us to move data around the systems that we build, and the graphical capabilities of our user interface allows us to diagram and explore the data-flow of our scripts and systems.
Friday, 20 September 2013
Software & Hardware - How complexity and risk define engineering practice and culture.
Striking cultural differences exist between companies that specialise in software development and those that specialise in hardware development & manufacturing.
The key to understanding the divergence of these two cultures may be found in the differing approaches that each takes to risk and failure; driven both by a difference in the expected cost of failure and a difference in the cost associated with predicting failure.
One approach seeks to mitigate the impact of adverse events by emphasising flexibility and agility; the other approach seeks to minimise the chances that adverse events occur at all, by emphasising predictability and control.
In other words, do you design so you can fix your system quickly when it breaks (at the expense of having it break often), or do you design your system so that it very rarely breaks (but when it does, it is more expensive to fix)?
The answer to this question not only depends on how safety-critical the system is, but how complex it is too. The prediction-and-control approach rapidly becomes untenable when you have systems that reach a certain level of complexity -- the cost of accurately predicting when failures are going to occur rapidly becomes larger than the cost of the failure itself. As the complexity of the system under development increases, the activity looks less like development and more like research. The predictability of disciplined engineering falls apart in the face of sufficient complexity. Worse; complexity increases in a combinatorial manner, so we can very easily move from a predictable system to an unpredictable one with the addition of only a small number of innocuous looking components.
Most forms of mechanical engineering emphasise the second (predictive) approach, particularly for safety critical equipment, since the systems are simple (compared to a lot of software) and the costs of failure are high. On the other hand, a lot of software development emphasises the first (agile/reactive) approach, because the costs associated with failures are (normally) a lot less than the costs associated with development.
Of course, a lot of pejorative terms get mixed up in this, like: "sloppy engineering", and "cowboy developers", vs "expensive failures" and "moribund bureaucracy" but really, these approaches are just the result of the same cost/benefit analysis producing different answers given different input conditions.
Problems mainly arise when you use the wrong risk-management approach for the wrong application; or for the wrong *part* of the application. Things can get quite subtle quite quickly, and managers really need to be on top of their game to succeed.
One of the challenges in developing automotive ADAS systems is that a lot of the software is safety critical, and therefore very expensive to write, because of all of the (necessary) bureaucratic support that the OEMs require for traceability and accountability.
Equally, a lot of the advanced functionality for machine vision / Radar / Lidar signal processing is very advanced, and (unfortunately) has a lot of necessary complexity. As a result it is very very costly to develop when using the former approach; yet may be involved in safety-critical functions.
This is not by any means a solved problem, and very much requires detailed management on a case-by-case basis.
Certainly testing infrastructure becomes much more important as the sensor systems that we develop become more complex. (Disclaimer: my area of interest). Indeed, my experience indicates that for sophisticated sensor systems well over 80% of the effort (measured by both in hours of development & size of the code-base) is associated with test infrastructure, and less than 20% with the software that ends up in the vehicle.
Perhaps the word "test" is a misnomer here; since the role of this infrastructure is not so much to do V&V on the completed system, but to help to develop the system's requirements -- to do the "Data Science" and analytics that are needed to understand the operating environment well enough that you can correctly specify the behaviour of the application.
The key to understanding the divergence of these two cultures may be found in the differing approaches that each takes to risk and failure; driven both by a difference in the expected cost of failure and a difference in the cost associated with predicting failure.
One approach seeks to mitigate the impact of adverse events by emphasising flexibility and agility; the other approach seeks to minimise the chances that adverse events occur at all, by emphasising predictability and control.
In other words, do you design so you can fix your system quickly when it breaks (at the expense of having it break often), or do you design your system so that it very rarely breaks (but when it does, it is more expensive to fix)?
The answer to this question not only depends on how safety-critical the system is, but how complex it is too. The prediction-and-control approach rapidly becomes untenable when you have systems that reach a certain level of complexity -- the cost of accurately predicting when failures are going to occur rapidly becomes larger than the cost of the failure itself. As the complexity of the system under development increases, the activity looks less like development and more like research. The predictability of disciplined engineering falls apart in the face of sufficient complexity. Worse; complexity increases in a combinatorial manner, so we can very easily move from a predictable system to an unpredictable one with the addition of only a small number of innocuous looking components.
Most forms of mechanical engineering emphasise the second (predictive) approach, particularly for safety critical equipment, since the systems are simple (compared to a lot of software) and the costs of failure are high. On the other hand, a lot of software development emphasises the first (agile/reactive) approach, because the costs associated with failures are (normally) a lot less than the costs associated with development.
Of course, a lot of pejorative terms get mixed up in this, like: "sloppy engineering", and "cowboy developers", vs "expensive failures" and "moribund bureaucracy" but really, these approaches are just the result of the same cost/benefit analysis producing different answers given different input conditions.
Problems mainly arise when you use the wrong risk-management approach for the wrong application; or for the wrong *part* of the application. Things can get quite subtle quite quickly, and managers really need to be on top of their game to succeed.
One of the challenges in developing automotive ADAS systems is that a lot of the software is safety critical, and therefore very expensive to write, because of all of the (necessary) bureaucratic support that the OEMs require for traceability and accountability.
Equally, a lot of the advanced functionality for machine vision / Radar / Lidar signal processing is very advanced, and (unfortunately) has a lot of necessary complexity. As a result it is very very costly to develop when using the former approach; yet may be involved in safety-critical functions.
This is not by any means a solved problem, and very much requires detailed management on a case-by-case basis.
Certainly testing infrastructure becomes much more important as the sensor systems that we develop become more complex. (Disclaimer: my area of interest). Indeed, my experience indicates that for sophisticated sensor systems well over 80% of the effort (measured by both in hours of development & size of the code-base) is associated with test infrastructure, and less than 20% with the software that ends up in the vehicle.
Perhaps the word "test" is a misnomer here; since the role of this infrastructure is not so much to do V&V on the completed system, but to help to develop the system's requirements -- to do the "Data Science" and analytics that are needed to understand the operating environment well enough that you can correctly specify the behaviour of the application.
Monday, 16 September 2013
Dstillery
Media 6 Degrees (the new owner of my former employer, EveryScreen Media), has changed it's name to "Dstillery".
http://www.fastcompany.com/3017495/dstillery-is-picasso-in-the-dark-art-of-digital-advertising
This Fast Company article does a remarkably good job of explaining what the technology delivers. (Disclosure: I implemented the early mobile-phone IP-tracking algorithms when I worked at EveryScreen Media).
Whilst I do believe (from what I saw) that ESM/M6D/Dstillery take privacy very seriously; and will (continue to) behave in a responsible manner, I still feel a sense of unease when I think about the extent to which participants in the advertising industry are able to peer into people's personal lives.
To balance this (sort-of) criticism, I feel I should emphasise the fact that the advertising industry is, in general, a pile 'em high sell 'em cheap kind of affair, where advertising impressions are bought and sold by the million; where performance is measured in statistical terms; and where the idea of paying close attention to any one individual would be laughed off as a ludicrous waste of time.
However, it is possible that not all participants will feel that way, just as it is possible that not all participants will be as motivated to act responsibly as my former employers were.
There has been a lot of debate recently about NSA spying on what we read on-line and using our mobile phones to track where we go in the world. Well, you don't need anything like the NSA's multi-billion dollar budgets and mastery of (de)cryptography to do something that feels (like it could be) similarly invasive. (The advertising industry is an order of magnitude less creepy, but it is facilitated by the same social & technological developments, and is still heading in a very similar direction).
I think that this is something that we should think very carefully about, and just as we seek to find better ways to regulate our security services, so too we should (carefully; deliberately; deliberatively) seek to find ways to regulate the flow of personal information around our advertising ecosystem.
--
Edit: One thing that does bear mentioning -- the link up between M6D & ESM really is a smart move. From a data point of view, it is a marriage made in heaven (a bit of a no-brainer, actually); and I think that the insights that result from combining their respective data-streams will yield genuine benefits for their clients & the brands that they represent.
http://www.fastcompany.com/3017495/dstillery-is-picasso-in-the-dark-art-of-digital-advertising
This Fast Company article does a remarkably good job of explaining what the technology delivers. (Disclosure: I implemented the early mobile-phone IP-tracking algorithms when I worked at EveryScreen Media).
Whilst I do believe (from what I saw) that ESM/M6D/Dstillery take privacy very seriously; and will (continue to) behave in a responsible manner, I still feel a sense of unease when I think about the extent to which participants in the advertising industry are able to peer into people's personal lives.
To balance this (sort-of) criticism, I feel I should emphasise the fact that the advertising industry is, in general, a pile 'em high sell 'em cheap kind of affair, where advertising impressions are bought and sold by the million; where performance is measured in statistical terms; and where the idea of paying close attention to any one individual would be laughed off as a ludicrous waste of time.
However, it is possible that not all participants will feel that way, just as it is possible that not all participants will be as motivated to act responsibly as my former employers were.
There has been a lot of debate recently about NSA spying on what we read on-line and using our mobile phones to track where we go in the world. Well, you don't need anything like the NSA's multi-billion dollar budgets and mastery of (de)cryptography to do something that feels (like it could be) similarly invasive. (The advertising industry is an order of magnitude less creepy, but it is facilitated by the same social & technological developments, and is still heading in a very similar direction).
I think that this is something that we should think very carefully about, and just as we seek to find better ways to regulate our security services, so too we should (carefully; deliberately; deliberatively) seek to find ways to regulate the flow of personal information around our advertising ecosystem.
--
Edit: One thing that does bear mentioning -- the link up between M6D & ESM really is a smart move. From a data point of view, it is a marriage made in heaven (a bit of a no-brainer, actually); and I think that the insights that result from combining their respective data-streams will yield genuine benefits for their clients & the brands that they represent.
Wednesday, 4 September 2013
Antisocial development
Human beings are social creatures; to the extent that our sanity can be undermined by isolation.
Much of our behaviour is guided and controlled by observing others around us. We instinctively imitate our peers, and follow normative standards of behaviour. As a result, I believe that the most effective learning occurs in a social context; with the presence of peers to shape and guide behaviours.
I also believe that learning is at the centre of what we do as software developers. Functioning software is seemingly more a by-product of learning than the converse.
It is therefore a great pity that modern development methods seem tailor-made to encourage developers to work alone; to minimize the need for contact with their peers and colleagues, and to reduce the need for social interaction.
For example, part of the appeal of the modern distributed version control system is that it allows individuals to work independently and alone, without requiring coordination or synchronisation.
It is possible that this has a rational basis; After all Fred Brooks' analysis in: "The Mythical Man-Month" suggests that optimally sized teams are composed of a single solitary individual, Conway's Law also seems to suggest that maximum modularity is achieved by distributed teams.
Perhaps developer solitude really is the global optimum towards which we, as an industry, are headed. However, this clearly neglects the important social aspects of human behaviour, as well as our need to learn as a group.
I wonder if we ever see a resurgence of tools that support and encourage face-to-face social interaction and learning; rather than obviate the need for it.
Friday, 16 August 2013
Managing Complexity
Quoting from Djikstra's "Go-To Considered Harmful" essay:
If we can shovel most of the essential complexity into the stateless parts, we can make sure that this complexity is (at least) easily testable.
Equally, if we can keep the stateful parts of the program relatively free of complexity, then we can (at least) reason about (and control) how the state can change.
This argues for a system architecture that has parts that are programmed in an Object-Oriented manner as well as parts that are programmed in a pure-functional manner.
Objects enable us to make state private, and to limit and control how that state is manipulated (through public interfaces).
The use of pure functions makes it easier to reason about and test the more complex parts of system functionality.
It is when we mix statefulness and complexity that we really come unstuck.
Of course, it goes without saying that if you can avoid state you should, and if you can avoid complexity, you should ... but in my experience at least a little of both is usually required as a cost of doing business.
"My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed."And from Moseley and Marks' "Out of the Tar Pit" essay:
"We believe that the major contributor to this complexity in many systems is the handling of state and the burden that this adds when trying to analyse and reason about the system."We find it difficult to reason about how system state changes over time, particularly when lots of things can modify that state. It therefore makes sense to (mentally) partition our systems into stateless and stateful parts.
If we can shovel most of the essential complexity into the stateless parts, we can make sure that this complexity is (at least) easily testable.
Equally, if we can keep the stateful parts of the program relatively free of complexity, then we can (at least) reason about (and control) how the state can change.
This argues for a system architecture that has parts that are programmed in an Object-Oriented manner as well as parts that are programmed in a pure-functional manner.
Objects enable us to make state private, and to limit and control how that state is manipulated (through public interfaces).
The use of pure functions makes it easier to reason about and test the more complex parts of system functionality.
It is when we mix statefulness and complexity that we really come unstuck.
Of course, it goes without saying that if you can avoid state you should, and if you can avoid complexity, you should ... but in my experience at least a little of both is usually required as a cost of doing business.
Wednesday, 10 July 2013
Google Logic - The Challenge of Agility
Here is an interesting Guardian article examining Google's management culture. It starts of by contrasting traditional long term business planning and culture with the more "agile" approach that is exemplified by Google.
The article correctly notes that this approach to product development requires a culture that is fundamentally different from the culture and practices found at traditional (long-term-plan oriented) organizations.
These traditional cultures struggle to deal with shifting goals and a quickly changing competitive landscape.
In addition to these criticisms, I would like to add another of my own:
I do not think that these prediction & planning-oriented approaches are any good at dealing with complexity. This is particularly clear when faced with the complexity involved in the development of modern software systems.
The reasons are forehead-slappingly obvious:
Long term planning or indeed, any planning at all, relies on making predictions. Making predictions is difficult. Traditional companies respond by trying harder, by limiting the scope of what they try to predict, or by deluding themselves that they are better at it than they are. The longer your time-frame and the more complex your plan/product, the worse your predictions become. Indeed, the difficulty of making predictions increases catastrophically quickly with the complexity of your product - making planning for all but the simplest product development nothing better than a wild and irresponsible gamble. Various cognitive biases compound this difficulty, making it difficult for us to measure or even acknowledge complexity that we cannot see.
This is a very real and very serious multi-billion-dollar crisis - and it's impact extends far beyond the field of software development - although this is where it's effects are most easily and most keenly felt. How to escape from this tar-pit? How to deal with an unpredictable, complex, and fundamentally "unmanageable" world?
This, it is not easy.
It is clear that "agile/reactive" approaches are going to be a key, important part of the solution - but experience has shown that implementing an agile culture and achieving success is far from easy - it remains a long way from the turn-key exercise that we might wish it to be. Even Google, who you claim champions this approach, is hardly the shining beacon that you suggest. Good enough, perhaps, but hardly exemplary.
The agile culture needs to be able to plan to reduce the impact of adverse events, without knowing in advance what they will be. Similarly, it needs to plan to increase the impact of favourable events, again without knowing in advance what they will be. At the same time, it needs to balance this with the need to keep on top of and maintain a complex and growing inventory of capabilities and skills; technical and human capital, all of which is susceptible to the forces of entropy and decay.
It has been said that developing software (in the large) is akin to growing a plant - it requires tending & guiding as it grows, rather than planning and "engineering" (in the bridge-building sense). I imagine very much that the senior leadership of Google feel the same way.
Friday, 28 June 2013
The Next Big Thing
You might not have noticed it yet, but the leaves on the grapevine are rustling, and there is a whispering breeze in your ear. This, my friends, is the sound of the winds of change: gathering, growing, building momentum. There is a storm brewing, friends, and the game: It is afoot!
So many people: working quietly, diligently, are spear-heading that change, and so few of us realize that it is even happening, let alone comprehend the profound; monumental; world-changing implications of the technological tidal-wave that is coming our way.
Automotive industry dollars are funding this work; the dream of safer roads, of self-driving cars is driving it forwards, but this dream on four wheels is merely the tip of the iceberg: the gateway drug that leads us to an intoxicating, and perhaps slightly scary future.
What are self-driving cars but mass produced autonomous robots? Autonomous vehicles and robots will be to my daughter's generation what computers and the internet were to mine; except this change will be bigger and far more wide-reaching, as the machines step off the desktop and climb out of the server-room, and march onto the streets; as our mobile phones and tablets sprout wheels and wings and legs and arms -- as our algorithms and neural networks and distributed systems stop passively sucking in click-stream data and twitter sentiment scores, and start actively participating in a world that they can see, hear and touch.
The physical world that my parents inhabited is not so very different from the physical world that I inherited, but the world that I pass on to my daughters will change so quickly, so rapidly, that I cannot begin to imagine what it will look like when they are my age, and working away on shaping a future for their children.
What a singularly monumental change is afoot! What a wonderful, exhilarating time to be alive! And how exciting to be a part of it!
http://bits.blogs.nytimes.com/2013/07/07/disruptions-how-driverless-cars-could-reshape-cities/?smid=tw-nytimes
So many people: working quietly, diligently, are spear-heading that change, and so few of us realize that it is even happening, let alone comprehend the profound; monumental; world-changing implications of the technological tidal-wave that is coming our way.
Automotive industry dollars are funding this work; the dream of safer roads, of self-driving cars is driving it forwards, but this dream on four wheels is merely the tip of the iceberg: the gateway drug that leads us to an intoxicating, and perhaps slightly scary future.
What are self-driving cars but mass produced autonomous robots? Autonomous vehicles and robots will be to my daughter's generation what computers and the internet were to mine; except this change will be bigger and far more wide-reaching, as the machines step off the desktop and climb out of the server-room, and march onto the streets; as our mobile phones and tablets sprout wheels and wings and legs and arms -- as our algorithms and neural networks and distributed systems stop passively sucking in click-stream data and twitter sentiment scores, and start actively participating in a world that they can see, hear and touch.
The physical world that my parents inhabited is not so very different from the physical world that I inherited, but the world that I pass on to my daughters will change so quickly, so rapidly, that I cannot begin to imagine what it will look like when they are my age, and working away on shaping a future for their children.
What a singularly monumental change is afoot! What a wonderful, exhilarating time to be alive! And how exciting to be a part of it!
http://bits.blogs.nytimes.com/2013/07/07/disruptions-how-driverless-cars-could-reshape-cities/?smid=tw-nytimes
Wednesday, 26 June 2013
Thoughts on Compressed Sensing
Imagine that you have a sparsely sampled signal (with many missing measurements), and you want to recover a densely sampled version of the same signal.
If you can find a function that is capable of transforming the original signal into a representation that is also sparse (with many zero or trivially non-zero coefficients), then you have a good heuristic that you can use to find an approximation of the original signal:- maximising sparsity in the transform domain. This is really just Occam's razor - if you have good reason to suppose that the data can be fit by a simple model, then choosing the simplest model that fits the data is an optimal guessing strategy.
Good examples of such functions include the Fourier transform or various wavelet transforms - indeed, any parametric approximation or model would probably do the job.
If the transform domain really is sparse, and each coefficient in the transform domain impacts multiple measurements in the original domain, then (with high probability) you will be able to guess the "right" coefficients in the transform domain by simply
picking the set of coefficients in the transform domain which both reproduces the original signal and minimizes the number of non-zero components in the transform domain.
However, choosing the most parsimonious model is only one kind of regularization that we can (and should) deploy. We should also think about the set of coefficients that represents the most physically and statistically plausible model.
Thus, when choosing our approach to regularization, we need to consider three things:
* Parsimony (As represented by sparsity, approximated by the L0 and L1 norms).
* Physical plausibility (A transform domain closely related to a physical model is needed here).
* Statistical plausibility (A transform domain where coefficients are as independent as possible is needed here - to reduce the number and dimensionality of the joint distributions that we need to calculate).
These three requirements are not necessarily harmonious. Does that mean we need three different transforms? How do we reconcile differences when they crop up?
If you can find a function that is capable of transforming the original signal into a representation that is also sparse (with many zero or trivially non-zero coefficients), then you have a good heuristic that you can use to find an approximation of the original signal:- maximising sparsity in the transform domain. This is really just Occam's razor - if you have good reason to suppose that the data can be fit by a simple model, then choosing the simplest model that fits the data is an optimal guessing strategy.
Good examples of such functions include the Fourier transform or various wavelet transforms - indeed, any parametric approximation or model would probably do the job.
If the transform domain really is sparse, and each coefficient in the transform domain impacts multiple measurements in the original domain, then (with high probability) you will be able to guess the "right" coefficients in the transform domain by simply
picking the set of coefficients in the transform domain which both reproduces the original signal and minimizes the number of non-zero components in the transform domain.
However, choosing the most parsimonious model is only one kind of regularization that we can (and should) deploy. We should also think about the set of coefficients that represents the most physically and statistically plausible model.
Thus, when choosing our approach to regularization, we need to consider three things:
* Parsimony (As represented by sparsity, approximated by the L0 and L1 norms).
* Physical plausibility (A transform domain closely related to a physical model is needed here).
* Statistical plausibility (A transform domain where coefficients are as independent as possible is needed here - to reduce the number and dimensionality of the joint distributions that we need to calculate).
These three requirements are not necessarily harmonious. Does that mean we need three different transforms? How do we reconcile differences when they crop up?
Friday, 14 June 2013
Code review tool affordances
I have had a bit of experience using different code review tools: Gerrit most recently, and Atlassian's FishEye in the roles preceding that.
Strangely, the most effective code reviews that I have participated happened at Sophos, where no tool at all was used: The rule was simply to get another developer to sit next to you when you did a commit, so you could talk him through the code line by line before submitting the change.
The experience there was that most (if not all) of the problems were spotted by the original developer, not by the reviewer, who frequently lacked the contextual awareness to identify anything deeper and more significant than the usual picayune: formatting and layout errors -- the real value came from being forced to re-examine and explain your work - controlling the searchlight of your attention into a more disciplined and fine-grained search pattern than is possible alone.
The other types of review to which I have been party:- asynchronous, distributed reviews mediated by a web tool of some sort, as well as formal half-dozen-people-in-a-room-with-slides style reviews have, in my experience, proven far less effective.
So, I sit here wondering if we can rescue the asynchronous distributed code review tool, either through an alternative approach or the application of a formal and disciplined approach of some sort ... or if it is doomed to more-than-uselessness?
Strangely, the most effective code reviews that I have participated happened at Sophos, where no tool at all was used: The rule was simply to get another developer to sit next to you when you did a commit, so you could talk him through the code line by line before submitting the change.
The experience there was that most (if not all) of the problems were spotted by the original developer, not by the reviewer, who frequently lacked the contextual awareness to identify anything deeper and more significant than the usual picayune: formatting and layout errors -- the real value came from being forced to re-examine and explain your work - controlling the searchlight of your attention into a more disciplined and fine-grained search pattern than is possible alone.
The other types of review to which I have been party:- asynchronous, distributed reviews mediated by a web tool of some sort, as well as formal half-dozen-people-in-a-room-with-slides style reviews have, in my experience, proven far less effective.
So, I sit here wondering if we can rescue the asynchronous distributed code review tool, either through an alternative approach or the application of a formal and disciplined approach of some sort ... or if it is doomed to more-than-uselessness?
Monday, 3 June 2013
Dual functional & OO interfaces for program decomposition into stateless & stateful parts.
If you expose complex logic through a functional interface, it is (much) easier to test due to it's statelessness and lack of access controls.
On the other hand, parts of the program that are necessarily stateful are best handled via OO interfaces, as this allows us to limit the number of possible state transitions. Access to the stateful variables themselves is restricted (as private), and state transitions are allowed only via a limited number of (public) methods.
Management of finite resources (file handles, memory etc...) is a simple stateful part of the program, and is handled neatly by RIAA.
On the other hand, parts of the program that are necessarily stateful are best handled via OO interfaces, as this allows us to limit the number of possible state transitions. Access to the stateful variables themselves is restricted (as private), and state transitions are allowed only via a limited number of (public) methods.
Management of finite resources (file handles, memory etc...) is a simple stateful part of the program, and is handled neatly by RIAA.
Tuesday, 28 May 2013
Types, Tensions and Risk.
The dichotomy between strongly typed languages (such as C) & dynamic scripting languages (such as Python) has already been debated & explored ad nauseam. This contribution, akin to flogging a horse long since dead and buried, will be of little interest to anybody, but it does help me clarify my own position on the matter, so here is my tuppence worth anyway:
Strong typing is supposed to buy you (the developer) a degree of protection from certain types of error. However, that protection comes at a significant cost, and really only provides a worthwhile benefit in a limited set of circumstances.
First of all you do not really reap the benefits of it *just* by using a strongly typed language. You really have to go out of your way to make use of the type system to get the benefits. You have to define your own types so that the system can enforce the non-convertibility of mutually incompatible data types. You also loose a helluva lot of flexibility, because every data item must be specifically typed.
Indeed, developers working with strongly typed languages waste a lot of time writing "generic" frameworks that do nothing else other than circumvent the type system; simultaneously losing the benefit of strong typing whilst also adding to the complexity and cognitive burden of others working on the same system.
Languages with weaker type constraints gain a *lot* of productivity simply by avoiding this whole "let's write something generic" song-and-dance routine. It also focuses the mind on alternative approaches to QA & testing. However, this additional flexibility & productivity often results in a *lot* more code being written (which must be maintained), a tendency which can quickly become self-defeating. (Sometimes what looks initially like exceptional productivity is just exceptional waste in disguise).
Strong typing, used properly, *does* provide value, under certain circumstances, in forcing decisions about data types to be made up-front, rather than deferred to a later stage. It also helps to document interfaces and to make them more specific.
It seems to me that weak typing & the generic interfaces that we end up with as a result result are most useful when we have to defer decisions about the specific data types travelling through those interfaces until later in the development process. Strong typing; leading to the development of specific interfaces are most useful when we are able to specify exactly what data types will be passing through our interfaces early on in the development process.
I.e. strong typing is more congruent with a predictive approach to risk-management, whereas weak typing is more congruent with a reactive / agile approach to risk-management. Of course, this is not a hard-and-fast rule, and successful adoption of either risk-management approach requires high levels of discipline.
Anyway, it is interesting to note the interplay between detailed technical decisions (Strong vs Weakly typed programming languages) and broader business decisions (Approach to risk management & decision scheduling).
Strong typing is supposed to buy you (the developer) a degree of protection from certain types of error. However, that protection comes at a significant cost, and really only provides a worthwhile benefit in a limited set of circumstances.
First of all you do not really reap the benefits of it *just* by using a strongly typed language. You really have to go out of your way to make use of the type system to get the benefits. You have to define your own types so that the system can enforce the non-convertibility of mutually incompatible data types. You also loose a helluva lot of flexibility, because every data item must be specifically typed.
Indeed, developers working with strongly typed languages waste a lot of time writing "generic" frameworks that do nothing else other than circumvent the type system; simultaneously losing the benefit of strong typing whilst also adding to the complexity and cognitive burden of others working on the same system.
Languages with weaker type constraints gain a *lot* of productivity simply by avoiding this whole "let's write something generic" song-and-dance routine. It also focuses the mind on alternative approaches to QA & testing. However, this additional flexibility & productivity often results in a *lot* more code being written (which must be maintained), a tendency which can quickly become self-defeating. (Sometimes what looks initially like exceptional productivity is just exceptional waste in disguise).
Strong typing, used properly, *does* provide value, under certain circumstances, in forcing decisions about data types to be made up-front, rather than deferred to a later stage. It also helps to document interfaces and to make them more specific.
It seems to me that weak typing & the generic interfaces that we end up with as a result result are most useful when we have to defer decisions about the specific data types travelling through those interfaces until later in the development process. Strong typing; leading to the development of specific interfaces are most useful when we are able to specify exactly what data types will be passing through our interfaces early on in the development process.
I.e. strong typing is more congruent with a predictive approach to risk-management, whereas weak typing is more congruent with a reactive / agile approach to risk-management. Of course, this is not a hard-and-fast rule, and successful adoption of either risk-management approach requires high levels of discipline.
Anyway, it is interesting to note the interplay between detailed technical decisions (Strong vs Weakly typed programming languages) and broader business decisions (Approach to risk management & decision scheduling).
Friday, 10 May 2013
How to engineer an Artificial Intelligence
Intelligence is a defining human characteristic, so it is somewhat doubtful that we would ever be able to pin down what it is precisely enough to make an artificial version of it ... we will always try to define and redefine the term "intelligence" in such a manner that humans remain separated and protected as a separate and distinct class of entities ... our collective ego demands no less.
However, if we put our collective ego to one side, we can surmise that intelligent behaviour consists of some general purpose learning and behaviour-directing mechanisms sitting on top of a whole heap of special-purpose sensorimotor mechanisms.
Engineering an artificial system to emulate some or all of that behaviour is a monumental engineering task. Irrespective of the fundamental breakthroughs in learning, generalisation and planning that may or may not be required, we are still left with an exceedingly large and complex software engineering challenge.
In my mind, this represents the primary obstacle to the development of a true Artificial Intelligence: Not better science, but better software engineering practices and (particularly) better software engineering tools to help manage, visualise, understand and communicate complicated software systems.
Now, this is an interesting problem, because software engineering at a large scale is much less concerned with hard technical challenges and much more concerned with "soft" human challenges of communication and politics. The same can be said for any engineering challenge where the complexity of the subject matter, combined with it's lack of visibility makes communication and documentation a key technical challenge.
Let us look at the communication challenge first: One thing that is becoming more apparent in our increasingly information-rich environment is that communication is constrained much more by the time-management and motivation-management of the reader than by the availability of the information.
In other words, it is not enough just to make information available, you have to manage the relationship between the reader and the information as well; so that information is not just dropped in front of the reader, but the relationship between the reader and the information is managed to support learning and effective decision making also.
How might one achieve this?
However, if we put our collective ego to one side, we can surmise that intelligent behaviour consists of some general purpose learning and behaviour-directing mechanisms sitting on top of a whole heap of special-purpose sensorimotor mechanisms.
Engineering an artificial system to emulate some or all of that behaviour is a monumental engineering task. Irrespective of the fundamental breakthroughs in learning, generalisation and planning that may or may not be required, we are still left with an exceedingly large and complex software engineering challenge.
In my mind, this represents the primary obstacle to the development of a true Artificial Intelligence: Not better science, but better software engineering practices and (particularly) better software engineering tools to help manage, visualise, understand and communicate complicated software systems.
Now, this is an interesting problem, because software engineering at a large scale is much less concerned with hard technical challenges and much more concerned with "soft" human challenges of communication and politics. The same can be said for any engineering challenge where the complexity of the subject matter, combined with it's lack of visibility makes communication and documentation a key technical challenge.
Let us look at the communication challenge first: One thing that is becoming more apparent in our increasingly information-rich environment is that communication is constrained much more by the time-management and motivation-management of the reader than by the availability of the information.
In other words, it is not enough just to make information available, you have to manage the relationship between the reader and the information as well; so that information is not just dropped in front of the reader, but the relationship between the reader and the information is managed to support learning and effective decision making also.
How might one achieve this?
Sunday, 14 April 2013
Procyclic
The Carnot cycle provides us with a fun little analogy that we can abuse and overextend in any number of ways:-
Energy is turned into work by harnessing and controlling a cycle of heating and cooling; expansion and contraction; increasing and decreasing pressure.
One side without the other provides only transient and worthless motion.
Brought together in opposition; made to alternate; managed and controlled in a reciprocating framework, these transient forces may be tamed and made to perform useful and beneficial work over an extended period of time.
Creative Cycles
In the same way, our innovative capacity does not produce new ideas in a single unbroken stream of creativity, but rather in fits and starts.
Contemplation and intensity must be brought together in alternating opposition:
Contemplation so that the mind may range wide, gathering inputs from disparate sources as one might pick blackberries from a brambly hedgerow.
Intensity so that the mind may delve deeply into the detail, working through the consequences of each decision; picking apart the idea to find it's essence, and turning the idea over and over to find the right terminology the right language, the right conceptual framework within which it may be best expressed
and exploited.
One phase without the other is unproductive. Bring the two together in alternation, and you get productive work. Manage that alternation, and you can start to exploit our real potential for creativity.
Communication Cycles
There has been some discussion about the utilization and design of shared workspaces to maximise spontaneous communication (prompted by Marissa Meyer's edict banning remote working at Yahoo). In my mind, this focuses on only half of the solution: perhaps unsurprisingly, in light of the theme of this piece, I believe that communication and isolation must alternate.
When working in a totally isolated manner, one cannot determine what is needed, one does not know what is important. When sitting in the middle of a bustling open plan office, one often knows what is important, but one lacks the space and calm isolation needed to actually do anything about it. It is noteworthy that many effective organisations mandate specific periods; specific opportunities for interaction, typically involving food or drink, perhaps we could also mandate specific periods for isolation and calm introspection, a "quiet time" each day free from phone calls, emails and other forms of communication.
This may be unnecessary, however, since the ubiquitous headphone-wearing developers sitting in an open-plan-office provides a very flexible compromise, offering opportunities both for focussed work and for collaboration.
Economic Cycles
As with the small, so with the large. "The era of boom and bust economics is over" was a statement that ran not only counter to common sense, but possibly counter to some sort of law of statistical physics.
To believe that we have any real positive control over the economy is a dangerous conceit. A vague and sloppy influence, perhaps, but not control in any recognisable meaning of the word.
The presence of investment cycles for individual businesses as well as individual people is a well understood fact, as is the web of interactions that exists between those economic actors, and (so I would suppose) the presence of effects that promote the synchronisation of these individual cycles into macroeconomic feedback loops.
Why attempt to eliminate these macro-scale cycles by controlling government spending? Why not, instead, as our analogy suggests, control their amplitude and frequency so that the harm that they do is minimised.
Indeed, as Nasim Taleb points out, a small "bust" can fend off a larger and more damaging one. Targeted cyclical and perhaps aperiodic variations in the tax burden might help stress, (or threaten stress) economic actors in a manner that promotes decoupling and desynchronisation, and inhibits catastrophic modes-of-failure, but, as stated above, it would be dangerously arrogant to assume that we really know what is going on, and what the effect of our actions is, or will be.
So we have to exert control, not in a direct and forceful manner, but in a way that tends to the ecosystem, but does not seek to dominate it.
Anyway, I am straying dangerously far from my area of expertise, and my intuition oft runs astray, so let me leave things here before I go too far off the beaten track.
Energy is turned into work by harnessing and controlling a cycle of heating and cooling; expansion and contraction; increasing and decreasing pressure.
One side without the other provides only transient and worthless motion.
Brought together in opposition; made to alternate; managed and controlled in a reciprocating framework, these transient forces may be tamed and made to perform useful and beneficial work over an extended period of time.
Creative Cycles
In the same way, our innovative capacity does not produce new ideas in a single unbroken stream of creativity, but rather in fits and starts.
Contemplation and intensity must be brought together in alternating opposition:
Contemplation so that the mind may range wide, gathering inputs from disparate sources as one might pick blackberries from a brambly hedgerow.
Intensity so that the mind may delve deeply into the detail, working through the consequences of each decision; picking apart the idea to find it's essence, and turning the idea over and over to find the right terminology the right language, the right conceptual framework within which it may be best expressed
and exploited.
One phase without the other is unproductive. Bring the two together in alternation, and you get productive work. Manage that alternation, and you can start to exploit our real potential for creativity.
Communication Cycles
There has been some discussion about the utilization and design of shared workspaces to maximise spontaneous communication (prompted by Marissa Meyer's edict banning remote working at Yahoo). In my mind, this focuses on only half of the solution: perhaps unsurprisingly, in light of the theme of this piece, I believe that communication and isolation must alternate.
When working in a totally isolated manner, one cannot determine what is needed, one does not know what is important. When sitting in the middle of a bustling open plan office, one often knows what is important, but one lacks the space and calm isolation needed to actually do anything about it. It is noteworthy that many effective organisations mandate specific periods; specific opportunities for interaction, typically involving food or drink, perhaps we could also mandate specific periods for isolation and calm introspection, a "quiet time" each day free from phone calls, emails and other forms of communication.
This may be unnecessary, however, since the ubiquitous headphone-wearing developers sitting in an open-plan-office provides a very flexible compromise, offering opportunities both for focussed work and for collaboration.
Economic Cycles
As with the small, so with the large. "The era of boom and bust economics is over" was a statement that ran not only counter to common sense, but possibly counter to some sort of law of statistical physics.
To believe that we have any real positive control over the economy is a dangerous conceit. A vague and sloppy influence, perhaps, but not control in any recognisable meaning of the word.
The presence of investment cycles for individual businesses as well as individual people is a well understood fact, as is the web of interactions that exists between those economic actors, and (so I would suppose) the presence of effects that promote the synchronisation of these individual cycles into macroeconomic feedback loops.
Why attempt to eliminate these macro-scale cycles by controlling government spending? Why not, instead, as our analogy suggests, control their amplitude and frequency so that the harm that they do is minimised.
Indeed, as Nasim Taleb points out, a small "bust" can fend off a larger and more damaging one. Targeted cyclical and perhaps aperiodic variations in the tax burden might help stress, (or threaten stress) economic actors in a manner that promotes decoupling and desynchronisation, and inhibits catastrophic modes-of-failure, but, as stated above, it would be dangerously arrogant to assume that we really know what is going on, and what the effect of our actions is, or will be.
So we have to exert control, not in a direct and forceful manner, but in a way that tends to the ecosystem, but does not seek to dominate it.
Anyway, I am straying dangerously far from my area of expertise, and my intuition oft runs astray, so let me leave things here before I go too far off the beaten track.
Friday, 5 April 2013
Thursday, 28 March 2013
Wednesday, 27 March 2013
On the importance of attention-to-detail in software tools development.
As the Apple example demonstrates, attention to detail, in aggregate, results in a superior product, which enables you to justify charging a premium price.
This truism is particularly apt when it comes to tools, because the affordances and micro-features that one's day-to-day tools offer have a huge impact on the way that we do our day-to-day job. The extent to which we developers follow the path of least resistance is, in many ways, quite sad, but this most human of characteristics places incredible power in the hands of the tool-makers. Best practice is often defined by the tools that we use. As a crude example, witness how Hungarian notation fell out of favour as soon as popular IDEs started to display "type hints" when the mouse hovered over the variable.
Make it slightly easier to do something, or to arrange text in a particular way, and developers will change their behaviour in response.
For many years (2001-2011) I used TextPad on windows, and came to particularly appreciate it's block select mode when managing vertically aligned characters. The shortcut for this feature (Ctrl+Q, B) has been burned irreversibly into my motor cortex. I used this feature not only to manage vertically-aligned assignment statements, but also to quickly add blocks of end-of-line comments, and to support a declarative programming style that relied heavily on in-source tables of parameters & other data.
This declarative approach to software construction had a particularly pleasing synergy with the MATLAB language and the uncomplicated, linear loop-free aesthetic which is made possible by it's impressive collection of array-oriented library functions.
More recently, (2011-2013) I have been using Sublime Text as my editor-of-choice. (As well as a little vim, when I have no choice). Whilst the multiple selection feature of Sublime text is super-duper-awesome in it's own way, I still miss TextPad's block select feature, particularly it's ability to automatically insert whitespace at the end of lines, allowing one to easily reclaim blocks of space to the right of the logic for descriptive commentary. (80 column limits be damned as an onions-in-the-varnish anachronistic throwback).
Anyway, seemingly small things, but still important: What seem like trivial affordances really do shape the way that one thinks and works. For the past year I switched away from Textpad + MATLAB towards Sublime Text & vim + Python, and the combined strictures of PEP8 & PyLint, together with the subtly different affordances of the Sublime Text vs. TextPad steered my development style away from the declarative parameter-driven approach towards a more "traditional" programming style, to the moderate detriment of the "quality" of the applications that I produced.
Despite Python's numerous, obvious and justifiable claims to superiority, there exists an ineffable and emergent quality that language and tools together supply, and a quality that was not (quite) captured in the tools that I was using for the past year.
Another observation: When moving the caret quickly along a line of text using the arrow keys with the "ctrl" key pressed down, the cursor in the MATLAB editor stops very slightly more frequently than other editors, resulting in a slightly "sluggish" feel to the navigation. Paying attention to matters such as this is the equivalent, to a tool-maker, of the auto-maker paying attention to the "thunk" noise that the car door makes when it closes. A feature that operates almost on a subliminal level to create the impression of quality, and, to the tool-user, the feeling of "floating" through the text file making changes, rather than the feeling of wading through a muddy swamp.
Another observation: Little bits of automation to make life easier. When I highlight a word in Sublime Text and press the open-parentheses "(" key, the word is surrounded in parentheses. When I do the same in the MATLAB editor, the word is deleted and replaced by the open parenthesis character.
I am a craftsman. I want to love my tools. MATLAB has elements of greatness, but a large number of flaws as well. Some of these flaws cannot be addressed without making the tool something other than what it is.
Fair enough.
But some can be addressed with passion, attention-to-detail, design sense, and strong, detailed technical leadership.
This truism is particularly apt when it comes to tools, because the affordances and micro-features that one's day-to-day tools offer have a huge impact on the way that we do our day-to-day job. The extent to which we developers follow the path of least resistance is, in many ways, quite sad, but this most human of characteristics places incredible power in the hands of the tool-makers. Best practice is often defined by the tools that we use. As a crude example, witness how Hungarian notation fell out of favour as soon as popular IDEs started to display "type hints" when the mouse hovered over the variable.
Make it slightly easier to do something, or to arrange text in a particular way, and developers will change their behaviour in response.
For many years (2001-2011) I used TextPad on windows, and came to particularly appreciate it's block select mode when managing vertically aligned characters. The shortcut for this feature (Ctrl+Q, B) has been burned irreversibly into my motor cortex. I used this feature not only to manage vertically-aligned assignment statements, but also to quickly add blocks of end-of-line comments, and to support a declarative programming style that relied heavily on in-source tables of parameters & other data.
This declarative approach to software construction had a particularly pleasing synergy with the MATLAB language and the uncomplicated, linear loop-free aesthetic which is made possible by it's impressive collection of array-oriented library functions.
More recently, (2011-2013) I have been using Sublime Text as my editor-of-choice. (As well as a little vim, when I have no choice). Whilst the multiple selection feature of Sublime text is super-duper-awesome in it's own way, I still miss TextPad's block select feature, particularly it's ability to automatically insert whitespace at the end of lines, allowing one to easily reclaim blocks of space to the right of the logic for descriptive commentary. (80 column limits be damned as an onions-in-the-varnish anachronistic throwback).
Anyway, seemingly small things, but still important: What seem like trivial affordances really do shape the way that one thinks and works. For the past year I switched away from Textpad + MATLAB towards Sublime Text & vim + Python, and the combined strictures of PEP8 & PyLint, together with the subtly different affordances of the Sublime Text vs. TextPad steered my development style away from the declarative parameter-driven approach towards a more "traditional" programming style, to the moderate detriment of the "quality" of the applications that I produced.
Despite Python's numerous, obvious and justifiable claims to superiority, there exists an ineffable and emergent quality that language and tools together supply, and a quality that was not (quite) captured in the tools that I was using for the past year.
Another observation: When moving the caret quickly along a line of text using the arrow keys with the "ctrl" key pressed down, the cursor in the MATLAB editor stops very slightly more frequently than other editors, resulting in a slightly "sluggish" feel to the navigation. Paying attention to matters such as this is the equivalent, to a tool-maker, of the auto-maker paying attention to the "thunk" noise that the car door makes when it closes. A feature that operates almost on a subliminal level to create the impression of quality, and, to the tool-user, the feeling of "floating" through the text file making changes, rather than the feeling of wading through a muddy swamp.
Another observation: Little bits of automation to make life easier. When I highlight a word in Sublime Text and press the open-parentheses "(" key, the word is surrounded in parentheses. When I do the same in the MATLAB editor, the word is deleted and replaced by the open parenthesis character.
I am a craftsman. I want to love my tools. MATLAB has elements of greatness, but a large number of flaws as well. Some of these flaws cannot be addressed without making the tool something other than what it is.
Fair enough.
But some can be addressed with passion, attention-to-detail, design sense, and strong, detailed technical leadership.
Confidence and Competence and Information Overload
The internet is a wonderful thing.
Every problem has a thousand solutions: documented; discussed; dissected.
This sumptuous surfeit of suitable solutions seems satisfactory; superb even, save for one small snag:
My effective competence, my ability to work quickly and solve problems, is driven largely by my confidence in my own abilities, my own evaluation of the completeness of my knowledge versus the task at hand.
When I was younger, the arrogance of youth made this easy. I plowed on ahead, ignorant of my own ignorance, I performed, gained accolades, and life was good.
However, faced with ever rising flood-waters issuing from the fire-hose of information that is the internet today, it is all too easy to start comparing what I know with what I could know, which, expressed as a ratio, is always going to be depressingly close to zero. With self-confidence thus undermined, the inevitable consequence of this is for me to loose sight of the fact that I have enough information at hand to get on with it, without worry or concern, and to get dragged down into the rabbit-hole.
This problem is compounded when one tries to pick up a new tool or technology, an activity that (given the pace of technological advance) one needs to do more-or-less continuously.
Now, I love learning, but I cannot resist the temptation to compare my knowledge and level of expertise in the latest tool to my knowledge and level of expertise in the last, and, in the worst-case-scenario one or two weeks worth of experience is never going to compare favourably with six or seven years.
In situations such as this, I have to keep telling myself to HTFU and demilquetoastify my attitude.
Every problem has a thousand solutions: documented; discussed; dissected.
This sumptuous surfeit of suitable solutions seems satisfactory; superb even, save for one small snag:
My effective competence, my ability to work quickly and solve problems, is driven largely by my confidence in my own abilities, my own evaluation of the completeness of my knowledge versus the task at hand.
When I was younger, the arrogance of youth made this easy. I plowed on ahead, ignorant of my own ignorance, I performed, gained accolades, and life was good.
However, faced with ever rising flood-waters issuing from the fire-hose of information that is the internet today, it is all too easy to start comparing what I know with what I could know, which, expressed as a ratio, is always going to be depressingly close to zero. With self-confidence thus undermined, the inevitable consequence of this is for me to loose sight of the fact that I have enough information at hand to get on with it, without worry or concern, and to get dragged down into the rabbit-hole.
This problem is compounded when one tries to pick up a new tool or technology, an activity that (given the pace of technological advance) one needs to do more-or-less continuously.
Now, I love learning, but I cannot resist the temptation to compare my knowledge and level of expertise in the latest tool to my knowledge and level of expertise in the last, and, in the worst-case-scenario one or two weeks worth of experience is never going to compare favourably with six or seven years.
In situations such as this, I have to keep telling myself to HTFU and demilquetoastify my attitude.
Tuesday, 19 February 2013
Batteries Included
Software developers seem to be significantly more productive today than they were even a few years ago. What has happened? Have techniques improved? Have skill levels increased? Are best-practices being more closely adhered to?
One cannot easily isolate any one factor as bearing sole responsibility for the performance improvements, but I suspect that a great deal of our gratitude should be directed towards improvements in the availability, quality and performance of software libraries, together with growing communities around those libraries, promoting education and adoption in the wider engineering community.
After all, the one thing that makes software development so different from other disciplines are the tremendous cost savings available through reuse. The less logic your organization is responsible for, the less the expenditure on development, documentation and maintenance. The main expense incurred when using tools and libraries is in education and training ... this is perhaps not something that we as an industry have fully adjusted to:- neither in the way that we (as professionals) specialize; nor in the way that we (as organizations) recruit new talent or nurture existing talent.
In fact, the notion of talent as something that is exclusively innate and inimitable is a dangerous one. When one makes an experienced hire, the commodity that one is purchasing is (primarily) knowledge, rather than the commission of work/effort to be expended. The knowledge that is required to not have to do the work, to be more precise. Perhaps we need to introduce new work and employment patterns beyond the traditional notions of contractor and perm employee?
Subscribe to:
Posts (Atom)