Thursday, 20 September 2012

Multi-project integration.

This is a response to the question: "How to maintain same code fragments on multiple projects":

The DRY principle states that there should be a single authoritative source of truth. This dictates the use of a single *source* repository for all program logic, with no duplication; files organized to promote sharing and reuse of documents.

The pragmatic requirements of communication in a distributed, multi-team environment suggests that there should be multiple independent repositories, one for each project or collaboration.

This presents a problem, as the requirements (at first glance) would seem to be contradictory. However, it is possible to use scripting & automation to smooth over this awkward fact, and to create two separate but consistent interfaces to the body of work.

The one unified repository acts as the single authoritative source of truth regarding all of the projects. The build process for each project copies all the files used by that project (and only those files) into an intermediate location, then build from that intermediate location. (Unison or some similar tool can be used to move deltas instead of whole files).

These intermediate locations can be used as local working copies for the set of secondary, derived, or downstream repositories. Post-commit hook scripts on the authoritative repository will update all of the intermediate locations, and, for each in turn, check if it has changed, and make the same commit to the corresponding secondary repository if a change is detected.

Likewise, post-commit hook scripts on the intermediate repositories can be implemented to update files in the central repository, enabling inbound changes from collaborators to propagate to the authoritative repository.

This way, the multiple secondary repositories are kept in sync with the single authoritative source repository, and the build process ensures that the secondary repositories contain all of the (possibly shared) documents and other files that are needed for the build to succeed. Finally, and most importantly, the development and build process ensures that files may be edited in one place and one place only, and that all copies are kept consistent and up to date.

This solution is tool agnostic, it should be possible to use Svn, Hg or Git as repositories, and bash or batch to implement the synchronizing logic. Something like Unison may be useful to synch the downstream repositories with the authoritative repository,