Cracking the Nut - Open Sourcing from a Single Source Tree

Why a single source tree ?

At a previous employer the code started out with the main code base in a single checkout with only infrastructure components of the code in their own repositories. As the code base grew the build time gradually become painful. To resolve this developers began to pull out components into their own repositories. This initially solved the problem of slow build times and improved productivity for the developers working on a single component. However, it soon became apparent that making changes to components that had many internal dependencies was particularly cumbersome. It also made it difficult to make changes across multiple repositories.

This was solved by moving everything back into a single source tree. What about the slow build times? The short answer is Buck - a build tool from Facebook which is inspired by Google's internal build tool called Blaze, now open sourced as Bazel. In summary these build tools allow you to easily modularize your repository and only build and test modules that have changed and any of their dependencies. This allows small code changes to be built very quickly while still making changes across the entire organisation easy and atomic.

Managing Open Source components

Our own open source components were treated as external dependencies and therefore their source code was kept in their own repositories. This meant changes to an open source component required a commit to the external repository, an artifact to be built and then downloaded and committed to the internal repository. This was not as simple as changes to any other code in our single source tree.

We didn't want this problem at TransFICC as we want to open source as many components as possible without any associated overhead. A change to an open source component should be no different from a change to any internal component.

To achieve this we view our internal single source tree as the source of truth, which is a Git repository that contains all our open source modules. We maintain our open source modules within the single source tree and make use of Git Subrepo to clone the modules subdirectory to a public Git repository. We then have a Job in our Continuous Delivery pipeline that automatically pushes changes that pass the build to the public source repository. See image below.



Comments

Post a Comment

Popular posts from this blog

Obtaining the time in Java with nano-second precision

Image
At TransFICC for MiFID compliance (as explained in my previous blog) we need to obtain the time of an event within 100 microsecond accuracy with respect to UTC. To achieve this in Java we first need to be able to obtain the time from the OS with better than millisecond precision. We can then make use of Precision Time Protocol (PTP) to ensure the system clock is synchronised to UTC with sufficient accuracy. Take a look at the excellent posts from Luke Bigum on how to achieve this . As of Java version 8 it is not possible to directly obtain the time with higher than millisecond precision. Java 8 System.currentTimeMillis() will return the time since the Unix epoch in milliseconds. Higher precision time is possible on some underlying systems such as Linux. On Linux the code that the OpenJDK JVM executes is: The full source can be found  here . The code makes the function call gettimeofday(). This returns the time since the Linux epoch in micro seconds however it is rounded to th
Read more

Persisting JMH Results for Regression Detection

JMH is a tool to measure Java code that is executed within  micro-second time scales. You may ask the question why do I care about the performance of code that by definition is executed in a very short period of time? It is important to only optimise code that will have a significant effect on the overall application. Sometimes a small portion of the your application's code is executed many times or your latency requirements of your application is in the micro-second time scale. Once your application is performing as required it can be difficult to maintain that performance whilst making code changes. An automated collection of micro-benchmarks running on key components of your application is an invaluable tool in maintaining the required performance. As a starting point a history of your JMH results needs to be stored in order to analyse trends in benchmark results. Influx JMH reporter is an open source tool we have written that parses the JMH JSON results file from a ben
Read more

Time, Where does it come from?

Image
This is part of a two part series on time measurement. Part 2 will explain how to obtain accurate time measurement in Java. This first part explains the general concept of time measurement. How do we measure time? With a clock of course. So how do we build a clock? The first thing we need is to observe something that occurs at a constant rate. Humans have built clocks based on observations of changes in their environment that appeared to occur at a constant rate. Sundial Some of these earliest clocks were Sundials. The shadow cast by the Sundial moves based on the rate of rotation of the earth on its axis. This observation of the Earth's rotation is what Greenwich Mean Time was based on. At the Royal Observatory in Greenwich astronomers would observe the sun and other stars in order to calibrate their clocks. In principal on average the sun would appear directly above in the sky at 12h00.00. This is why its called Mean time. The current
Read more