As we work with larger and more complex systems (i.e. Linux), more and more of our time is spent on integration and pulling different pieces together. We often need to debug or understand code we did not write — especially in build systems. To work effectively in this scenario you must be able to quickly search through a lot of source code. Therefore, we are always looking for ways to make this more efficient.
One pattern that is very useful is separating source and build files into two separate directories at the top level of the project. There are several reasons for this:
- It is easy to grep the source (or today we ripgrep) if it is not littered with transient build data.
- Its easy to reset the build by simply deleting the build directory.
This is especially important for large builds like OpenEmbedded, where a build directory can be on the order of 40GiB. If your grep tool has to process 40GiB of data every time it runs, it will be very slow.
This separation should occur at the top level of the project. Ideally you want something like this for an OpenEmbedded build:
├── build/ │ ├── sstate-cache/ │ └── tmp/ ├── downloads/ ├── sources/ │ ├── bitbake/ │ ├── meta-bec/ │ ├── meta-browser/ │ ├── meta-freescale/ │ ├── meta-freescale-3rdparty/ │ ├── meta-freescale-distro/ │ ├── meta-openembedded/ │ ├── meta-qt5/ │ ├── meta-variscite-fslc/ │ ├── meta-myproject/ │ ├── meta-mycompany/ │ └── openembedded-core/ ├── conf/ └── etc ...
The sources directory contains OpenEmbedded metadata (core, 3rd party, and your own custom meta-myproject layers). The downloads directory might contain 3rd party sources or tar balls that are downloaded by the build tool. And the build directory contains transient build data that can be deleted and re-created at any time. This configuration is used in the Yoe distribution. It should be noted that in this example, the desire is to cleanly separate OE metadata (source) from build data. Source code for individual packages is considered build data from an OE perspective. Most of the development work in this case is creating/customizing OE recipes, images, etc, thus the need is to quickly grep the various OE meta layers.
This idea can be contrasted with the Poky reference distribution directory layout:
├── bitbake ├── build │ ├── cache │ ├── conf │ ├── downloads │ ├── sstate-cache │ └── tmp ├── documentation ├── meta ├── meta-poky ├── meta-selftest ├── meta-skeleton ├── meta-yocto-bsp └── scripts
In this case, there are multiple meta directories in the top level. If you want to search all of the OE meta data, you need to instruct your search tool to exclude the build directory. Additionally, the build directory is dynamically generated, which includes the conf directory. If you want to completely reset your build, you need to clean several directories in the build directory. For a production build, you need some of the conf files to be in revision control; therefore, they are more difficult to manage when they are dynamically generated, or copied from another location. Why not move conf to the top and use it there?
Most decisions are tradeoffs, so there may be good reasons for the Poky directory structure. For a reference (demo) distro, simplicity and ease of setup may be valued over reproducibility. For production builds (builds used in products), rock solid reproducability is important. To get that, you need:
- all source and configuration is locked down in version control
- a build should be a minimal number of steps: checkout source, a very simple setup/configuration step, and then a build command. There should be no complex scripts that copy files around that may be modified later, or manual steps.
- the output is ready to program into the target. There are no manual post processing steps.
- keep things as simple as possible.
The separation of source and build can also be used/observed with many other build systems. With CMake, if you have a project directory named myproject, you can do something like:
mkdir myproject-build cd myproject-build cmake ../myproject make
Qt Creator also sets up a separate build directory by default.
With the linux kernel, you can specify a separate build directory with the ‘O’ option:
cd linux mkdir ../linux-build make O=../linux-build
Think about directory structure — it can make a difference.