Git submodules can now track branches

As of version 1.8.2, Git submodules can now track branches instead of specific commits.  This is good news as in many cases, this is exactly the behavior we want.  However, Git submodules are still not as flexible as Google repo, but since submodules are built into Git, the submodule command is a good solution in many cases.

The “git submodule update –remote” command is the key to tracking branches with submodules.   The following is from the Git man pages:

This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote tracking branch. The remote used is branch’s remote (branch.<name>.remote), defaulting to origin. The remote branch used defaults to master, but the branch name may be overridden by setting the submodule.<name>.branch option in either .gitmodules or .git/config (with .git/config taking precedence).

This works for any of the supported update procedures (–checkout, –rebase, etc.). The only change is the source of the target SHA-1. For example, submodule update –remote –merge will merge upstream submodule changes into the submodules, while submodule update –merge will merge superproject gitlink changes into the submodules.

In order to ensure a current tracking branch state, update –remote fetches the submodule’s remote repository before calculating the SHA-1. If you don’t want to fetch, you should use submodule update –remote –no-fetch.

So, if you already have a Git submodule set up, its a simple matter to run git submodule update –remote to update the submodule to the latest master branch.  If you want a different branch, simple edit .gitconfig.

[submodule "meta-bec"]
   path = meta-bec
   url =
   branch = test

Now, if you run git submodule update –remote, Git will update the meta-bec submodule to the latest on the test branch.

This functionality is purely a convenience feature in the submodule update command.  In the actual repository, Git still stores submodules pointed to a particular commit.  The same thing could be accomplished with something like git foreach “git fetch && git checkout test”.  The branch option in .gitmodules functions more as documentation and convenience.  It is very handy to be able to look at .gitmodules and quickly determine that submodule X is tracking branch Y.  Normally, this would have to be documented elsewhere, or figured out in some other way.  Also, for build systems where you want the build to always track the production branches of various projects, update –remote gives you a convenient way to update the build tree.


A quick way to share files from any directory

Did you ever need a quick way to share files from a directory on your computer?  Or perhaps transfer a large file to another person?  With nodejs and express, you can easily set up a temporary web server that allows users to browse and access a list of files in a directory.  For convenience, I created a simple github project that can be cloned into any directory, and then a server started in a matter of seconds.  Yes, you could upload files to a server, or share them with a file sharing service, but if you can expose a random port on your computer to the person who needs the files, then this is faster, and does not require any intermediate steps.  Check out for more information.


OpenEmbedded Source Mirrors

When using OpenEmbedded for product development, there are several reasons you may want to consider setting up a source mirror for your OpenEmbedded build:

  • over time, sources disappear from download locations
  • various servers for source packages may be off-line at the time a build is run
  • some servers may be very slow, which slows down your build
  • occasionally the checksums of a source package will change

For a production build system, you want the build to be reliable and consistent, so this means not depending on 3rd party web sites/servers for a clean build to complete.  Fortunately, OpenEmbedded makes it easy to set up a source mirror with the PREMIRRORS variable. When bitbake tries to fetch source code, it tries PREMIRRORS, the upstream source, and then MIRRORS.  There are several advantages to using a PREMIRROR variable over a MIRROR for your source mirror:

  • your source mirror will be used first, thus slow web sites are not an issue
  • if the checksums of the package change, the build will not fail because its still using the original source package from the mirror.  You are guaranteed to be always using the same source package.

Setting up a source mirror is as simple as copying the contents of your downloads directory to a web server, and then populating the following variable in local.conf:

PREMIRRORS_prepend = "\
     git://.*/.* http://my-server/sources/ \n \
     ftp://.*/.* http://my-server/sources/ \n \
     http://.*/.* http://my-server/sources/ \n \
     https://.*/.* http://my-server/sources/ \n"

The Poky reference manual has more details.


Perisistent device names for USB serial ports

Currently, my workstation has two 8-port USB<->RS232 devices, one dual port USB<->RS422/RS485 adapter, and several single port adapters such as the very useful BUBII.  So with around 20 USB->serial devices, figuring out which /dev/ttyUSBx entry corresponds to which port is not really practical.  However, with udev in Linux, you can easily give static names to each device.  This is especially convenient to do with FTDI devices because each FTDI device has a serial number.  In devices such as the 8-port RS232 adapter, there are 4, 2-port FTDI chips.  I could not find a serial number available in the adapter with a Prolific IC, so I would avoid those until that is sorted.

udevadm can be used to discover the serial number for a FTDI device:

udevadm info --attribute-walk -n /dev/ttyUSB0|grep serial 

After the serial number is known, rules can be created in the /etc/udev/rules.d/99-usb-serial.rules as shown in this example.  Now, serial ports can be accessed using convenient names such as /dev/ttyUSB_beagle.


A quick way to set up an OpenEmbedded feed server

During development with OpenEmbedded (oe-core, meta-oe, meta-angstrom), I often find it useful to set up a feed server so that packages can quickly be installed on the target system without manually copying them over or building a new image.  One way to do this is copy your deploy/ipk directory to an existing web server (perhaps Apache running on your workstation), or configure Apache to point at your OE build directory, etc.  But, it might be more convenient if your build system could directly create an opkg feed with no extra configuration.  In the below example, we use node.js + express.js to create a feed server (basically just a web server that serves up the ipk files).


// nodejs script to start a http server with the feed from this directory
// default is port 4000
var express = require('express')
var app = express()
app.use('/', express.static(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/'))
app.use('/', + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/'))
console.log("feed server started on port 4000")

The function is used to create a directory listing that can be browsed.  With most other web servers, at least this much code is required just for configuration.  With node.js, this is the code to create an entire server from scratch!  This node.js app can be started with a bash function in the environment:

function oe_feed_server()
  cd $OE_BASE
  bitbake package-index
  node tools/feed-server/app.js
  cd -

The package-index target is used to rebuild the index files that list the available packages.

On the target system, the /etc/opkg configuration files must be modified to point to the feed server, and then you can run:

opkg update; opkg install <package>

An exmaple build environement with this integrated is located:

It might be argued that it is just as easy or easier to set up a more conventional web server.  However, the benefit of node.js is that it is a full blown programming environment.  You can quickly extend it to provide a web interface, perhaps add functionality to automatically push updated opkg configuration files to the target system that point to your feed server, etc.  It is much more than just a web server.


OpenEmbedded: configuring openssh to allow a blank password

Noticed the following when browsing around in the OpenEmbedded sources the other day:

ROOTFS_POSTPROCESS_COMMAND += "openssh_allow_empty_password ;"

This allows a blank password for development, which is conveient for running ssh/scp commands to the device.  The above can be placed in an image recipe.

The command modifies the PermitEmptyPasswords config option in /etc/ssh/sshd_config or /etc/default/dropbear.


Setting the root password in an OpenEmbedded image

During development, often a blank root password is used for the embedded Linux target system.  However, when deploying an embedded Linux system, often there is a requirement to set the root password to something non-obvious.  One way to do this is boot the system, and change the password using the passwd command.  Then copy the password hash from the /etc/shadow file into the below line in your image recipe.

sed 's%^root:[^:]*:%root:password_hash_from_etc_shadow:%' \
< ${IMAGE_ROOTFS}/etc/shadow \
> ${IMAGE_ROOTFS}/etc/;\
mv ${IMAGE_ROOTFS}/etc/ ${IMAGE_ROOTFS}/etc/shadow ;"

The ROOTFS_POSTPROCESS_COMMAND is useful for simple modifications like this to the rootfs image.


Running a reboot cycle test shell script with systemd

One of the easiest ways to stress test an embedded Linux system is to continuously reboot the system. Booting is a difficult activity for a Linux system (similar to waking up in the morning). The CPU is maxed out. There are a lot of things happening in parallel. Software is initializing. There is a lot of filesystem activity. Often, if there is an instability in the system (and especially the filesystem), continuously rebooting will expose it.

Now that we use systemd for most new systems, we need to run a simple script X seconds after the system boots to increment the boot count, and reboot. This is the classic domain of a shell script. However, for a simple task like this, it’s a little cumbersome to create a seperate shell script and then call this script from a systemd unit. The following is a way to embed the script directly in a systemd unit.

Put the following in: /lib/systemd/system/cycletest.service

Description=Reboots unit after 30s

ExecStart=/bin/sh -c "\
test -f /cycle-count || echo 0 > /cycle-count;\
echo 'starting cycletest';\
sleep 30;\
expr `cat /cycle-count` + 1 > /cycle-count;\
systemctl reboot;\


To install and start the script:

  • systemctl daemon-reload
  • systemctl enable cycletest.service (enable the service to start on reboot)
  • systemctl start cycletest.service (start the service, should reboot in 30s)

Bitbake has a new way of displaying build status

Now instead of displaying a scrolling log, bitbake will display a simple output that lists which tasks it is working on at the moment:

Currently 4 running tasks (185 of 3093):
0: gmp-native-5.0.5-r0 do_configure (pid 22919)
1: lzo-native-2.06-r1 do_configure (pid 27103)
2: expat-native-2.1.0-r0 do_compile (pid 7463)
3: ncurses-native-5.9-r10.1 do_compile (pid 9820)

This really allows for a clear view of how the parallel threads option (BB_NUMBER_THEADS) in bitbake works.


Mounting a UBIFS partition using systemd

Systemd is becoming the defacto system and service manager for Linux, replacing the SysV init scripts.  The Angstrom distribution has supported systemd for some time now. Recently, I needed to mount a UBIFS filesystem in one of my projects.  The main application is being started with systemd, so it seemed like a good fit to also use systemd to mount a data partition needed by the application. Systemd can use entries from /etc/fstab, but one additional wrinkle in this system is that I also wanted to run the UBI attach in a controlled way. This can be done with a kernel command line argument, but there are times in this system where we will want to format the data partition, so this requires a detach/attach operation.

The resulting systemd units are:


Description=Mount data partition


Description=Attach data ubi partition

ExecStart=/usr/sbin/ubiattach /dev/ubi_ctrl -m 6
ExecStop=/usr/sbin/ubidetach /dev/ubi_ctrl -m 6

add the following to my-application.service


The only real problem I ran into is the “After” statement in the data.mount unit.  It turns out that the mount will run before the attach operation is finished unless this is included.

The RemainAfterExit seems to be required so that ExecStop can be run when the unit is stopped.

(There may be a better ways to do all this, so comments are welcome!)

One of the benefits of systemd is that everything is very controlled.  If the dependencies are specified properly, there are no race conditions.  Additionally, if you need to manage units (services, mounts, etc) from a program, it is much easier to check the states as everything is very consistent.  For example, if you want to query the state of a unit, the systemctl show <unit>command will return easily parsed output (as shown below):

After=data-attach.service systemd-journald.socket -.mount
Description=Mount data partition
InactiveExitTimestamp=Thu, 01 Jan 1970 00:00:17 +0000

If the data-attach service is stopped, systemd automatically unmounts the data partition first — very nice!

Hopefully this example illustrates how to do simple tasks in systemd.  It appears that instead of having a complex script or program to initialize a system, the systemd “way” is to create a number of small units that get connected with dependencies.  This seems like a much more controlled and flexible approach.


A Review of Graphical Application Solutions for Embedded Linux Systems

One of the decisions we face when building Embedded Linux systems is what components to use. With Open Source software, there is often more than one good option. Graphical libraries are no exception. In this article, we’ll examine GTK+, Qt, EFL, Android, and HTML/Javascript.

There are many factors that go into a choice like this, but some of them are:

  • Does the application need to run on Windows or MacOS?
  • Does the GUI need to be viewed remotely over a network?
  • Are dynamic effects (think iPhone) desired?
  • Does the application need to run on low end CPU’s (ones without a GPU)?

Putting some effort into selecting the right GUI technology is important for the following reasons:

  • Most of the development effort for the product will likely be in the graphical application, so it makes sense to maximize the productivity of the application developers.  Typically there might be 3-10 application developers on a project for every system software developer.
  • You want a technology that will scale with new revisions and improvements of the product.
  • You want a technology that will be supported and improved long term.

With Open Source software, things are always changing.  Leading technologies change.  This can be illustrated by following Intel’s support for their open source software platforms.  While some of this may be driven by politics, we can also see clear technical and licensing reasons why these shifts were made.

  • Moblin (GTK+), became public around 2007, merged into MeeGo in 2010
  • MeeGo (Qt), announced in 2010-02, cancelled 2011-09
  • Tizen (HTML/Javascript, EFL), announced in 2011-09

We can see similar shifts in other companies:

  • Nokia started with GTK+ as the GUI technology for their tablet computers, and then shifted to Qt
  • Samsung has been using GTK+ on top of DirectFB, and now is moving toward EFL/HTML/Javascript for some phones
  • Many phone manufactures are producing Android products
  • Palm moved from a proprietary GUI to HTML in webOS


GTK+ is part of the GNOME desktop project and is perhaps the most used graphical library for desktop Linux applications, and in the past has been very popular in Embedded systems.  Nokia has invested heavily in GTK+ in the past with its early tablet products (N770, N800, N900, etc).  However, with the advent of the iPhone and faster processors with GPUs, everything has changed.  The standard is now dynamic GUI’s with sliding effects, etc.  The Clutter project is a library that can be used to build dynamic GUIs and fits in with the GNOME stack.  GTK+ supports Windows and MacOS, but probably not as well as Qt.


Qt is a very mature project that is also extensively used for desktop projects, and recently is being used on some of Nokia’s phones.  Qt was originally developed by the Norwegian company Trolltech. Originally, Qt was only offered with either a proprietary license or the GPL license.  This meant if you wanted to write a proprietary application using Qt, you had to purchase Trolltech’s commercial license. This factor alone probably made GTK+ a much more popular solution for many years for Embedded Linux GUI’s, including most cell phone stacks.  In 2008, Trolltech was acquired by Nokia, and shortly after that, Qt was offered under the LPGL license, giving it the same license as GTK+.

One of Qt’s compelling features introduced in Qt 4.7 is its QML (or Qt Quick) technology.  This allows you to write declarative GUI’s in a Javascript like syntax with many automatic bindings.  There is good support for dynamic operations like sliding effects, and the performance is reasonable, even on low-end systems without a GPU.

In the future, Qt 5.0 will require OpenGL, and hence be relegated to high end ARM CPU’s with a GPU, or desktop systems.

Qt’s cross platform support is excellent, and provides good native support for Linux, MacOS, and Windows.

Recently, Nokia has made efforts to set up Qt as more of a community project, instead of retaining exclusive control over it.


EFL (Enlightenment Foundation Libraries) is a project that started out as the Enlightenment window manager, and grew into a set of general purpose libraries.  It claims to be to be more efficient than GTK+ and Qt, and work with or without hardware acceleration.  Recently, EFL seems to have garnered the commercial interest of Samsung, Intel, and others involved in the Tizen project.  According to a presentation by Carsten Haitzler (one of EFL’s founders and developers), Samsung was using GTK+ and DirectFB, but switched to EFL after seeing the performance.  Perhaps the most compelling story for EFL is the high performance over a range of hardware capabilities (from simple phones with low end processors, to high end smart-phones running OpenGL).  Parts of EFL are also used in the commercial GUI FancyPants.


Android is an interesting GUI solution, especially as many developers have experience working on Android applications.  Android now seems to be used in many applications where Windows CE was used in the past, probably due to its polished application development tool-set.


Application development with HTML and Javascript is one of the more interesting developments because many embedded systems are headless (don’t have a local display). Couple this with the fact that many users now have smartphones or tablets readily available, and it may make sense to simply use an external device for displaying the UI.  There is often a requirement for accessing the UI of a device remotely, and in this case, HTML/Javascript works very well.  If the UI needs to be displayed locally, then a fairly powerful CPU is required (an ARM Cortex-A8, etc) to run a modern web browser.  If there is no physical display on the device, and the UI is accessed remotely on a computer or mobile device, then a less powerful CPU is required because the embedded device does not actually have to do any of the rendering.  HTML/Javascript also has the benefit that it is a very popular technology, thus there are many experienced developers.


Each of the above technologies has benefits and drawbacks.  Understanding your project’s requirements, and what each solution offers is key to making the best decision.


A Linux Kernel Tracing Tutorial

The Linux kernel has a fairly extensive tracing infrastructure that is quite useful for debugging.  There are a number of things you can do with tracing, but the focus of this article will be the traditional printk type debugging we often end up doing to trace initialization issues with a driver.  The following links provide additional information on the linux kernel tracing infrastructure:

In this example, I am working on a new audio driver.  The typical experience with a new driver is that you install it and nothing happens because something is not registered correctly with the Linux driver model.  So, the first thing I do is start with with the platform_device_add() function in my drivers init function.  To observe the kernel activity around the kernel platform code, I can do the following:

cd /sys/kernel/debug/tracing/
echo 0 > tracing_on (keep trace from filling up until we set filter)
echo function_graph > current_tracer
echo platform* > set_ftrace_filter
echo 1 > tracing_on
cat trace_pipe (leave running in a different shell)
<insmod my driver>

After executing the above, we see the following.  For this example, trace_pipe is preferred because the trace is then emptied and only new information is shown.

0) + 30.518 us   |  platform_device_alloc();
0)               |  platform_device_add() {
0)   0.000 us    |    platform_uevent();
0) + 30.518 us   |  platform_uevent();
0)   0.000 us    |  platform_uevent();
0) + 30.518 us   |    platform_match();
0) + 30.518 us   |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();


0) + 30.518 us   |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0) ! 3936.767 us |  }
0) + 30.518 us   |  platform_uevent();
0) + 30.518 us   |  platform_device_alloc();

From the above, I can conclude that the platform_match() is not succeeding, because I would expect some more activity.  At this point I chose to add a printk:

diff --git a/drivers/base/platform.c b/drivers/base/platform.c
index 7a24895..f9ce0c7 100644
--- a/drivers/base/platform.c
+++ b/drivers/base/platform.c
@@ -662,6 +662,8 @@ static int platform_match(struct device *dev, struct device_driver *drv)
        struct platform_device *pdev = to_platform_device(dev);
        struct platform_driver *pdrv = to_platform_driver(drv);

+       trace_printk("pdev->name = %s, drv->name = %s", pdev->name, drv->name);
        /* Attempt an OF style match first */
        if (of_driver_match_device(dev, drv))
                return 1;

Now, if I re-run the trace, I see the following:

 0)               |      /* pdev->name = soc_audio, drv->name = davinci_emac */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = soc-audio */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = omap-pcm-audio */
 0)   0.000 us    |    }
 0) ! 4241.943 us |  } /* platform_device_add */

From the above, it looks like we have a simple mismatch between “soc_audio” and “soc-audio.”  Fixing this problem, and re-installing the module, we now have:

 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = soc-audio */
 0)   0.000 us    |    }
 0) + 91.553 us   |    platform_drv_probe();
 0) ! 4241.943 us |  } /* platform_device_add */

Now we can see that the names match, and the probe function is now being called.  At this point, we may want to turn on tracing of some additional functions to try to determine what is happening next.

echo "platform* snd* mydriver*" > set_ftrace_filter

And the result:

 0)               |      /* pdev->name = soc-audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = soc-audio */
 0) + 30.517 us   |    }
 0)               |    platform_drv_probe() {
 0)               |      snd_soc_register_card() {
 0) + 30.518 us   |        snd_soc_instantiate_cards();
 0) ! 17852.78 us |      }
 0) ! 17883.30 us |    }
 0) ! 22125.24 us |  } /* platform_device_add */

With the above additional information, we can continue to learn more about the flow through the kernel.

While all of the above could have been done with printk’s, it would have been more time consuming.  The kernel function tracing capabilities allow us to quickly get a high level view of the flow through the kernel without manually adding a bunch of printk statements.  The kernel tracing features are completely contained in the kernel without requiring additional user space utilities which makes it very convenient to use in embedded systems.  The low overhead is also important in resource constrained embedded systems.


The easy way to get serial terminal in Linux

When doing embedded Linux development, most of us spend out time tethered to a target system with a serial cable, which is used for a serial console.  Minicom is the defacto serial terminal software for Linux.  However, Minicom is a little fussy in that you typically have to set it up for each port you want to use.  This is no big deal, but is generally difficult for new users to understand, and yet another hurdle.  And with 8-port USB->serial adapters, I have a lot of ports to set up.

Just recently, I discovered that screen can be used as a serial terminal program:

screen /dev/ttyUSB0 115200

A few notes on using:

  • to exit screen: Ctrl-a k
  • to write a hardcopy of the screen image: Ctrl-a h
  • to get help: Ctrl-?

All the neat features of screen are two numerous to list here, but one more that I’ll point out is the scrollback/copy feature (activated by Ctrl-a [ ).  This allows you to scroll back and the navigation works much like VI — what could be nicer?


Verizon UML290 and Sprint U600 USB Modems in Embedded Systems

Recently I tested support for the Verizon UML290 and Sprint U600 USB Cellular modems in an embedded Linux system.  Both modems support 3G and 4G networks, but only the 3G modes were tested due to lack of 4G coverage at the testing location.

Fortunately, both modems function very similar to previous modems, so with the drivers available in the Linux kernel, and standard pppd support in OpenEmbedded, they worked fine.

The Verizon UML290 modem provides a challenge in that it must be manually switched between 4G and 3G modes.  Typically this is done automatically by the vzaccess program Verizon supplies with the modem that runs on Windows.  The solution for this system was to manually set the modem to 3G mode as detailed on the following page:

It appears that some embedded systems such as the Cradlepoint routers have implemented automatic 3G/4G switching support for the UML290, so this is no doubt possible with a little effort.

The Sprint U600 modem appears to default to 3G, or automatically switch inside the modem.

The same pppd scripts can be used with both modems:

# /etc/ppp/peers/verizon_um290
user a
password v
connect "/usr/sbin/chat -v -f /etc/ppp/peers/verizon_um290_chat"
# /etc/ppp/peers/verizon_um290_chat
'' 'ATZ'
'OK' 'ATDT#777'

To initiate a connection:

pppd call verizon_um290

With Verizon cellular modems, it appears that port 22 is often blocked, so if you need to access a remote device via ssh, you may need to run ssh on a higher port number.  With 4G networks, it appears that the networking setup may be different in that a public IP address may not be assigned.  From the above page, we find the following text:

This fix will also work for users looking to use their device for remote based applications because it assigns a public facing IP address (3G ONLY). With eHRPD you’re assigned a private IP in either 3G or 4G mode, which has prevents UML290 users from accessing remote applications.

Perhaps the Rev A HDR Service mode will also work in 4G mode, but its seems as cellular networks become more complicated, there will be more issues to deal with in using USB Cellular modems for remote access in embedded systems.

Past articles on USB Cellular modems:


Git and Distributed Development

This is part of an ongoing series of articles on the Git version control system.

The “many repository” paradigm has been partly driven by the distributed development paradigm.  Git solves the problem of multiple developers working in multiple repositories very well.  Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories.  Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure.

As we consider the product development process, we need to consider the life cycle of a product.  Most products live for at least several years, and will go through several software iterations.  If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc.  But, we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size.  The below Figure 2 illustrates how you might organize a typical project.  Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process, and merge the changes with our own modifications.  We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense.

There are many other design flows possible.  Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for a QA processes, maintenance repositories for supporting old releases, etc.

Even at a personal developer level, Git’s distributed capabilities offers many advantages.  Each Git workspace is actually a full Git repository.  This means you can check changes in locally, re-organize your changes, be able to track changes when off-line, etc.  For this reason, many developers are now using git-svn when they need to work with Subversion repositories.  With git-svn, you have all the benefits of using git locally, and yet you can easily synchronize with Subversion repositories.  And this leads us to our next topic: cheap branches (coming soon).