Most platforms we work on are very modular: you build components which plug into a larger system. They tend to be isolated into a single directory, and therefore its quite easy to version them. You can either use svn:externals or git submodules to independently track each component while having a fully working system.

Here is an example…

├── ./app
└── ./plugins
    ├── ./plugins/plugin_a
    ├── ./plugins/plugin_b
    └── ./plugins/plugin_c

The Problem

Unfortunately, it’s not a perfect world, and some platforms are not as well architected. Alternatively, they may be architected that way, but you may still want some files above that isolated area. This typically poses a problem. How can we effectively combine several components, each with its own tree of files, into one working copy?

What if we’re stuck with four projects like this – each with no top level “plugin” directory?

├── ./admincp
├── ./includes
│   └── ./includes/functions_something.php
├── ./index.php
└── ./something.php
├── ./admincp
│   └── ./admincp/security.php
├── ./images
│   └── ./images/secure.png
└── ./includes
    ├── ./includes/functions_security.php
    └── ./includes/xml
        └── ./includes/xml/product-security.xml

These two projects have to fit into the core platform’s filesystem tree in order to work. We’ve been dealing with projects made up of a dozen or so smaller projects, and it was a nightmare to manage. Initially, we had one large repository, which was fine, but as soon as you want to start re-using any component it becomes problematic. Symbolic links can work, but we didn’t want to have to maintain that configuration for a growing number of projects, on my computer as well all staff.

The other headache of a large repository (or project) is the number of files you need to work with. The context of a given task becomes much larger, and it’s harder to find the files that are relevant.

Enter Union Filesystems

Our first stab at this was looking for a sync program to handle syncing multiple directories into a master one. That seemed reasonable, but was still not ideal. After more searching, we came across union filesystems. Specifically, AUFS: Another Union Filesystem. Now, there may be other better ones at some point, but that’s beyond the scope of this article…

AUFS is a type of filesystem that allows you to stack a series of directories into a single target. Depending on your development platform, you may not be able to do it. I made the plunge of switching to Ubuntu a while back for this very reason: if you are developing for Linux systems, you should be building them on a Linux system. It’s possible to install for other Linux type systems, but very cautious.

Here is the command…

mount -t aufs -o br=/src1:/src2:/src3 none /path/to/stacked/dir

You can essentially pull in those 3 source directories, stack them at /path/to/stacked/dir, and run your application from that directory. Directories stack from right to left, so anything on the left will take precedence. You can also specify how the stacked directory behaves when modified. This is where it gets very cool.

Let’s say we have 3 plugin directories: plugin1, plugin2, plugin3, and our base system: platform. Let’s say we want to perform a fresh install of platform to instance/.

Here is the command…

mount -t aufs -o br=local:plugins/plugin1:plugins/plugin2:plugins/plugin3/vendor/platform none instance

I’ve shortened the paths for the sake of readability (you should be using full paths), but that’s what the command would look like. I’ve introduced a new folder as well called local which would contain all of our changes on it. We can use this for several things:

  • Versioning our local installation
  • Having a folder only contain system-created files (log files, uploads, etc.)

AUFS creates a bunch of .wh..wh* files in the furthest-left directory. This keeps track of any deleted files. This is also very cool: we can delete files from our live instance, and the filesystem reflects that, but the source files are never actually deleted.

The other handy use-case of this is deploying an instance of a platform for users to screw around on without any harm to the source system itself. If they screw something up, you can restore them easily enough. You can also run a ton of sites from a single set of source files.

For local development, it makes sense to have your VCS ignore the .wh* files and use it for project configuration and other local project artifacts that you want versioned.

Workflow

Being able to split up repositories may offer some changes to your existing workflow. It allows you to version more things than you typically could, which means you may end up with more small repositories.

When developing, I’d load the source files in your editor, and never directly edit the stacked directory. This allows you to work on very small projects, only having to worry about what is needed and relevant to that project.

The only flaw we’ve run into is the mount command doesn’t always pick up new files quick enough. You may find yourself re-running the mount command as part of your workflow. To me that is a easy to tradeoff that I’m happy to take.

Also, be warned that these mounts are not permanent unless you either run them at start-up, or put them in your fstab file.

Rapid Project Development

If you often build projects consisting of a platform and multiple components, you can quickly generate new “builds” (stacked deploys) and mess around with them efficiently.

To be honest, I have no idea how this would perform in a production environment. If you give it a try, let me know how it goes.

Not a Replacement for Goood Architecture

For the primary use case I’ve described, this is sort of an anti-pattern. Versioning and packaging systems handle this problem far better. However, be aware that not all platforms were created with modularity in mind (or they failed), in which case traditional tools do not work very well.

I wouldn’t do this for any of my newer projects (er-platforms), except perhaps as experimentation, and I much prefer the use of git submodules, or even a packaging system such as Composer (more on that later!).

This has already saved us piles of time, so hopefully somebody else finds it useful, or at least interesting.