GIT Nested repositories: Composer vs. SubModules vs. Subtree vs.?

Question

I've finally incorporated GitHub and Composer dependency management on my workflow. It's definitely a huge step forward, although I remain very confused about GIT managing the "nested" dependecies.

As I'm using an awesome Wordpress Stack ROOTS/BEDROCK, my simplified directory structure looks like this:

|-- /project
|   |-- /.git                    // git repository for the skeleton/stack of the project
|   |-- composer.json            // list of dependencies, most of them are my own repositories on GitHub
|   |-- /vendor
|   |   |-- /php-dependency-1    // 3rd party dependencies not directly related to Wordpress
|   |-- /web
|   |   |-- /app                 // acts as "wp-admin" folder
|   |   |   |-- /mu-plugins       
|   |   |   |   |-- /SUBREPOSITORY-1    // my own framework feature, public, GitHub
|   |   |   |   |-- /SUBREPOSITORY-2    // my own framework feature, public, GitHub
|   |   |   |-- /plugins
|   |   |   |   |-- /SUBREPOSITORY-3    // my own plugin, public, GitHub
|   |   |   |-- /themes
|   |   |   |   |-- /SUBREPOSITORY-5-PARENT-THEME    // parent theme used on my framework, public, GitHub
|   |   |   |   |-- /SUBREPOSITORY-6-CHILD-THEME     // work for client, private, BitBucket
|   |   |-- /wordpress           // Wordpress CMS
|   |   |   |-- /wp-admin
|   |   |   |-- /wp-includes

"Subrepositories" are defined in my composer.json on the root of the project and are downloaded from GitHub by composer install. So far so good.

But! I expect to tweak my parent-theme and some mu-plugins a lot, I need to be able to push/commit from each of my projects they will be included. As you know, you can't really test wordpress theme without a wordpress installation ...

So ... which way to go? There IS A LOT of posts about this topic and most of them mention SubModules, but if I get the idea of Composer correctly, they are kind of in conflict with each other.

Just use nested .git repositories seem great for my case, altough it does not seem to work - if I try to push/commit nested repo, either "everything is up to date" or I get messages such as Your branch is ahead by 1 commit. So just "nesting it" is a no a go?

Thanks in advance and sorry for a little confused tone of the question, I drowned a little in the topic. :) Any help would be much appreciated.

Answer 1

I have a couple of questions, and in consideration of that, the answer below is quite generic. If you answer my questions, I will gladly update it.

1. Does composer pull the repos on an update? OR reclone the repo? (Is it even doing updates?)

   • If it reclones then using it for updates will risk overwriting your working tree on those subrepos (use it for install ONLY or remove it all together)
   • If it isn't doing updates (or dependency recursion -- see below), then it is adding unecessary complexity to your project (remove it and use one of the options below)

2. Is composer actually doing any dependency management (i.e. recursing to find nested dependencies)? Or is it simply cloning the git projects as subfolders?

   • If it is, then yes, submodules may be innapropriate for your case, as they are an alternative dependency management system, but if your subprojects also manage their dependencies with submodules then doing a git clone --recursive should work for managing them as well

3. Do you want your master project to track new changes to your subprojects?

   • If yes: have a look at option #2: subrepositories
   • otherwise: try option #1: submodules
   • [there is a third option which I will link to, but I haven't used it so can't explain in detail (comments/edits appreciated)]

Option #1: Submodules

You can also manage an individual submodule by cd LOCAL_DIR_NAME_I and using normal git commands

1. Setting up:

git submodule add REMOTE_URI_1 LOCAL_DIR_NAME_1
...
...
git submodule add REMOTE_URI_N LOCAL_DIR_NAME_N
git commit -m "Add submodules..."

2. Cloning (the main project)

   git clone MAIN_URI REPO && cd REPO && git submodule update --init --recursive

   --init will copy the configuration from .gitmodules to .git/config before performing the update (if necessary), and --recursive will do that action recursively in each submodules.

   or

   git clone --recursive MAIN_URI

   --recursive tells git to update and init all submodules on cloning

3. Updating (will preserve unsaved changes)

   • Local copy has no un-pushed changes (updates all submodules by default):

   git submodule update [LOCAL_DIR_NAME_I ... LOCAL_DIR_NAME_J]

   • Local copy has un-pushed changes (updates all submodules by default):

   git submodule update --remote --rebase [LOCAL_DIR_NAME_I ... LOCAL_DIR_NAME_J]

4. Publishing/Pushing

This tries a submodule push first and if successful performs a main project push

git push --recurse-submodules=on-demand

Option #2: Subrepositories

You've said you have problems using this method, but I don't understand what they are. Please elaborate if possible.

(the git book also talks about subrepos, but I can't for the life of me find where, right now; let me know if you find it)

1. Setting up:

NOTE: the master repo will not track changes to subrepo's .git, just to the files themeselves

The slash (/) after the directory name is essential to avoid creating a submodule

git clone REMOTE_URI_1 LOCAL_DIR_NAME_1 && git add LOCAL_DIR_NAME_1/
...
...
git clone REMOTE_URI_N LOCAL_DIR_NAME_N && git add LOCAL_DIR_NAME_N/
git commit -m "Add subrepos..."

If using Composer: (and it is doing the clones for you) you can simply do the adds and commit, but maybe you can configure composer to do this as well.

2. Management

Manage an individual subrep by: `cd LOCAL_DIR_NAME_N' and use normal git commands

Remember that changes to your subrepo files will be tracked by your main repo

The biggest issue here is with cloning (if you want colaborators to be able to work on the subprojects) since your subrepo .git files aren't tracked. Including a file, remote.info in each subrepo that stores the remote should solve this. Then add a script to your main repo that does for each subdirectory cd SUBDIR && git init && cat remote.info | xargs git remote add origin. Depending on what Composer is doing (see questions above) you may want to add a composer update command to this script

Option #3: Subtree Merging

I'm not entirely confident on the subtleties of this method, so I will just link to it

Try this link for a bit of a tutorial

Option #N: Any way you want

Of course you could setup numerous other work flows that in some cases might be simpler. For example you could stick with Composer for dep management and clone your subprojects outside of your main project, creating untracked symlinks in the main project to allow easy access to those files when working on the main project. This could be automated with a script (as could a batch push of all these repos). You could probably even parse composer.json to automatically do this for new (git-based) dependencies.

Note: It seems to me that you don't need to be using Composer at all. If this assumption is incorrect, it is possible that none of the 3 options above will solve your problems.