Git commits are duplicated in the same branch after doing a rebase

Question

I understand the scenario presented in Pro Git about The Perils of Rebasing. The author basically tells you how to avoid duplicated commits:

Do not rebase commits that you have pushed to a public repository.

I am going to tell you my particular situation because I think it does not exactly fit the Pro Git scenario and I still end up with duplicated commits.

Let's say I have two remote branches with their local counterparts:

origin/master    origin/dev
|                |
master           dev

All four branches contains the same commits and I am going to start development in dev:

origin/master : C1 C2 C3 C4
master        : C1 C2 C3 C4

origin/dev    : C1 C2 C3 C4
dev           : C1 C2 C3 C4

After a couple of commits I push the changes to origin/dev:

origin/master : C1 C2 C3 C4
master        : C1 C2 C3 C4

origin/dev    : C1 C2 C3 C4 C5 C6  # (2) git push
dev           : C1 C2 C3 C4 C5 C6  # (1) git checkout dev, git commit

I have to go back to master to make a quick fix:

origin/master : C1 C2 C3 C4 C7  # (2) git push
master        : C1 C2 C3 C4 C7  # (1) git checkout master, git commit

origin/dev    : C1 C2 C3 C4 C5 C6
dev           : C1 C2 C3 C4 C5 C6

And back to dev I rebase the changes to include the quick fix in my actual development:

origin/master : C1 C2 C3 C4 C7
master        : C1 C2 C3 C4 C7

origin/dev    : C1 C2 C3 C4 C5 C6
dev           : C1 C2 C3 C4 C7 C5' C6'  # git checkout dev, git rebase master

If I display the history of commits with GitX/gitk I notice that origin/dev now contains two identical commits C5' and C6' which are different to Git. Now if I push the changes to origin/dev this is the result:

origin/master : C1 C2 C3 C4 C7
master        : C1 C2 C3 C4 C7

origin/dev    : C1 C2 C3 C4 C5 C6 C7 C5' C6'  # git push
dev           : C1 C2 C3 C4 C7 C5' C6'

Maybe I don't fully understand the explanation in Pro Git, so I would like to know two things:

1. Why does Git duplicate these commits while rebasing? Is there a particular reason to do that instead of just applying C5 and C6 after C7?
2. How can I avoid that? Would it be wise to do it?

Answer 1

Short answer

You omitted the fact that you ran git push, got the following error, and then proceeded to run git pull:

To [email protected]:username/test1.git
 ! [rejected]        dev -> dev (non-fast-forward)
error: failed to push some refs to '[email protected]:username/test1.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Despite Git trying to be helpful, its 'git pull' advice is most likely not what you want to do.

If you are:

• Working on a "feature branch" or "developer branch" alone, then you can run git push --force to update the remote with your post-rebase commits (as per user4405677's answer).
• Working on a branch with multiple developers at the same time, then you probably should not be using git rebase in the first place. To update dev with changes from master, you should, instead of running git rebase master dev, run git merge master whilst on dev (as per Justin's answer).

A slightly longer explanation

Each commit hash in Git is based on a number of factors, one of which is the hash of the commit that comes before it.

If you reorder commits you will change commit hashes; rebasing (when it does something) will change commit hashes. With that, the result of running git rebase master dev, where dev is out of sync with master, will create new commits (and thus hashes) with the same content as those on dev but with the commits on master inserted before them.

You can end up in a situation like this in multiple ways. Two ways I can think of:

• You could have commits on master that you want to base your dev work on
• You could have commits on dev that have already been pushed to a remote, which you then proceed to change (reword commit messages, reorder commits, squash commits, etc.)

Let's better understand what happened—here is an example:

You have a repository:

2a2e220 (HEAD, master) C5
ab1bda4 C4
3cb46a9 C3
85f59ab C2
4516164 C1
0e783a3 C0

You then proceed to change commits.

git rebase --interactive HEAD~3 # Three commits before where HEAD is pointing

(This is where you'll have to take my word for it: there are a number of ways to change commits in Git. In this example I changed the time of C3, but you be inserting new commits, changing commit messages, reordering commits, squashing commits together, etc.)

ba7688a (HEAD, master) C5
44085d5 C4
961390d C3
85f59ab C2
4516164 C1
0e783a3 C0

This is where it is important to notice that the commit hashes are different. This is expected behaviour since you have changed something (anything) about them. This is okay, BUT:

Trying to push will show you an error (and hint that you should run git pull).

$ git push origin master
To [email protected]:username/test1.git
 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to '[email protected]:username/test1.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

If we run git pull, we see this log:

7df65f2 (HEAD, master) Merge branch 'master' of bitbucket.org:username/test1
ba7688a C5
44085d5 C4
961390d C3
2a2e220 (origin/master) C5
85f59ab C2
ab1bda4 C4
4516164 C1
3cb46a9 C3
0e783a3 C0

Or, shown another way:

And now we have duplicate commits locally. If we were to run git push we would send them up to the server.

To avoid getting to this stage, we could have run git push --force (where we instead ran git pull). This would have sent our commits with the new hashes to the server without issue. To fix the issue at this stage, we can reset back to before we ran git pull:

Look at the reflog (git reflog) to see what the commit hash was before we ran git pull.

070e71d HEAD@{1}: pull: Merge made by the 'recursive' strategy.
ba7688a HEAD@{2}: rebase -i (finish): returning to refs/heads/master
ba7688a HEAD@{3}: rebase -i (pick): C5
44085d5 HEAD@{4}: rebase -i (pick): C4
961390d HEAD@{5}: commit (amend): C3
3cb46a9 HEAD@{6}: cherry-pick: fast-forward
85f59ab HEAD@{7}: rebase -i (start): checkout HEAD~~~
2a2e220 HEAD@{8}: rebase -i (finish): returning to refs/heads/master
2a2e220 HEAD@{9}: rebase -i (start): checkout refs/remotes/origin/master
2a2e220 HEAD@{10}: commit: C5
ab1bda4 HEAD@{11}: commit: C4
3cb46a9 HEAD@{12}: commit: C3
85f59ab HEAD@{13}: commit: C2
4516164 HEAD@{14}: commit: C1
0e783a3 HEAD@{15}: commit (initial): C0

Above we see that ba7688a was the commit we were at before running git pull. With that commit hash in hand we can reset back to that (git reset --hard ba7688a) and then run git push --force.

And we're done.

But wait, I continued to base work off of the duplicated commits

If you somehow didn't notice that the commits were duplicated and proceeded to continue working atop of duplicate commits, you've really made a mess for yourself. The size of the mess is proportional to the number of commits you have atop of the duplicates.

What this looks like:

3b959b4 (HEAD, master) C10
8f84379 C9
0110e93 C8
6c4a525 C7
630e7b4 C6
070e71d (origin/master) Merge branch 'master' of bitbucket.org:username/test1
ba7688a C5
44085d5 C4
961390d C3
2a2e220 C5
85f59ab C2
ab1bda4 C4
4516164 C1
3cb46a9 C3
0e783a3 C0

Or, shown another way:

In this scenario we want to remove the duplicate commits, but keep the commits that we have based on them—we want to keep C6 through C10. As with most things, there are a number of ways to go about this:

Either:

• Create a new branch at the last duplicated commit¹, cherry-pick each commit (C6 through C10 inclusive) onto that new branch, and treat that new branch as canonical.
• Run git rebase --interactive $commit, where $commit is the commit prior to both the duplicated commits². Here we can outright delete the lines for the duplicates.

¹ It doesn't matter which of the two you choose, either ba7688a or 2a2e220 work fine.

² In the example it would be 85f59ab.

TL;DR

Set advice.pushNonFastForward to false:

git config --global advice.pushNonFastForward false