• Developers Corner
• Source Control

Some people dream of replacing ToME's CVS repository with something else.

The trick is though that at present, CVS pretty much matches best with ToME's development methodology, and we've invested quite a few man-hours into adding functionality to our CVS installation. So a change actually is difficult to justify, no matter how inferior CVS is to other systems.

Distributed Development

Most of the newer, better source control systems out these days are distributed. That is, they have a global namespace of repositories, and are designed to facilitiate coordination between individual developers holding their own repositories, rather than coordinating multiple developers' access to a single repository.

ToME's development model is highly centralized. There is one canonical repositry: DarkGod's on Pav's server. Any changes and must be synced and applied to that tree, but the traditional method of doing this with a distributed system, is for the lead developer to accept patches via email and merge them from the other developers' repositories. For ToME that method is unacceptable for a few reasons:

• It's a new barrier to entry to ToME development to have to host one's own publicly-accessible source repository.

• DarkGod as a patch merger would present an unacceptable bottleneck because he's not exactly a full-time developer, and sometimes disappears for weeks at a time.

• Flatly, it's just too much work to do this:
  • Commit to own repository
  • Submit change to central repository
  • Wait for commit
  • Merge anyone else's updates made to central repository
  • Resume working

The leading candidates in this category are Bazaar and GNU Arch, in no particular order.

Patch Queue Manager

Fortunately a solution to this problem, of fitting a centralized development process into a distributed source control system, was already solved in the Arch community by the design and creation of a Patch Queue Manager tool. The PQM accepts OpenPGP-signed changesets via email (using procmail) or in a specified patch queue directory (which we'd likely submit to using http posts to a php script), accepting those from developers who are listed in its keychain, and applies them to the central repository.

There is also a PQM available for Bazaar. In fact, it's the same one, that happened to be practical to port because Bazaar's design is strongly influenced by Arch's design.

So if ToME were to switch from CVS to Arch or Bazaar, then we would have to set up and use a PQM; a non-trivial task.

Note, though, that a PQM would provide the ideal location for implementing the key CVS add-ons we've cobbled together for ourselves: the mailing list and the ACLs.

Bazaar

One major downside to Bazaar is that it's not compatble with the python included in Mac OS 10.4, which is version 2.3.5. So installing that means installing a whole new python elsewhere.

Subversion

What about Subversion, you ask? In the view of this author, Subversion doesn't provide enough benefit over CVS for us to make the effort to move. Yes, full-tree commits are an improvement over the per-file stuff we have now in CVS, but that's basically the only benefit we'd get.

In particular, Subversion still has essentially the same merge model as CVS, where a merge is the application of a diff between two versions, rather than the application of a set of changes as its own first-class object, so if "DarkGod fearssss branches" in CVS, then he'll still fear them in Subversion.

Git

Git falls into the same category as Arch and Bazaar, but might not have a PQM available at this time, instead favoring centralized development using ssh. That method would not make our ACLs possible, to my knowledge.

Chatter

AerdanRunestar: I pretty much agree with Neil's assessment regarding SVN. I would prefer bazaar, because I already have experience with it and because I'd already talked this over with DarkGod and he seems to like it as well. (Enough so, in fact, that he asked if there's an CVS to Bazaar conversion utility (there is).) Another consideration is that I don't know how well-written GNU Arch is (since, y'know, it's GNU); as there's already an acceptable alternative, I'd rather not look over GNU Arch's sources to determine whether it's workable or not.

• NeilStevens: GNU Arch is very well written. Tom Lord is a wizard; one of the old MIT AI guys. He was able cleanly and maintainably to write the first version in bash for crying outloud. He only wrote tla, the C version, to improve performance once he no longer needed the maintenance ease of shell scripts versus C.

  AnonymousHero: There's no question that Tom Lord is very skilled. Unfortunately, he also seemed to take the competion from Canonical's Bazaar (then basically just a set of UI improvements for Arch) rather badly and AFAICT completely abandoned revctrl from one day to the next. I feel it is generally a bad idea when the development of something as fundamental as a VCS is so dependent on one person.

AnonymousHero: A few random comments:

• GNU Arch is basically a dead end.
• Bazaar is incredibly easy to work with UI-wise. In fact,

   bzr init; bzr add ; bzr commit -m "This is version X.Y.Z"
   [hackhackhack]
   bzr diff

is my preferred way of generating patches for stuff; even when that stuff is ultimately controlled in some other VCS. Don't be fooled by the lowish version number, by the way. I've been using bazaar for close to a year both professionally and for all my hobby projects and I've never had a single problem with it.

• Git is another truly distributed alternative. UI-wise I've found it to be a bit lacking, though "cogito" improves the UI considerably while remaining full compatibility with "standard" git. Git is also blazingly fast on huge codebases, but I don't think this particular feature really matters wrt. the ToME codebase.

• I know from my own personal experience that I'm much more likely to hack on and contribute to projects when the process is as easy as it is with most distributed VCSs.

NeilStevens: GNU Arch is both maintined in its present version, and apparently being improved for a version 2. Plus, it's 100% free software. In no way that I can tell is it a dead end. And I've used it professionally for over three years and have grown to become comfortable with it; vital when I'm likely to be DG's first line of support regardless of which SCS we use.

AnonymousHero: I went to the GNU Arch website and could not find any information about a version 2... Both Bazaar and Git are also "100% free software", so I don't really think that enters into it. Btw, I also used GNU Arch before trying Bazaar (NG) and frankly there is just no comparison whatsoever. Bazaar is about 1000x easier to use, and as far as I can tell its functionality is a strict superset of Arch. No pointless messing about with registering archives and such nonsense either :). You really should give it a try; it's really easy to get started and it has a very shallow learning curve (as opposed to Arch) and very few gotchas for beginners (again unlike Arch).

NeilStevens: As for Git, isn't that the one written to be a perfect fit for Linux? Wouldn't the problems that apply to Arch and Bazaar apply double for Git? OK, I just added a Git section above.

AnonymousHero: It was indeed written to be a replacement for BitKeeper which Linus and a few other kernel developers had been using up until then to manage the Linux kernel source. It's come a long way from those humble beginnings, however, and for instance X.org is now using git to manage all their source. I don't particularly like Git since the UI is a bit too low-level for my tastes, but that can be helped by using Cogito. What problems are you referring when you say that they will "apply double" wrt. using Arch or Bazaar?

NeilStevens: See above in the Distributed Development section.

AnonymousHero: Oh, I see. Well, the problem of "double merges" doesn't really happen with Bazaar if you just submit bundles instead of plain diffs to the upstream maintainer. They contain plain text which looks almost identical to a regular old 'diff -u', but also contain all the metadata associated with the changes. This means that bzr won't be confused by your own changes when you merge the upstream repository back into your own local repository. I should also note that you don't really have to wait for upstream to merge either. The usual way I organize my work when I have to coordinate with upstream (which actually happens to be CVS!) is simply to have several local branches, something like this:

• ~/wd/NAME.upstream: This branch is always a mirror of CVS. (I regularly do "cvs up; bzr add; bzr commit" to keep it in sync with CVS)
• ~/wd/NAME.integration: This branch is a temporary "integration" branch where all the changes from the 'maintenance' branch and the CVS are merged.
• ~/wd/NAME.featureA: For developing feature A. Usually branched off the 'integration' branch.
• ~/wd/NAME.featureB: For developing feature B. Usually branched off the 'integration' branch.
• ~/wd/NAME.maintenance: This is a maintenance branch where all bugfixes against the upstream "trunk" get applied/tested before being committed to CVS.

With this system of branches I can keep hacking to my heart's content on features, etc. since everything to be merged upstream is safely stowed away in a branch of its own for possible reworking. The ease of making branches and keeping them up to date makes this completely effortless.

The only thing which bzr is missing right now which would make things even easier (fever branches) would be 'cherry picking' so that you could just select particular commits ('out of order') from a branch and apply those to a different branch. I believe someone is working on supporting cherry picking.

Regarding git: I believe that git also works around the "double merge" problem, but I can't remember the exact mechanism -- perhaps they have something similar to bundles, I can't remember. I think the main difference between git and bzr in the above "multiple branches" scenario is that git is more centered around switching between branches "in place", so that you would only have a single local repository (combined with working directory) and would switch between branches by doing

$ git checkout featureA
[hackhackhack]
$ git checkout featureB
[hackhackhack]

all within the same working directory.

You are right, though, that a PQM would make things much easier when you want one canonical branch but multiple comitters. (This is assuming that it's impossible that, say, you and DG both can log into the same machine via sftp. If you could arrange that, then the solution is trivial both with Bazaar and git.)

NeilStevens: If it were just me and DG, yes, this problem would be easy to solve. We have 20 accounts for ToME CVS at the moment, though, with more to come over time. And not all these users are necessarily to have equal rights to commit. I don't think Pav is interested in giving out that many logins to his server.

Cherry picking is absolutely vital right now, too, thanks to our tendency to have a stable branch and a development branch. I know I'd have had some extra work with E-Team (which is maintained in Arch), when DG surprised me with the alpha 12 release, if I couldn't cherry pick the alpha 12 fixes from my mainline, with its feature changes. And ToME would only make that work harder still.

So I think git is pretty well ruled out. It appears though that if bundles can be made to work well and automated, such that we can create and submit with a single command, then have that bundle merged automatically by the server, then Bazaar would definitely edge out Arch.

AnonymousHero: Ok, I didn't realize that there were that many people with commit access. I did a quick Google search and this was all I could find about patch queue-like functionality for Git. No idea how well/if it works, though.

Regarding cherry picking with Bazaar, I found this which shows how one can do it. It's not the most elegant approach, but it seems reasonable.

Regarding bundles: I can't say I've used them very much, but they seem to be working pretty well for the Bazaar developers themselves :). The 'bzr bundle' command produces a fully self-contained bundle on stdout, so it should be a simple matter of saying

bzr bundle ... | gpg --clearsign | /usr/sbin/sendmail pqm@example.com

to get your patch merged (assuming no conflicts).

AnonymousHero: If anyone still cares about this, there's now http://launchpad.net which provides free hosting for Bazaar projects. They also provide lots of other stuff like bug tracking, wiki, etc. but AFAIK that's not really of interest to the ToME hackers. Team branches may be of particular interest. (Dislaimer: I am *not* affiliated with Launchpad or Canonical in any way, I just thought this might be of interest to the ToME hackers.)