Pauses Café

I came upon 2 funny blogs today. Or should I say flamy?

• J-spot
• fate, aka If you have nothing bad to say, say nothing.

In particular, I spent some minutes on the Ceki Gülcü is sick entry.

The good Ceki, of log4j’s fame, has started a new project called slf4j. Apart from having a name as unpronouncable as its author’s lastname, this project’s interest seems to not be understood by the blog entry author not by a poor chap who end up being flamed in the comments.

slf4j is a replacement API for commons-logging. commons-logging is a library aimed at library developers that allows client code to not depend on a particular logging toolkit. This is not something developer of standalone applications should really care about (because they will most probably never have to change their logging toolkit), but it matters a lot to those who write libraries because they don’t know in which kind of environment their library will end up running in.

slf4j allows to chose the underlying logging toolkit at deployment time, thanks to some sort of auto-discovery mecanism. But commons-logging also allows for the underlying logging toolkit at deployment time!
So what is the difference? From a pure theorical usage perspective, not much. From a programmer perspective, on the other side, there are some benefits:

• the API is less intrusive
• it doesn’t suffer commons-logging classloader issues

I haven’t had a time to look at the library yet (nor read the full classloader issues document. Dude it’s late), but if all the issues mentionned there are true, it’s worth trying.

BTW, do you use Sun’s Logging framework? I’ve never really switched to it. Log4j is my friend. It works, it’s highly compatible with most of the free software I use, and I personally don’t like the configuration part of the official logging package.

I am particularly frightened when I see core functionality like logging or XML parsing being integrated in the release to an SDK. Those with older SDK versions won’t be able to use it and those with newer SDK versions may hit issues when deploying. It’s really the case with XML libraries (xerces and xalan), which often cause deployment issues when the classpath when one’s classpath is not as it should.

There’s an other entry on the BileBlog, about good/bad design. That’s something I am particularly interested in these days, and will probably address this issue in a forthcoming post. Some food for the brain on that matter with a little entry on Collection vs Iterator by Bruce Eckel.

And while am at it, anyone knew about Safe haus? Seems like a red codehaus to me. Wonder what’s the story behind it…

CruiseControl can sometimes be problematic to setup: it seems like that a a lot of people searching for information on how to set up CruiseControl end up on this blog.

Those interested may also find the following files useful. They are a copy of most of the files required for the moca CruiseControl installation to run, including config.xml, Linux init.d scripts and some ant/maven wrapper scripts. The setup handles multiple projects. As the machine currently runs CruiseControl 2.3.1-moca-dev, the latest config.xml shows an example of plugin-preconfiguration (althought not with nested elements, which is a new feature of 2.3.1). I will try to keep them up to date.

Nevertheless, the first place where to search for information is still the CruiseControl wiki.

Interesting how unrelated things can bind themselves together.

Some days ago, Jason (ThoughtWorks and CruiseControl fame) had some proposal for a newer JavaBean convention. I personally don’t like it much, see more about that below.

Today I read that Tim Bray complains about the naming convention of the Process class which I have been playing a lot with these days, backporting the still unofficial commons-exec (extracted from Ant), into CruiseControl, and now cleaning up the commons-exec code itself, still in the sandbox. Furthermore pursuing on Tim’s site I end up on this older ProcessBuilder entry, where he ends up discussing the strangeness ot the ProcessBuilder API. Well, Tim if you listen, this is exactly what Jason proposed earlier: write setters so that we can chain them together:

ProcessBuilder pb = new ProcessBuilder();
Process p = pb.command("ls").directory("/tmp").start();

I am not convinced, but some may like it. On the positive side it clearly reduces code size. But I feel like not being able to say what a command does from its name forces one to read the API and the doc to know what it does. Exactly what Tim encountered. I can also see some multiple interface issues happening here…

interface A {
  A setter1(String);
}
interface B {
  B setter1(String);
}
class C implements A, B {
  ?? setter1(String);
}

I don’t understand why Sun developers were allowed to use these new coding conventions in the ProcessBuilder class. Mixing code will confuse developers. It already has. For what benefits? If Java had started from the start with this naming convention, then why not? But the aforementionned issues would have to be weighted against.

Finally, Tim if you deal with Process handling, you may want to look at the commons-exec code and give us some feedback?

update: here’s a quick analysis of the differences on how to spawn Processes in SDK < 5.0, SDK >= 5.0 and using commons-exec, along with a proposal to redesign it slightly.

Now that you’ve secured your ssh (*), you may want to be able to access that from the jira svn plugin.

Jira-svn uses the javasvn library under the hood.

• First thing is to create a user that will access the repository. This user. Let’s call it svn-jira.
• Make sure this user can retrieve the code from your svn repository without ssh. E.g. do a svn co file:///your/path/to/svn/repository
• Try ssh svn-jira@localhost. If you get a password prompt, then your ssh server is not secure enough.
• Add this user to the list of allowed users in your sshd_config and restart your ssh daemon
• Create a ssh key for that user. E.g. use ssh-keygen -t dsa -b 2048. I didn’t put a password.
  You should end up with:

  ....
  Your identification has been saved in /home/svn-jira/.ssh/id_dsa.
  Your public key has been saved in /home/svn-jira/.ssh/id_dsa.pub
  ...
  

• Now make sure this key will be accepted:

  cat /home/svn-jira/.ssh/id_dsa.pub >> /home/svn-jira/.ssh/authorized_keys2
  chmod 600 /home/svn-jira/.ssh/authorized_keys2
  

• Test connection. With ssh svn-jira@localhost, you should now login correctly. Check /var/log/auth.log
• Now let’s configure Jira-SVN plugin. We have to pass the appropriate JVM system property so that the private key is used. But the private key has to be in a readable location. In my case, I copied the private key in /etc/tomcat5/svn-jira_id_dsa and modified the tomcat5 start up script in /etc/tomcat5/tomcat5.conf, adding “-Djavasvn.ssh2.key=/etc/tomcat5/svn-jira_id_dsa” to the JAVA_OPTS definition. Make sure the files has the correct permission and restart tomcat5:

  > ls -la svn-jira_id_dsa
  -rw-r-----  1 root tomcat5 1196 Sep 22 13:15 svn-jira_id_dsa
  > /etc/init.d/tomcat5 restart
  

• Check again /var/log/auth.log to see what is going on. You should see something like:

  Sep 22 13:23:49 deesse sshd[3604]: Accepted publickey for svn-jira from ::ffff:192.168.1.9 port 52857 ssh2
  

• If you have any issue add the following lines to jira/WEB-INF/classes/log4j.properties:

  log4j.category.com.atlassian.jira.plugin.ext.subversion = DEBUG, console
  log4j.category.com.atlassian.jira.plugin.ext.subversion.additivity = false
  

• If your issues are indexed, but the subversion tab won’t appear, check your Jira project scheme permissions, they might miss the jira-developpers user associated to the View Version Control permission

You’re done!

Now what I would like to do is restrict the operations the svn-jira can do using its newly access. Just in Case. Anyone knows how to do that? One should usually modify the authorized_keys2 file, and add restrictions in front of the key, but I don’t know what to add for a javasvn/jsch ssh backend.

(Too bad Jsch doesn’t have internal logging. It would help. Strange they don’t have one as they thank openssh for having a -v -v -v command line option.)

(*) shameless link to my personal blog.

This summer I presented quilt (v 0.4.2) and CruiseControl (v 2.3) to various attendants. I now make these presentations available.

Some eye candy:

Quilt:

CruiseControl:

Feel free to comment.

In the past months, I have been using a setup that allows me to automatically do CI of a (project plus a set of patches). The idea comes from my use of Linux, in particular Debian, where packages coming from upstream have to be patched by the distribution to:

• add distribution specific changes (doc, logo, different installation,…)
• backport changes (especially security fixes) to a stable package

My primary reason for creating this setup was to help the CruiseControl project. Not being a committer, I helped the patch review process. I also took time to complete the user submitted patches with documentation and unit tests and sent occasional patches. These patches got sent back to the developer mailing list or to the issue tracker. But they sometimes rot or were lost. So I decided to create a setup that would make me and the other developers be more productive. This setup helped me to quickly add patches to the queue while getting timely notice when something would break. After all, getting timely notice is one of the benefits of CI, why not expand the idea outside of the usage of the official tree?

How does it work

There are 2 parts to the setup: Patch Management and Continuous Integration.
Notes:

• I renamed the project been worked on from cruisecontrol to myproject in order to remove confusion.
• The CruiseControl config extracts are valid with CruiseControl 2.3.

Patch Management

The basic foundation for this work is quilt. Quilt helps maintaining a serie of patches. quilt is a command line tool that support many commands, but in our case we will need only the push and pop commands which apply/remove patches from the queue. Quilt patches are stored in a single directory called patches along with a file series. This file defines the order in which to apply the patches and also allows for comments.

To work, the patches directory needs to be copied into the working directory, and quilt needs to be invoked from within this directory.

Continuous Integration

One also needs a Continuous Integration tool. CruiseControl was to be of course that tool. But this setup could work with any other CI tool, as long as it supports running scripts.

In order to make it work, there will be some needs to bridge those 2 things together. CruiseControl supports ant and ant supports running scripts. That should be sufficient.

overview & directory layout

The CruiseControl directory layout is not very special:

  /home/cruisecontrol/config.xml
                  .../checkout/myproject-patched-trunk
                  .../logs/myproject-patched-trunk
                  .../artifacts/myproject-patched-trunk

Then the local working directory will contain:

  myproject-patched-trunk/build.xml (official ant build file for myproject)
  myproject-patched-trunk/cc_build.xml (build file used by CC to make build and notifications)

We first need to add a new project inside the cruisecontrol config.xml. A cruisecontrol project is configured in a XML block.

    <project name="myproject-patched-trunk">
        ...
    </project>

This block will define modificationsets, builders, and publishers along with some other less important settings.

modificationset

Modificationsets are used to identify changes and trigger builds.

For practical reasons, I decided to put my patch queue on the file system of the CI server, in a directory visible from the web (that allows automatic web publishing). But one could decide to store them in another SCM.

CruiseControl will then have to look for changes in both the remote official cvs repository and in the local filesystem, so we need 2 modificationsets:

    <modificationset requiremodification="true">
      <cvs localworkingcopy="${projectdir}"/>
      <filesystem folder="${ccrundir}/public_html/projects/myproject-trunk/patches"/>
    </modificationset>

builders

To summarize the build needs to be wrapped with patch management operations. We apply the latest version of the patches before building and pop them after. When the project is not being built, the local tree is similar to the last retrieved trunk version. That’s our invariant.

We then need a single builder that will make the following operations:

• pop all patches. Retrieving the latest version of the source code on top of a patched tree can lead to corruption of your patch queue.
• get the latest version of the patches (and associated scripts)
• update the local working directory (in my case using cvs up -Pd)
• apply the patches. If this fails, pop all applied patches
• build your software as usual
• pop all patches.

We add a cc_build.xml script in the root of the local working directory and register it to the project by adding the following builder:

    <schedule interval="120">
      <ant antscript="${default-antscript}"
           buildfile="cc_build.xml"
           target="build"
           antWorkingDir="${projectdir}"/>
    </schedule>

The cc_build.xml contains 2 tasks related to the build. One called quilt and one called build. The build one will do a simple build of the project (possibly delegating the work to the official build.xml). The quilt target calls a shell script to perform the aforementionned patch management tasks. To summarize this shell script does:

  remove_quilt_patches.sh
    # which does a quilt pop -a

  update_quilt_patches.sh
    # sync quilt_patches/patches myproject-patched-trunk/patches
    # sync quilt_patches/scripts myproject-patched-trunk/scripts

  apply_quilt_patches.sh
   # quilt pop -a
   # cvs up -Pd -C
   # quilt push -a

To trigger a build, one just need to update the quilt_patches directory (which happens to be in ~/public_html/projects/…).

publishers

Publishing happens after a build is done, whether it failed or not. Publishing is just a matter of generating some artifacts and sending some notifications (email).

We always send an email, but we publish artifacts only when the build was succesful. Once again we make use of the cc_build.xml ant build file to perform our notifications and distribution tasks.

    <publishers>
      <htmlemail spamwhilebroken="false" reportsuccess="fixes"
                 css="${ccreportingdir}/jsp/webcontent/css/cruisecontrol.css"
                 mailhost="localhost" buildresultsurl="${ccreportingurl}/buildresults/${project.name}"
                 subjectprefix="[Moca CC]" xsldir="${ccreportingdir}/jsp/webcontent/xsl"
                 skipusers="false" returnaddress="jerome.lacoste@gmail.com">
        <always address="jerome.lacoste@gmail.com"/>
        <always address="cruisecontrol-build@lists.sourceforge.net"/>
      </htmlemail>
      <onsuccess>
        <antpublisher antscript="${default-antscript}"
                      buildfile="cc_build.xml"
                      target="publish"
                      antWorkingDir="${projectdir}"/>
      </onsuccess>

The publish target in the cc_build.xml will perform various operations, including:

• create a changelog message. For that I use a summary.sh script that I store in the quilt patches directory to create a summary of the various patches statuses and scope. This script uses the changelog file (that I manually maintain) as well as parses the serie file
• send a mail with the link to the build and the summary
• publish the build and the sources in a directory.

misc

• It’s a good idea to create a different name for the distributed binaries. E.g. the artifacts distribution makes sure the builds are named after the machine that builds them, to differentiate from the official builds. The names would be similar to myproject-2.3.1-dev-moca-build-101.zip
• I created a simple rsync script that I use to publish my patches onto the CI server (in the public_html/ directory). It’s a good idea to trigger this script after making a full build on your local directory.
• The scrip that updates the patches and scripts directory can also be used from the developer machine to retrieve the version of the latest patches and scripts as stored and published on the CI server

Results and files

The resulting setup can be seen on moca.
The builds are distributed here and the latest set of patches and scripts can be browsed.

Here’s the complete set of files for the setup as used on the moca server to build a patched version of the cruisecontrol.

When to use it?

Why one would like to set up a similar setup?

• in each situation where one has a set of patches on top of a moving tree. This can be when maintaining a parrallel unofficial development tree, when maintaining a stable tree of program (like Linux distributions do), when maintaining an inhouse version of an FLOSS program. All that can be achieved by using a set of patches without having the overhead to use a branch in a SCM
• to automatically validate user submitted patches. It would be simple to automatically create a system where developer patches could be automatically queue to be applied on top of the current tree. Patches could be sent by mail or web forms (or even better taken from the Issue Tracking System). Patch would be applied on top of the trunk, a build would be issued and the developper could be timely notified. I heard that the guys at continuum are going to do something like that.

Limitations

• quilt doesn’t support binaries. I haven’t had much the need for this but it could be an issue.
• setup is currently Linux only
• CruiseControl doesn’t detect removed files in the filesystem modificationset
• single patch queue ‘committer’. This works because I am alone to work on this patch queue. More than one worker would probably require one to use a SCM instead of a file system based solution to store the patches
• it’s easier to use quilt than maintain a fork or a separate tree. Allows for easy publishing of the whole setup of patches, which makes it easy for them to be applied on the main trunk
• does not catch when a patch is broken but a second one in the queue hides it
• if the anonymous access to your SCM is delayed (as is Sourceforge CVS), you can have some timing issues.
• CruiseControl has a known issue when it comes to dealing with hanging processes. That happens a lot if you make a tool that works with remote SCM. But I am going to fix that.
• the order you chose on your patch queue, may not be the same as the one taken by the commiters. Reordering patches can be a pain (not supported by quilt)
• the current publication process is not atomic

Tips?

• do not rewrite patches after publishing them. Someone may have grabbed them. Better to make new versions or complementary patches.
• comment your patches in their header

Conclusion

This solution while not perfect has served us well in the past months. I started using quilt some time ago, before making this setup and I highly recommend any open source developer to have a look at it.

For those interested, I also made a presentation on quilt.

JavaZone 2005 was great. I had the pleasure to spend some time after the conference with some great guys. We had a little tour in Oslo on Friday and a couple of meals together. I would then like to thank (in no particular order) Jeff Genender, Vincent Massol, Kito D. Mann, Patrick Lightbody and Trygve Laugstøl. Thanks also to Matt Filios and Ross for the beers at the conference. I will post pictures as soon as I recover them.

Little note: I am pretty sure Vincent had made a great time at the Compuware contest and that he should probably have won the first prize. Next time someone organizes a contest, give receipts to the constestants, otherwise they will feel ripped off. On the other side, I am glad that a guy like Johannes won the TV. He deserves it for all the work he’s doing in the community Well done Johannes!