Pauses Café

While maven2 is not yet officially released, some projects have started adopting it for their build, because of the many improvements it brings, in particular better dependency management, increased flexibility and simpler configuration.

Maybe are you considering using it but wonder how to integrate it with your other tools. What about continuous integration?

Maven2 comes with it’s own Continunous Integration tool called continuum, and if you use maven2 projects you should probably try this tool. But what if you already use CruiseControl? As of this writing, CruiseControl doesn’t support Maven2 builders. It probably will some day. Until then, how can you build a m2 project using CruiseControl?

What are our options?

• use a builder that supports generic build scripts. Unfortunately (and strangely) CruiseControl doesn’t yet support that
• wrap our m2 script in a supported builder. E.g. by making ant call m2 using the appropriate Exec task
• trick CruiseControl into believing it is building a known builder (maven1). For that we can define a custom mavenscript that in fact calls m2 transparently

The third option is interesting. After all maven1 and maven2 are similar. For that a little bit of work is required and the following issues must be solved:

• the current maven builder requires 3 attributes: goal, mavenscript and projectfile. Given these attributes the MavenBuilder will trigger the following command:

    yourscript -p yourprojectfile yourgoals
  

• the builder parses the maven output to understand what it going on. So the mavenscript will have to convert the mavn2 output into maven1 like output.

That doesn’t sound too hard. The approach I describe below follow this solution. It is customized to work on Unix and assumes that you are already familiar with m2 and CruiseControl, and that both are properly installed.

First let’s create a new project in your config.xml. Add the bootstrappers, listeners, publishers your need, exactely like if you were to build a maven1 project. The main difference is in the builder definition:

   <schedule>
      <maven goal="ignored"
             mavenscript="${projectdir}/m2wrapper.sh"
             projectfile="${projectdir}/dummy"/>
   </schedule>

(ignore the \” appearing in this code block. They really should be “. It’s wordpress playing tricks with me.)

You can note here that I am using ant style properties in my config.xml. This is a nice feature of the forthcoming CruiseControl 2.2.2. It will make your config.xml much simpler.

Note also that I defined a dummy project file. The MavenBuilder class checks that the file exists so we will have to create it.

  cd checkout/your_project
  touch dummy

Now we only need to create this m2wrapper.sh script.

# note the arguments specified by the maven builder are completely ignored
# this also assumes that m2 is correctly installed
export JAVA_HOME=/usr/local/lib/j2sdk1.4.2
# tee is just used as a convenience to log the original output and parse it at the same time
/usr/local/tools/lib/m2/bin/m2 clean:clean install 2>&1 | tee m2.log | ./parser.sh

The parser script converts the m2 output in something the MavenBuilder will like. The builder searches for lines starting with “BUILD FAILURE” in the output. Maven2 prefixes its output with [INFO] which needs to be stripped out.

  #!/usr/bin/perl
  while(<stdin>)
  {
     if ($_ =~ /^[INFO] (.*)/) { print "$1n"}
     else {  print "$_"; }
  }

(Here also, ignore the \” appearing in this code block. They really should be “. It’s wordpress playing tricks with me.)

The result:

This works today using CruiseControl’s trunk. Is has several drawbacks. It is platform specific. It is highly fragile. If CruiseControl code or configuration changes or if maven2 command line and/or output changes it may stop working. For that I cannot guarantee that this approach will work in the future (hence the reason I didn’t put it in the official CruiseControl wiki). It’s just a very simple hack.

As a better long term solution, and if there’s interest in using m2 with CruiseControl, just ask. Someone will create a new builder that natively supports it.

Idea

Something is needed to ease the running of command line programs in Java. Several issues appear when using the official SDK Runtime#exec() and Process: multiple exec() method to chose from, quotes handling, lack of shell, no timeout management, input/output management…

Ant first solved the problem by creating a Commandline class some years ago and a couple of helper classes to provide more functionality on top of Runtime.exec().

These classes have been copied and sligthty forked in several projects including plexus, maven2, cruisecontrol. To me that clearly denotates its usefulness. So why not putting these classes available in a standalone library?

Note: this is similar to the general idea Vincent Massol had on reusable ant tasks, with a focus on command line and process execution.

Interest?

• Trygve from the plexus/m2 team said he was OK with making this a little bit more uniform.
• Jeffrey Fredrik from CruiseControl agreed as well.
• I have not contacted the ant project so far. I will do soon
• I am willing to work on making this happen.

If no consensus can be reached so that all projects end up using it, I think it could
still be useful to have this as standalone somewhere.

Requirements & features (proposal)

• handles the definition of a command line and its running in a flexible framework
• minimal external dependencies (0 if possible)
• run on the oldest SDK possible
• increased thread safety
• POJO objects where appropriate
• extension friendly
• automatic handling of command execution timeout
• library should not be in the way of clients. E.g. possibility for clients to monitor or kill underlying processes
• test friendly. In particular easy to mock the running of a command line, e.g. replacing Process inputs/outputs with normal streams. Maybe provide some mocks as part of the lib or in a jcli-mock lib.

Statuses:

I’ve merged several Commandline classes and have various design solutions for brainstormings. Now I would like to get feedback from the potentially interested parties and see where we can head.

Speak up!

Do you have any issues regarding:

• merging of the features. Is there something you need or want to add to the existing classes?
• standardization of the API. Should we take the opportunity to clean the classes a little bit? I think yes.
  I’ve done it a little bit given the requirements & features I defined earlier.
• agreement on the implementation (exception handling, logging, dependencies on external tools)
• licenses clash? how do we cope with small changes made by several individuals on different projects with compatible but different BSD licenses? I we can avoid distributing a class under 3 different licenses…
• one issue I have is that some helper classes (StreamPumper etc..) are not that CLI specific.

Removing them makes the package mostly empty though and would create an external dependency
There are also other classes that have been duplicated (e.g. OSEnvironment)
and could be tied to the Command line execution.
Should the cli classes be part of a larger ‘oses’ package?
Or should it just return to ant in a way that makes it possible for us to reuse it without depending on the ant specific classes?

• where to put this library? apache? (commons?) codehaus? sf? And of course which package name should be used?
  I feel the best would be to have it back in some sort of ant reusable classes, but I am not sure it meets the requirements to be in there.

Design ideas:

Update: having had a closer look a the ant classes, I think I should emphasize a little bit more on the existing ant Execute class which captures a lot of important knowledge. Nevertheless I will appreciate comments on what’s below while I go back playing with the code.

CommandLine is intentionally very similar to the ant Commandline class for compatibility reasons.
We could discuss whether we want to make the working directory a propriety of the Command line or of the running of the command line.
CommandLine, Argument and Mark POJO beans. No execute() method in CommandLine.

interface StreamConsumer {
  void consumeLine(String line);
}
class CommandLineExecutor {
   // several methods overriding the main method:
  // asynchroneous execution
  CommandLineExecution execute(CommandLine, int timeout, InputStream,
          StreamConsumer systemOut, StreamConsumer systemErr...);
}

class CommandLineExecution {
  // if we expose the real Process, we have no way to know how the process
  // was really interrupted (internal timeout vs outside interrupt()). In that
  // case better to expose a modified Process delegating to the execution to
  // rapport usage of the destroy() call, but then issues
  boolean wasInterrupted();

  // blocking. Similar to getPrOcess().waitFor(). Do we need to expose the
  // exception? If not, what do we return upon failure?
  int waitFor() [throws InterruptedException];

  // optional - shortcut for getProcess().destroy();
  void destroyProcess();

  // underlying process - is destroyProcess() sufficient?
  // having the Process would let clients more freedom.
  // Maybe don't provide it in the first version.
  [ Process getProcess(); ]

  // immutable version returned?
  CommandLine getCommandLine();
}

Usage:

CommandLine commandLine = new CommandLine();
commandLine.setExecutable("java“);
Argument arg = commandLine.createArgument();
arg.setValue(”–version“);

CommandLineExecution execution
    = CommandLineExecutor.execute(commandLine, 10, ….);
// want to expose your execution?
MBean execution = new CommandLineExecutionController(execution);
….
int exitStatus = execution.waitFor();
if (execution.wasInterrupted())
{
  …
}

For more decoupling, we could remove the dependency between the executor, the execution and commandline

interface Executable {
  Process execute();
  // more methods to describe the command line
  String getCommandLine();
}

Then we can have either

// #1
class CommandLine implements Executable {}

Execution execution = Executor.execute((Executable) CommandLine, 10, ...);

// #2
class CommandLine {
  Executable getExecutable();
}

Execution execution = Executor.execute(commandLine.getExecutable(), 10, ....);

We could also replace the Execution class by a more specialized interface, taking into account the aforementionned ideas:

interface InterruptableProcess {
  boolean wasInterrupted();
  int waitFor();
  void destroyProcess();
  // return a smart Process wrapper to control the destroy() calls?
  Process getProcess();
}

then

InterruptableProcess process = Executor.execute(commandLine, 10, ....);
// or
// InterruptableProcess process
//    = Executor.execute(commandLine.getExecutable(), 10, ....);...
int exitStatus = process.waitFor();
if (process.wasInterrupted())
{
  ...
}

Comments appreciated!

Update 1: I will probably rework a little bit the design proposal based upon what I saw in the latest ant code and some feedback I received.
Update 2: Do not forget to look into the new SDK 1.5 ProcessBuilder class.