Working with private repositories

ShiningPanda provides a hosted Continuous Integration service that many of you are using with private repositories hosted on GitHub, BitBucket, launchpad or other such services.

Until now accessing private repositories from ShiningPanda was not so straightforward. It often required for each build environment (Linux and Windows) to:
- log into it from the dashboard,
- generate a RSA key,
- register this key with your project.

And then your Jenkins instance couldn't even poll the private repository because it didn't have access to the key.

Cumbersome!

From now on the process is extremely straightforward. Just go to your dashboard, in the Build Environments section: there you can generate/revoke as many keys as you need! Of course you will still need to register these keys with your favorite source hosting provider.

example of public SSH key as it appears in the Dashboard

These keys are made available:
- to all your build environments (Linux and Windows),
- to your Jenkins instance so that it can pool private repositories.

We hope you will enjoy this new feature!

Python & Java: a unified build process (2/4)

In our previous blog post dedicated to Python and Java, we saw how Maven can orchestrate a unified build process for these two languages.

But most of the time, all artifacts within a build should take the same version. It was not the case in the sample project of our previous blog post, so let's find a way to unify this.

For Java it's easy: JARs and WARs are using the version located in their respective POMs, 0.1 in this case:

  com.shiningpanda
  jsample
  0.1
  jar
  

It would be great if Python projects could also get their versions from the POM. Let's see how to do that.

The POM version has to be propagated when calling setup.py. The easiest way is to set an environment variable with exec-maven-plugin. Modify setuptools/pom.xml in the sample project as follows:

  
  
    
      
        
          org.codehaus.mojo
          exec-maven-plugin
          
            
            
              ${project.version}
            
          
        
      
    
  

Now a VERSION environment variable containing the value of the POM's version tag is available for setup.py. It can be used to generate a __version__ module containing the project's version. The setup.py script of the pysample project would be modified like this:

# Folder containing setup.py
root = os.path.dirname(os.path.abspath(__file__))
# Path to __version__ module
version_file = os.path.join(root, 'pysample', '__version__.py')
# Check if this is a source distribution.
# If not create the __version__ module containing the version
if not os.path.exists(os.path.join(root, 'PKG-INFO')):
    fd = codecs.open(version_file, 'w', 'utf-8')
    fd.write('version = %r\n' % os.getenv('VERSION', '?'))
    fd.close()
# Load version
exec(open(version_file).read())
# Setup
setup(
    name = 'pysample',
    version = version,
    packages = find_packages(),
)

Note that we only generate the __version__ module if the PKG-INFO file does not exist. Indeed, an existing PKG-INFO file means that we're installing a source distribution previously generated by the setup.py sdist command.

Now all our artifacts are getting their version number from their POM. The versions in the POMs are easily handled thanks to the maven-release-plugin, but we will cover this in another blog post.

A Maven convention wants that version numbers are postfixed with a -SNAPSHOT between two releases. Setuptools uses more likely a .dev one, so feel free to process your POM version in your setup.py, for instance with:

os.getenv('VERSION', '?').replace('-SNAPSHOT', '.dev')

It can also be useful to get the revision version from your source code management tool. To do so, use the buildnumber-maven-plugin. Following the same principle, export an environment variable containing the revision that can be used in the setup.py to compute an artifact version (with a 0.1.dev-r1989 or 0.1.dev-rb0c1c6 pattern for example).

In addition to a Hosted Continuous Integration Service, ShiningPanda is also offering build and release management expertise, so if you have questions or if you are stuck with your internal build process do not hesitate to contact our service team!

Windows 7

ShiningPanda provides a Hosted Continuous Integration service that many of you are already using to test their Python projects on GNU/Linux.

From today everybody can start integrating on Windows 7!

The price is set at $0.64 per hour of build.

Windows 7 build environments come with:
- the most popular Python interpreters (CPython, PyPy, Jython and Iron Python);
- MySQL, PostgreSQL and MongoDB.

But also for those who are not doing only Python:
- Sun/Oracle JDK 5, 6 and 7;
- Microsoft Visual C++ (from Vistual Studio 2010 & 2008).

Do not hesitate to let us know what we can improve to make integration on Windows 7 better.

ShiningPanda plugin: IronPython here we go!

With the new Windows 7 build environments on ShiningPanda Hosted Continuous Integration Service, it was time for the ShiningPanda plugin to support IronPython.

Consider it done with the release 0.12 of the plugin!

To use an IronPython installation, just declare its home folder on the Configure System page in the Python section.

Two words on the IronPython installation itself. On Windows it's quite straightforward, select the desired version here, then download and run the recommended MSI installer.

For the versions without installer (1.0 and 1.1), get the binary distribution and extract it somewhere. Note that for these versions, the standard library is not bundled in so you'll have to install a CPython 2.4. Once done, edit the site.py file located in the Lib folder of your IronPython installation and append the following line (modify it according to your CPython 2.4 installation):

# site.py
sys.path.append("C:\Python2.4\Lib")

On Linux (we tested on Debian Squeeze), it's a little bit more complicated. You'll first have to install Mono. The version 2.10.8 (latest at this time) was the only one working with all the IronPython versions, so you'll probably have to download its source and compile it:

cd mono-2.10.8
./configure --prefix=/path/to/mono-2.10.8
make
make install

Then select the desired version here, download the binary distribution (for version 2.6 choose IronPython-2.6.1-Bin-Net20SP1.zip) and extract it somewhere.

The binary distributions does not contain the standard library (except for the 2.7), so you'll have to install the matching CPython:

• IronPython 1.0 and 1.1: CPython 2.4.3,
• IronPython 2.0: CPython 2.5,
• IronPython 2.6: CPython 2.6.

Same as 1.0 and 1.1 Windows versions, edit the site.py file located in the Lib folder of your IronPython installation and append the following line (modify it according the right CPython):

# site.py
sys.path.append("/path/to/cpython-X.X/lib/pythonX.X")

The ipy.exe also need to be launched with the mono executable. As the mono version can't be selected via the ShiningPanda plugin, I would recommend creating an ipy script alongside the ipy.exe (and an ipy64 one alongside ipy64.exe if exists):

#!/bin/sh
# ipy
exec /path/to/mono-2.10.8/bin/mono /other/path/to/ipy.exe "$@"

#!/bin/sh
# ipy64
exec /path/to/mono-2.10.8/bin/mono /other/path/to/ipy64.exe "$@"

I also noticed that IronPython 2.0 was not loading the site.py on startup. To fix this behavior, modify the script as follows:

#!/bin/sh
IRONPYTHONSTARTUP=/other/path/to/Lib/site.py
exec /path/to/mono-2.10.8/bin/mono /other/path/to/ipy.exe "$@"

If you have any questions, do not hesitate to contact our service team!

Python & Java: a unified build process (1/4)

Since I graduated, I always worked at companies dealing with more than one language to build their solution:

• an AWT application deployed via Java Web Start, displaying some Python interpreted DSL outputs,
• the use of GWT with a Django project,
• and of course the integration of Jenkins in ShiningPanda's Python backend for our Hosted Continuous Integration Service dedicated to Python (and soon Java).

The target was always to implement a unified build process for the comprehensive solution, building and packaging it by issuing a single command. Most languages have their own build tools, but as soon as you have more than one language at hand, issues arise.

This article describes how we are currently dealing with both Java and Python at ShiningPanda to build our Java and Python based service.

First of all we needed to find a conductor. With Java involved, Maven seemed the smartest choice. So the challenge was to integrate some classical setuptools based Python projects with Maven.

The general idea is to map some of the phases of the Maven lifecycle with the setuptools commands:

• clean with python setup.py clean,
• compile with python setup.py develop (not mapped with Maven install phase to come before the test phase),
• test with python setup.py test,
• deploy with python setup.py register sdist upload.

The following sample project builds a pysample Python project and a jsample Java project. Sources can be found here, and the project is organized as follows:

Let's have a look on the POMs, the project descriptors telling Maven what to do:

• First the root pom.xml: it defines the plugin versions and delegates to the subfolder POMs,
• Then setuptools/pom.xml: this POM defines all the mappings and will be inherited in all Python projects to avoid mapping duplications,
• Finally jsample/pom.xml: a classical Java POM defining the dependences for the jsample project,
• And pysample/pom.xml: a POM that inherits from setuptools/pom.xml.

As you may have guessed, the core of the solutions is setuptools/pom.xml. It uses the exec-maven-plugin to start Python subprocesses calling setup.py:

  
  
    
      
        
          org.codehaus.mojo
          exec-maven-plugin
          
            python
            ${basedir}
          
          
            
          
        
      
    
  

Then we only have to define the mapping in the executions tag:

  
    setuptools clean
    clean
    
      exec
    
    
      
        setup.py
        clean
      
    
  
  
    setuptools install
    compile
    
      exec
    
    
      
        setup.py
        develop
      
    
  
  
    setuptools test
    test
    
      exec
    
    
      ${maven.test.skip}
      
        setup.py
        test
      
    
  
  
    setuptools deploy
    deploy
    
      exec
    
    
      
        setup.py
        register
        sdist
        upload
      
    
  

Note that we're skipping the Python tests if the classical -Dmaven.test.skip=true property is set on command line.

Then pysample/pom.xml is quite straightforward (note the parent tag defining the inheritance):

  4.0.0
  
    com.shiningpanda
    setuptools
    0.1-SNAPSHOT
    ../setuptools/pom.xml
  
  pysample
  pom
  
    
      
        org.codehaus.mojo
        exec-maven-plugin
      
    
  

And so is the root pom.xml:

  
  
    
      
        
        
          org.codehaus.mojo
          exec-maven-plugin
          1.2
        
      
    
  
  
  
    setuptools
    pysample
    jsample
  

That's it. Now issue the following command in the root folder to build your full solution:

mvn clean install
# or to deploy (see our upcoming blog post)
mvn clean deploy

In the next articles we will deal with these topics:

• synchronize project versions,
• automate deploys and releases,
• tips for continuous integration.

So stay tuned! And if you need some help on your build process, contact our service team!

ShiningPanda new plans: Pro accounts starting at $12

ShiningPanda's Hosted Continuous Integration service pricing is upgraded today, for both open-source and private accounts, with a usage-based twist.

We are now offering exactly two plans:
- the Pro plan,
- the FOSS plan.

Pro Plan

The Pro plan is priced at $12 per month (or 9€).

It comes with $6 worth of credit, refilled every month, that can be spend as follows:
- for Linux: $0.48 per hour of build,
- for Windows (currently in beta): $0.64 per hour of build.

So you get either 12.5 hours of Linux build, or 9.3 hours of Windows build, or a mix of both every month.

FOSS Plan

The FOSS plan is free, and as implied by its name reserved to Free/Libre/Open-Source Software.

It comes with $6 worth of credits, refilled every month, that can be spend on Linux at a price of $0.48 per hour of build. So you get a total of 12,5 hours of Linux build every month.

Credit Packs

If the credits included in your plan are not enough, or if you want to make builds on Windows with a FOSS plan, you can buy Credit Packs.

Prices remain the same:
- for Linux: $0.48 per hour of build,
- for Windows (currently in beta): $0.64 per hour of build.

Find more information on Shining Panda's pricing page.

See you in Santa Clara!

We are getting ready to fly all the way to California, for the PyCon 2012 in Santa Clara next week-end.

We will have the booth #223, so please come and have a chat with us! About your projects, about Python, or even about how nice the weather is in California compared to France, we will be happy to talk!

If you see dudes wearing a T-Shirt with a huge panda head... that should be us ;)

See you all soon!