Friday, 15 August 2014

libAttachSQL 0.2.0 alpha released!

Hot on the heals of last week's release we have released version 0.2.0 alpha of libAttachSQL.  For those who have missed my previous blog posts, libAttachSQL is a lightweight C connector for MySQL servers I'm creating with HP's Advanced Technology Group.  It has an Apache 2 license so is good for linking with most Open Source licenses as well as commercial software projects.

Changes in this release:

  • Added support for query result buffering
  • Passive connect on first query is now asynchronous
  • Improved memory handling
  • Many documentation changes, including API examples
  • Many other smaller fixes
For more information see the libAttachSQL documentation and the release itself can be found on the libAttachSQL website.

We have had some great feedback so far.  Thanks to everyone who has contacted me since the first release.  As always if you have any questions feel free to contact me or file an issue on GitHub.

Sunday, 10 August 2014

libAttachSQL 0.1.0 alpha released!

As I briefly mentioned in my previous post, I have been working on a new project for HP's Advanced Technology Group called libAttachSQL.

libAttachSQL is a lightweight C connector for MySQL servers.  It is Apache 2 licensed (and therefore compatible with many open source licenses as well as commercial use) and has a new asynchronous API.  With the new API you send a command which returns immediately and you poll until the library tells you there are results ready, this is very useful for applications that have many things going on that you do not want held up by waiting for the MySQL server to process a query.  In later posts I will give usage examples of this.

I am a great believer in release early/often so on Friday, 5 weeks after I started writing code (and docs), I have released the first alpha version of this connector.  The source of this release can be downloaded here.  For now this is a source-only release just to give a taste of the project so far.  At some point before GA binary packages will be released too.  Documentation for the library can be found on Read The Docs.

What it can currently do:

  • Compile in CLang and GCC on Linux and Mac
  • Cross-compile for Windows using MinGW64 (in Fedora only)
  • Connect to MySQL servers using TCP or Unix socket file
  • Send basic MySQL queries and retrieve results
  • Using an API similar to prepared statements it can automatically escape and convert data for your queries
  • Not a lot else
As the project progresses we will be adding many more features such as prepared statement support.

This project is completely open, using many available free services as described in my previous blog post.  We welcome people to come and kick the tyres and contribute in as small or large way as possible.  This can be simply filing a bug or feature request, contributing docs or code, etc...  One thing we could really use right now is someone with Debian/Ubuntu expertise to help us create the Debian package scripts (I'm not an expert at these and am struggling to make it work).  There is a GitHub issue open for this.

If you have any questions about the library feel free to contact me, comment on this blog post, open a GitHub issue or come chat on the #libAttachSQL channel on Freenode.

Friday, 8 August 2014

How cloud hosted services are helping open source

One big project I'm working on for HP's Advanced Technology Group right now is an Apache 2.0 licensed C connector for MySQL servers called libAttachSQL.  The whole process, not just the code itself, is helping us learn about new and current techniques in Open Source development.  Whilst I will be writing many posts about libAttachSQL in the future, today's post is about the free hosted services we are using around it.


Almost all previous Open Source projects I have worked on in the past have been hosted on Canonical's Launchpad platform.  Over the last couple of years there has been a shift to using GitHub and almost everything I have worked on at HP has been hosted there.  Now there are many services that hook into GitHub so this seemed like the perfect opportunity to try some of them out.

The libAttachSQL project has its own organisation in GitHub and a couple of trees under this.  The service is fantastic and has grown a lot over the years in features and reliability.  The only thing I don't quite agree with is that they prefer a custom type of Markdown documentation over other formats.  Some reStructuredText support is there but it isn't as good as I would hope yet.  This is a really minor issue though and not something they should be knocked down on.

GitHub Pages

GitHub pages is a relatively new service created by GitHub.  Simply create a tree with a specific name, push some static content, and you are done!  There is also an easy method to get domains pointed to it so we have a GitHub page as the site for

Read The Docs

Every Open Source project I have worked on from Drizzle onwards has had its documentation in reStructuredText format which compiles into HTML, PDF and many other formats using a Python based tool called Sphinx (not to be confused with the search server).  In my opinion it is more flexible than Markdown format, especially when documenting APIs.

libAttachSQL's documentation was again written in reStructuredText format and is automatically compiled into HTML and PDF documentation using the free service Read The Docs.  This is hooked into GitHub so on a new push/merge Read The Docs will automatically generate a new version of the documentation.  We have pointed a subdomain to the Read The Docs output so that it can be easily accessed,

I am extremely pleased with this service, not only is it free for Open Source projects but it makes documentation even more aesthetically pleasing than the basic Sphinx templates do.

Travis CI

Every source code project needs Continuous Integration.  There are many solutions to this, one of the most popular being Jenkins.  As with the RST documentation format every project I have worked on from Drizzle onwards uses Jenkins to test every branch before and after merging.  I could have used Jenkins for this project but my goal is not to own the hosting of anything.  So, for libAttachSQL I setup Travis CI.  This is a hosted service that is free for Open Source projects and has a paid-for variant for private projects.

Our Travis setup will test compiling in CLang and GCC in Linux (Travis uses Ubuntu 12.04), running a test suite in each.  Every virtual test host comes with a MySQL server already running for you to use in your tests and it was very simple to set this up.  We also get the Travis tests to build the documentation with nitpick mode and warnings as errors so any minor documentation problems are picked-up early.  All this is done with a very simple YAML script (although ours has got a little more complex with adding support for Coverity).

At a later date I want our builds to also run Valgrind checks and on the provided OS X platform, but I will work on getting those running at a later date.

Travis is a fantastic service and a breeze to setup and use.  The interface shouldn't be too unfamiliar if you have used Jenkins before.  My wish is that it supported more platforms.  I would really love a Fedora based builder, a more up-to-date Ubuntu and possibly Windows builders.  Although they do have OS X builders which is fantastic.

Coverity Scan

Coverity Scan is a static code analyser which is free to Open Source projects hosted on GitHub, it also hooks in nicely with Travis CI with Travis providing the analysis data from the code and builds to break down on the site.  This was the most complex of all of these services to setup but has given some fantastic results so far.  It has found 13 potential bugs in my code that CLang's lint and Valgrind didn't find.  This is really impressive, for starters there are incredibly strict flags set for building the project from git, also there was only one false positive.  Unfortunately there is a quota limit for Open Source projects so we only run this occasionally rather than every merge.


We have managed to have all of the services that we would need to setup and manage setup for us completely for free and no hosting for us to manage.  And these are all awesome services and most were very quick to setup.  I thank all of the companies providing these services, it has easily shaved a week off my time setting up machines to host our project and many more hours managing the services.

Over the next couple of weeks I will be talking a lot more about the libAttachSQL project, so look out for those posts.