Sunday, 24 February 2013

First version of Drizzle Tools for MySQL servers released

Today marks the first release of Drizzle Tools for MySQL servers.  Drizzle Tools aims to be a collection of useful utilities to use with MySQL servers based around the work on the Libdrizzle Redux project.

In this first version there is one utility in the tree called 'drizzle-binlogs'.  If you've seen me talk about this tool before it is because it used to be included in the Libdrizzle 5.1 source but has now been moved here to be developed independently.  For those who haven't 'drizzle-binlogs' is a tool which connects to a MySQL server as a slave, retrieves the binary log files and stores them locally.  This could be used as part of a backup solution or a rapid way to help create a new MySQL master server.

Due to the API changes before the Libdrizzle API became stable Drizzle Tools requires a minimum of Libdrizzle 5.1.3 to be installed.

I wanted to release this sooner but unfortunately most of my time has been taken up with the first release of the project I manage and develop for my day job (HP Cloud's Load Balancer as a Service, more about this in a future blog post).

In the not too distant future there will be more tools included in the Drizzle Tools releases, I have the next one already 50% developed.  In the mean time you can download the first version here.

Saturday, 9 February 2013

Using the Libdrizzle Binlog API

Now that we have frozen the 5.1 API of Libdrizzle I can blog about how to use parts of the API.

In this blog post I will cover connecting to a MySQL server and retrieving the binary logs.

First of all we need to connect to the MySQL server

#include <libdrizzle-5.1/libdrizzle.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <inttypes.h>

int main(void)
  drizzle_st *con;
  drizzle_binlog_st *binlog;
  drizzle_return_t ret;
  con= drizzle_create("localhost", 3306, "user", "pass", "", NULL);
  ret= drizzle_connect(con);
  if (ret != DRIZZLE_RETURN_OK)
    printf("Could not connect to server: %s\n", drizzle_error(con));
    return EXIT_FAILURE;

We are connecting to a local server on port 3306 with the username "user" and password "pass".  We don't need a default database so we leave that empty and set the options to NULL.

After we are connected to the MySQL server we need to initialize the binary log transfer.

  binlog= drizzle_binlog_init(con, binlog_event, binlog_error, NULL, true);

This states that we want an event to call function called "binlog_event" and an error to call a function called "binlog_error".  We could pass a pointer data to callback functions but for this example we have set it to "NULL".  The final parameter specifies whether or not we want to verify the checksums in MySQL 5.6 binary logs.

Then we need to start the binary log transfer.

  ret= drizzle_binlog_start(binlog, 0, "", 0);
  if (ret != DRIZZLE_RETURN_EOF)
    return EXIT_FAILURE;  }
  return EXIT_SUCCESS;

This states that we want to start the binary log at the first position of the first file using server ID 0.  With server ID 0 the MySQL server will end the transfer at the end of the last binary log file.  The function will return at the end of the transfer with DRIZZLE_RETURN_EOF or an error code.

All we need now are error callback and event callback.  So lets start with the error callback.

void binlog_error(drizzle_return_t ret, drizzle_st *con, void *context)

  (void) context;
  if (ret != DRIZZLE_RETURN_EOF)
    printf("Error retrieving binlog: %s\n", drizzle_error(con));

Finally we need to event callback.

void binlog_event(drizzle_binlog_event_st *event, void *context)
  (void) context;
  uint32_t length;
  printf("Timestamp: %"PRIu32"\n", drizzle_binlog_event_timestamp(event));
  printf("Type: %"PRIu8"\n", drizzle_binlog_event_type(event));
  printf("Server-id: %"PRIu32"\n", drizzle_binlog_event_server_id(event));
  printf("Next-pos: %"PRIu32"\n", drizzle_binlog_event_next_pos(event));

  length= drizzle_binlog_event_length(event);
  printf("Length: %"PRIu32"\n", length);
  data= drizzle_binlog_event_data(event);
  printf("Data: 0x");
  for (i=0; i<length; i++)
    printf("%02X ", data[i]);


On every event received this function is called and will basically spit out the event details.

For a further example see the source of the drizzle_binlogs tool in the Drizzle Tools tree.

Libdrizzle 5.1.3 released

A couple of days ago we released Libdrizzle 5.1.3.  With this release of the C connector for MySQL servers we are freezing the 5.1 API and declaring it stable.  This is also one of our biggest releases after incorporating code from a Seattle developer day.  The diff since 5.1.2 is over 6000 lines long and around 180KBytes, incorporating many bug fixes and improvements.

The most notable changes in this release are:
  • the drizzle_binlogs tool has been removed, it is now in the Drizzle Tools tree which will have its first release soon.
  • the connection API has been refactored, options processing has been re-written and the connection API has been simplified in general
  • drizzle_escape_string has been made safer
  • drizzle_hex_string and drizzle_mysql_password_hash has been removed
  • internal state system refactored to be safer
  • binlog API changed to use a callback based API
  • general API cleanups
To download the source and documentation of the latest release simply go to the download page on Launchpad.

Saturday, 2 February 2013

Developing Libdrizzle

This weekend I am supposed to be giving a talk at FOSDEM on Libdrizzle.  Unfortunately my kids and I all fell ill on Thursday (my wife appears to be immune) so I had to cancel my plans (infecting 5000 people didn't seem wise :)

Instead I am writing this blog post about Libdrizzle and my part in it which covers some of what I was going to talk about.

History of Libdrizzle

Libdrizzle started out as a from-scratch C connector for Drizzle and MySQL originally created by Eric Day.  It was designed to be high performance and use common standards to make it easy to work on.  In the summer of 2010 it was merged into the main Drizzle tree where development has been focused.  There were several attempts to split it out again but until now none were truly successful.

For a few years Brian Aker and I have been randomly discussing things we could have done to make the API easier for developers and users.  As well as features we could possibly add in the future.  Fast forward to Summer 2012 and I decided to do something about this.

The Dawn of Libdrizzle Redux

Around July 2012 I took the Libdrizzle 1.0 code from Drizzle trunk and got it to compile separately using CMake.  I then had the groundwork to work on some of the things we had talked about.  I called this the Libdrizzle Redux project since we were bringing new life to Libdrizzle, primarily as a MySQL Connector.

Initially this was going to be version 3.0 but it was pointed out by Henrik that 3.0 and 4.0 were used as library versions in certain packages so we went straight to 5.0.

After getting to the "It Compiles!" state the first thing I did was strip out the server side API.  I had spoken to many people over the years about the connector and each one only used the client side API, not the server side.  The other thing that was made plainly clear was the API was too complex.  There were many functions that were confusing as to whether the library or the application was supposed to allocate and free objects.  So this was massively simplified very quickly.

Libdrizzle Today

Between Brian and I there have been massive changes to the Libdrizzle code and we now have the basis for a regression suite which has already found several problems with the original code.  Just off the top of my head:
  • There is a new Prepared Statement API
  • There is a new Binary Log Retrieval API (in 5.1.3 this will become a callback API) which is already MySQL 5.6 checksum compatible
  • Massively simplified API, the number of things needed to setup a connection and execute a query have been reduced
  • Non-blocking API has been made more stable
  • Windows support (via. MinGW)
  • Buffer and state system massively improved
  • CMake has been replace with DDM4
  • We compile in C++ with many warnings switch on (C++ compiler tends to find more problems than the normal C compiler)
  • 106 revisions/merges in the BZR repo.  Most of this since November 2012.
Last Sunday Brian and I had a hacking day in Seattle, we worked non-stop on Libdrizzle for nearly 12 hours, I doubt there were many sections of code that were not touched that day.

Why I Develop Libdrizzle

My day job is the Project Manager on Load Balancer as a Service at HP Cloud (you should hear a lot more about that in coming months).  Whilst this is great I don't get to touch and debug a whole lot of code any more.  So Libdrizzle is something I do in my spare time to keep my brain fully active.

I also like to think that the code will be useful to someone.  I'm actually a fan of the BSD license in this context.  It means commercial code can easily statically compile with Libdrizzle which in-turn means more MySQL Server/Percona Server/MariaDB Server adoption and more support contracts for Oracle/Percona/SkySQL/etc...

Even if no one uses the code I still find it a fun challenge and something I will continue for as long as I can.

The Future of Libdrizzle

In the next few days we will be releasing Libdrizzle 5.1.3, this will contain all the work that was done in Seattle (and my flights to/from Seattle) including a few nice API improvements.  Along with this I will create the first release of Drizzle Tools.

Drizzle Tools is a project which contains utilities for use with a MySQL server, at the moment the only one included is drizzle_binlogs which connects to a remote MySQL server as a slave and downloads the binary logs to local files as a back.  It also has a continuous mode which effectively gives an up-to-the-second backup of the data.  I have the code half-written for the next tool which will not make the first release but should be useful to many people.

Of course anyone is welcome to come hack on Librizzle and Drizzle Tools, file bugs, ask questions, etc...  The more knowledge share the better :)

There will be more posts soon showing how to use Libdrizzle and drizzle_tools as well as announcing the new features, so watch this space!