m3ga blog

Sat, 15 Nov 2008

FP-Syd #9.

On the Thursday night just passed was the 9th meeting of FP-Syd, the Sydney Functional Programming group. We had about 30 people show up at the Sydney offices of Thoughtworks.

First up we had Jeeva Suresh talking about the "The Nix Language", the purely functional language behind the Nix package manager. Normal package managers like Debian apt system, operate on the global state of the system, mutating that global state as packages are added and removed. There are a couple of papers on the NixOS web site explaining it further.

The second presentation was by Thomas Sewell and was titled "Functional Programming : What does it mean?". Tom started off reviewing what it was functional programmers like about functional programming and suggested that a lot of it was to do with its connection to mathematics. He then went on to talk about theorem provers, in particular Isabelle and finished off by doing a live proof that the merge sort algorithm terminates and is correct. The audience was suitably impressed.

This was our first meeting at Thoughtworks and I'd like to extend a huge thanks to Nick Carroll, Raphael Speyer and Thoughtworks the organisation.

Posted at: 10:14 | Category: FP-Syd | Permalink

Tue, 28 Oct 2008

Zytouch Driver.

At bCODE, we're using some projected capacitive touchscreens by a UK company called Zytronic for our Ubuntu Dapper based embedded devices.

When was first looking for touchscreens back in 2006, we chose the ones form Zytronic because they connected via USB and they had Linux drivers. Unfortunately, these drivers turned out to be binary only and were compiled for Redhat 9. Of course the big difference between Redhat 9 and Dapper was the Redhat used Xfree86 and Dapper was using Xorg. In spite of that, the binary drivers did sort of work, but were flaky and at times chewed up way too much CPU for no apparent reason.

In order to get something working quickly, I snooped the USB bus of a touchscreen connected to a windows machine to get a basic idea of the USB communication. It turned out that under normal conditions, all data traffic was from the touchscreen to the host using USB bulk transfers. The odd thing was that the amount of data transferred per unit time was much more than one would expect from something like a touchscreen.

In order to explore the data more fully, I then used libusb to whip together a Linux program that could find the device (Product:Vendor identifier of 14c8:0002) on the USB bus and then sit in a loop reading the USB data and printing it to the screen in hexadecimal.

Watching the hex data scroll by, it quickly became obvious that the data was the raw X/Y capacitive sensor readings for the screen. One complete read from the screen would consist of 32 bytes, one for each of 16 x-direction columns and one for each of 16 y-direction rows. Getting good mouse pointer performance out of this data was a little difficult due to a number of factors:

• The data from the sensors was very noisy.
• The 8 bit sensor reading for any row or column might have a quiescent value quite high in the [0..255] range and would wrap around to a low value when a finger was present in that cell.
• The sensors suffered from bleed into adjacent sensors so that if a finger was placed in column X, the readings for the columns on either side of X was also higher than their quiescent values.

Fortunately I had quite a bit of Digital Signal Processing knowledge and already had all the conceptual tools I needed to deal with this raw data. In the end I came up with a solution [Note 1] that consisted of:

• Bias the sensor value so that the quiescent value was low in the [0..255] range and there was no wrap over.
• Add a per column/row sensitivity normalization factor.
• Time domain filtering of each row/column value (this filtering was later removed to improve responsiveness).
• Find the row and column with the maximum sensor value.
• Calculate a centre of mass of 5 values around the maximum sensor value to obtain a finer grain resolution than the 16 rows and columns would normally allow.
• Apply scaling to get the calculated X/Y position to a screen x/y position.

Once we had the X/Y position data we needed to get it into the Xserver. The easiest and quickest way to do this was using the XTest extensions.

bCODE has now been using this touchscreen driver for two years, even updating it to use a later model from Zytronic (Product:Vendor identifier of 14c8:0003) which did its own filtering and spat out high quality X/Y values.

The reason I am blogging all this now is that bCODE has decided to release the code for this driver under the terms of the GNU GPL V3. The code is copyright bCODE Pty Ltd and I am listed as the author and maintainer and the code with live on my web site. Most of the nitty gritty details are in the Readme.txt file. The driver source code also includes a calibration utility written in Ocaml using the Ocaml Cairo bindings.

Note 1: It should be noted that I did most of the reverse engineering work during a period when I was insanely busy. After that week, I had a very rough proof of concept driver that still needed quite a bit of work. In the following three weeks, my manager Sean, in his spare time, hacked on the code, fixed bugs and even improved X/Y positioning algorithm.

Posted at: 22:03 | Category: CodeHacking | Permalink

Sat, 18 Oct 2008

Foobar 2000 and the Rabbit.

Foobar 2000 (see also the Wikipedia entry) is a media player for a legacy operating system commonly known as Microsoft Windows. Secret Rabbit Code is an audio sample rate converter that I wrote, and released under the terms of the GNU GPL 2002.

As the sole author and copyright owner of Secret Rabbit Code I have also made it available under a commercial use license (PDF) that is currently earning me a small income. However, developing Secret Rabbit Code was difficult, took a huge amount of research and the development of many prototypes which were thrown away. Now, after 6 years, that income is coming close to covering the cost of developing that initial version. It still has some way to go to cover the cost of the subsequent maintenance and the improvements I have made.

In 2005 I became aware that someone had released a binary only plugin for Foobar 2000 that used Secret Rabbit Code to do sample rate conversion. The fact that this was a binary only release was not the only problem. There was also the problem of Foobar being under a license that was not GPL compatible.

First off, I emailed the ISP in France where the binary was being hosted and asked for the download to be taken down. I also tried to track down the author of the plugin since authorship was not obvious from the download. Within a day or two I was able to track him down via the Hydrogen Audio forums.

The final result of the discussion on that forum was that I decided I would write and release a Secret Rabbit Code based plugin for Foobar. This of course involved development for windows on windows a platform I rarely if ever use and which I actually prefer not to work on. However after some 10 or 12 hours work I had a working plugin that I posted on my website. I also added a Paypal donation button on the page hoping to get paid for the time and effort I put into creating the plugin.

Unfortunately, the donations have been few and far between. Since 2005 when the plugin was released I have only had about 10 people pay the measly US$10 I am asking for. That is despite the fact that I have proven interest in this plugin. The page has had over 30000 hits since 2005, and I get an email every couple of weeks asking if there is a more recent version using later versions of Secret Rabbit Code or a version for other versions of Foobar 2000.

So here's how it stands:

• I don't use Windows or the Foobar 2000 program.
• As someone who has been coding on Unix and Linux for well over a decade and rarely touches windows this work was a rather painful and unpleasant exercise.
• I spent my precious time developing this plugin for a platform I don't use
• For the 10-12 hours work I put into this plugin I have received about US$100 which means I got paid US$8 to $10 per hour for my time. This is a small fraction of my normal salary as a professional software engineer with well over a decade of experience.
• The fact that I release the Rabbit for free (as in free beer) under the terms of the GPL is irrelevant. I use dozens of GPL and other FOSS (Free and Open Source Software) programs every day and I consider the Rabbit and my other project libsndfile as my paltry contribution back to the FOSS cause.

Since most users of this plugin don't think I should be paid for my work creating it for use with Foobar on Windows, I currently have no intention of releasing a new version of this plugin. When and if I get a decent stream of payments for the work I have put in so far I will roll out a new version and maybe even an updated plugin for other versions of Foobar.

I will answer Foobar related emails from people who have donated or work for companies that paid for a commercial use license. The vast majority of other emails regarding the Foobar plugin will be directed to this blog post.

Posted at: 09:21 | Category: CodeHacking/SecretRabbitCode | Permalink

Fri, 17 Oct 2008

FP-Syd #8.

Last night was the 8th meeting of FP-Syd, the Sydney Functional Programming group with huge turnout of 36 people. We had had three presentations with a very heavy Haskell slant.

First up was Conrad "kfish" Parker giving a introduction to type level programming in Haskell. This talk was a simplified version of his paper in issue 8 of the Monad Reader which described solving the Instant Insanity puzzle in Haskell's type system.

Conrad's type level programming solution uses Haskell's type system (specifically a feature called Functional Dependencies) to generate a solution so that the executable program created by the compiler needs to do nothing more than print the result. This is possible because Haskell's type system is a Turing complete logic programming language (much like the way C++'s template system is a Turing complete functional language).

After Conrad's warm up we had Manuel M T Chakravarty giving us an introduction to Type Families, a new extension to the GHC Haskell compiler which Manuel and some others have just completed.

The motivation for Type Families was to allow the programmer to be able to give the compiler better hints on how to optimize things, but the resulting feature goes far beyond that. The example Manuel gave was defining a data type as an array of (Int, Bool) pairs which would normally be un-boxed requiring two pointer de-references to access a part of the pair at a given index. Type Families however allow the programmer to supply hints to the compiler that it should store these as a pair of contiguous arrays, one for the Ints and one for the Bools.

Finally Manuel mentioned that since Type Families can do everything Functional Dependencies can (and more), but in a cleaner way, that Functional Dependencies may eventually be removed from the language.

Last up we had Simon Winwood giving us a demonstration that Monads aren't just for I/O. Simon's presentation was in many ways a follow on from Manuel Chakravarty's "Monads are Not Scary" presentation from the 2nd FP-Syd meeting. He demonstrated an interpreter for a simply language and used the State monad to handle state, Reader monad to read input and the Error monad for error handling.

Once again a big thank you to Google for supplying the venue as well as food and drinks. Next month we will be moving to Thoughtworks for the first of a series of meetings. I look forward to more head exploding concepts.

Posted at: 19:39 | Category: FP-Syd | Permalink

Sat, 20 Sep 2008

Codecon 2008.

Well, its another Codecon and this year we're on the coast at Coledale, just north of Wollongong. On one side we have the beach:

and on the other side we have some pretty impressive mountains.

This year we have internet connectivity thanks to being within mobile phone range and Silvia sharing her broadband wifi. The funny thing was Silvia and John having a Skype conference call at 8am with a potential client in Hawaii.

As always, thanks to Peter Miller. for organising it.

Posted at: 11:10 | Category: | Permalink

Fri, 19 Sep 2008

FP-Syd #7.

Last night was the 7th meeting of FP-Syd, the Sydney Functional Programming group with 26 people attending. We had had three short presentations.

First up was Jeremy Apthorp explaining the Y Combinator using a Haskell-like syntax instead of the more usual LISP syntax.

The second presentation was Ben Sinclair giving us a introduction to the Scala language. Scala is a functional language targeting the Java Virtual Machine. Its syntax is Java like, but with elements of ML and things that looked a little like Python. Under the hood, Scala is a traditional object-oriented lagngauge where everything except a few primitive types are derived from a base Object class. Scala can also call into any Java library.

Finally we had Eric Willigers talking about parser combinators in Scala. Eric's first example was just a parser that did parenthesis and square bracket matching. He then showed a more complex example, a partial implementation of a parser for the C language. Interesting.

Once again a big thank you to Google for supplying the venue as well as food and drinks.

Posted at: 21:39 | Category: FP-Syd | Permalink

Sun, 14 Sep 2008

New Bicycle.

Last Saturday, my bicycle was stolen from the foyer of my apartment building.

Normally the bike was protected by a locked gate and a locked front door, but on that day the gate and the front door were left open for about 5 minutes while we waited for a painter to arrive to discuss the painting of our apartment. Some time between the gate being opened and the painter arriving, some lowlife walked in and stole my bike.

Since we moved house recently and are currently in the middle of renovating, everything is in boxes and I have no idea which box contains information like the frame serial number. I have reported the theft to police, but without a serial number the chances of the lowlife thief being caught are close to zero. I do however still fantasize about finding the culprit and breaking their legs.

I now have a new bike; a Scott P2 Sportster city bike, a slightly more up market version of the Scott P4 that was stolen.

It has disk brakes instead of the caliper brakes of my old bike. I also got a few other extras so that it better matched the old bike:

• Schwable Marathon Plus tires, the toughest tires I've ever had.
• A more comfortable seat.
• Mud/splash guards.
• Better pedals.
• A bike computer.

Its also quite a bit lighter than the old bike and is a real pleasure to ride. I now need to work on keeping it out of the hands of thieves.

Posted at: 12:26 | Category: Cycling | Permalink

Fri, 05 Sep 2008

Tax Accountant.

My wife and I got our tax done today. We went to a branch of one of the many well known firms specialising in tax, just like we have for the past several years. The consultant who handled our affairs was the same one who did our tax last year. He's a really nice fellow and we even got around to talking about free audio codecs like Ogg/Vorbis and FLAC.

During the consultation we were talking about an account for our daughter that is currently in my wife's name. We were not sure of the interest that had accrued in the account and the consultant suggested that we log onto our internet banking account using the consultant's computer.

Despite huge alarm bells going of in my head, neither my wife nor the tax accountant thought anything of it. I hesitated for a moment and then said "Sorry, I don't think thats a good idea. I simply don't trust a windows machine." As an alternative, my wife used a telephone in another office to retrieve the information using the telephone banking system and that was the end of it.

Thinking about this further I came to the conclusion that this is potentially really dangerous. If some BadPerson was able to install a keystroke logger on all the computers of all the consultants in a company like this and if accessing client internet banking accounts was common in this company, the BadPerson could potentially retrieve the user name and passwords of a huge number of the company's clients. The BadPerson would then be able to either skim or even empty all the accounts at will.

Just to be clear, the BadPerson in the above senario would not the tax accountant. It could possibly be a tachnical person within that company, or, thanks to the woeful security inadequacies of windows, organised criminals somewhere outside Australia who are able to install the key logger via a vulnerability in Internet Explorer.

After leaving the tax accountant I explained the dangers to my wife. I strongly suggested that she never access internet banking other than from our computers at home or when travelling together, from my computer. In particular I suggested that she stop accessing internet banking from her work machine (a managed and supposedly locked down windows machine) and to change the passwords on her account.

Rule number one for internet banking, never access your account from a machine you don't trust unless you immediately change the password via some other, separate, secure method. I'm actually rather surpised that the tax consultants for companies like this don't have at least a little computer security awareness training.

Posted at: 22:00 | Category: Security | Permalink

Mon, 25 Aug 2008

SSH Session Sharing.

Here's a neat trick I got from johnf who got it from jdub who got it from lifeless who according to him "got it from some random post a few years ago".

The trick is for anyone who uses slogin to do multiple logins to a single machine. By adding the following to the ~/.ssh/config file:

  Host *
      StrictHostKeyChecking yes
      ControlMaster auto
      ControlPath ~/.ssh/master-%r@%h:%p

all connections to that machine after the first will share the same initial connection. The big advantage is that for remote machines that are a little slow to log into normally, these shared connections will almost instantaneous.

Johnf tells me that the one caveat is that occasionally if you shut down the first connection the others may hang and its not possible to reconnect until you manually delete the ControlPath file.

Posted at: 18:10 | Category: Tech | Permalink

Sat, 23 Aug 2008

FP-Syd #6.

Last night was the 6th meeting of FP-Syd, the Sydney Functional Programming group with 24 people attending.

The first presentation was by James Kozianski titled "Evaluators and Higher Order Abstract Syntax". James showed that when using a language like Scheme (or LISP) where code and data are both stored as S expressions, that abstract syntax tree of an embedded DSL can actually be encoded in the host language's own data structures. This is rather a neat trick and it was great to see a presentation in Scheme. Thanks James.

David Hearnden gave the second presentation titled "Deltaware: Incremental Change Propagation through Model Transformations". He started of explaining Model Driven Engineering, a simple example of which would be generating database schema from the definition of a set of classes/objects. The problem that David's presentation covered was how to deal with changes to the original classes/objects that need to be propagated to the database schema. He then showed how logic programming languages provided a solution to this problem that functional (and imperative) languages did not (ignoring the possibility of Greenspunning logic language features into a functional language). This was a very thought provoking presentation. Thanks David.

I'd also like to thank Google and the Google employees who helped make this meeting happen. We will be having one more meeting at Google before moving on to Thoughtworks' Sydney office for a block of four meetings.

Posted at: 08:34 | Category: FP-Syd | Permalink

Wed, 20 Aug 2008

Just Drawing Stuff on the Screen.

Richard Jones laments that drawing stuff on the screen is harder than it should be. I haven't seen his code, but it looks like he might be trying to do it with Ocaml and GTK which probably is more difficult than it should be. GTK isn't really meant for that sort of stuff.

Fortunately, there is a really well designed and thoroughly thought out library for doing graphics called Cairo, which even has a really great set of Ocaml bindings. On Debian/Ubuntu, the Cairo bindings can be installed using:

   sudo apt-get install libcairo-ocaml-dev

I messed about with Ocaml and Cairo about a year ago and came up with this little demo.

  (*
  **    http://www.e-dsp.com/what-are-fourier-coefficients-and-how-to-calculate-them/
  **
  **    http://en.wikipedia.org/wiki/Fourier_series#Definition
  *)

  type fourier_series_t =
  {   a0 : float ;
      an : float array ;
      bn : float array ;
      }


  let initial_size = 200

  let two_pi = 8.0 *. atan 1.0

  let sum_float_array ary =
      Array.fold_left (fun x y -> x +. y) 0.0 ary


  let calc_series max_n ary =
      (*
      **    This uses a rough numerical approximation to integration.
      **    As long as the array is long enough (say 1000 or more elements), the
      **    results should be reasonable.
      *)
      let len = float_of_int (Array.length ary) in
      let calc_Xn trig_func n =
          let n = n + 1 in
          let ary = Array.mapi (
		  		fun i x -> x *.
                trig_func ((float_of_int (n * i)) *. two_pi /. (len -. 1.0))
				) ary
          in
          2.0 *. (sum_float_array ary) /. len
      in
      let a0 = (sum_float_array ary) /. len in
      let an = Array.init max_n (calc_Xn cos) in
      let bn = Array.init max_n (calc_Xn sin) in
      { a0 = a0 ; an = an ; bn = bn }


  let waveform_of_series outlen series =
      (*
      **  Given a fourier series, calculate a single cycle waveform of the
      **  specified length.
      *)
      let calc_point i =
          let x = two_pi *. (float_of_int i) /. (float_of_int (outlen - 1)) in
          let asum = sum_float_array (Array.mapi (
                    fun i an -> an *. (cos (float_of_int (i + 1) *. x))) series.an
                    )
          in
          let bsum = sum_float_array (Array.mapi (
                    fun i bn -> bn *. (sin (float_of_int (i + 1) *. x))) series.bn
                    )
          in
          series.a0 +. asum +. bsum
      in
      Array.init outlen calc_point


  let fold_over_clipped_sine gain len =
      let point i =
          let x = gain *. sin (two_pi *. (float_of_int i) /. (float_of_int len)) in
          if x > 1.0 then x -. 2.0
          else if x < -1.0 then x +. 2.0
          else x
      in
      Array.init len point


  let redraw w series _ =
      let cr = Cairo_lablgtk.create w#misc#window in
      let { Gtk.width = width ; Gtk.height = height } = w#misc#allocation in
      Cairo.save cr ;
      (   Cairo.identity_matrix cr ;
          let border = 20.0 in
          Cairo.move_to cr border border ;
          Cairo.line_to cr border (float_of_int height -. border) ;
          Cairo.stroke cr ;

          let wave_width = width - 100 - (int_of_float border) in
          let middle = float_of_int height /. 2.0 in
          let wave_height = 0.7 *. (middle -. border) in

          Cairo.move_to cr border middle ;
          Cairo.line_to cr (border +. float_of_int wave_width) middle ;
          Cairo.stroke cr ;

          Cairo.move_to cr (border +. float_of_int wave_width) border ;
          Cairo.line_to cr (border +. float_of_int wave_width)
                                      (float_of_int height -. border) ;
          Cairo.stroke cr ;

          Cairo.set_source_rgb cr 1.0 0.0 0.0 ;
          let wave_data = waveform_of_series wave_width series in
          Cairo.move_to cr border (float_of_int height /. 2.0) ;
          Array.iteri (fun i x ->
                        Cairo.line_to cr (border +. float i)
						      (middle -. wave_height *. x))
                        wave_data ;
          Cairo.stroke cr ;
          ) ;
      Cairo.restore cr ;
      true


  let () =
      if Array.length Sys.argv != 2 then
      (   Printf.printf "Usage : %s <series length>\n\n" Sys.argv.(0) ;
          exit 0 ;
          ) ;

      let series_len = int_of_string (Sys.argv.(1)) in

      let w = GWindow.window ~title:"Fourier Series Demo" ~width:600 ~height:400 () in
      ignore (w#connect#destroy GMain.quit) ;

      let b = GPack.vbox ~spacing:6 ~border_width:12  ~packing:w#add () in
      let f = GBin.frame ~shadow_type:`IN ~packing:(b#pack ~expand:true ~fill:true) () in
      let area = GMisc.drawing_area ~width:initial_size ~height:initial_size
                    ~packing:f#add ()
      in
      let array_len = 1000 in
      let wave = fold_over_clipped_sine 1.2 array_len in
      let series = calc_series series_len wave in

      ignore (area#event#connect#expose (redraw area series)) ;

      w#show () ;
      GMain.main ()

The above code can be compiled using:

    ocamlopt -I +cairo -I +lablgtk2 cairo.cmxa lablgtk.cmxa cairo_lablgtk.cmxa \
	    gtkInit.cmx fsdemo.ml -o fsdemo

and the output looks like this:

So while I agree that the 140 of lines of code here is about 30 times as much as Richard's code from his ZX80 days, I also think the results are at least 30 times as good.

Posted at: 22:29 | Category: CodeHacking/Ocaml | Permalink

Mon, 18 Aug 2008

Nemiver : A GUI debugger for GNOME.

The many years, Linux has lacked a good GUI debugger for C and C++ programs. Yes, everyone knows about GNU GDB, but that is a command line debugger and really not very useful for stepping through a program. There was also the Data Display Debugger (DDD) which uses the Motif widget set, usually supplied by the Lesstif Project. Unfortunately, Lesstif development has basically been abandoned and OpenMotif is not really an option because its license fails to meet term 8 of the Open Software Definition.

This means that for many years, developers on Linux have tended to avoid the "stepping through code with a debugger" approach to debugging. While I think that single stepping is not the most applicable to every debugging problem, there are times when single stepping is useful and possibly also the fastest way to track down a problem.

However, I was recently made aware of a new GUI debugger for the GNOME (ie really the Linux) desktop, Nemiver.

The only problem with nemiver is that the version in Ubuntu Hardy is a little old and was giving me a few troubles. However, after building and installing a version 2.2.0-2 package of libgtksourceviewmm-2.0 from source I was able to build nemiver from SVN and so far its working way better than DDD ever did.

So, here it is, a good looking, stable and capable GUI debugger for Linux.

Posted at: 19:39 | Category: CodeHacking | Permalink

Mon, 04 Aug 2008

Sydney's Drivers Worst for Bike Rage.

According to this article in the Sydney Morning Herald Sydney's drivers are the worst for bike rage; aggression show towards cyclists simply for being on the road.

From the article:

Sydney drivers are more hostile towards cyclists than motorists of any other country, says a world authority on transport.

Christian Wolmar, a Briton who was invited by the State Government to promote bicycle use in NSW, said Sydney's cycling infrastructure is 10 years behind that of London.

Further on in the article it quotes a spokeswoman for the Roads Minister, Eric Roozendaal, saying:

"Sydney has around 2000 kilometres of cycle facilities while London only has 550 kilometres,"

Obviously this sounds good until you realize that Sydney is 12,144 square kilometers in area while Greater London is only 1579 square kilometers. That means that Sydney has 164 meters of cycle lane per square kilometer while London has 348, almost double.

Furthermore, most cycle lanes in Sydney are not just totally inadequate, but outright dangerous.

Posted at: 08:17 | Category: Cycling | Permalink

Wed, 23 Jul 2008

This is Not a Cycle Lane.

Throughout the center of Sydney there are cycle lanes marked like the ones here on Crown St, Woolloomooloo. The cycle lane itself is about a meter wide and the distance between the parked cars and the inside marker of the cycle lane is about 30cm. Also notice that there is traffic in the on coming lane.

The problem with these cycle lanes is that they are potentially lethal for cyclists. The problem of course are the parked cars. If a single car contains a person who opens a driver's side door at the wrong time, that door will almost certainly block most of the cycle lane leaving any unlucky cyclist with a split second to either try to brake or swerve out into the car lane to avoid the car door. Obviously swerving into the car lane might put the cyclist in the path of cars or trucks which are probably even more dangerous than the open car door that the cyclist was trying to avoid.

The only safe option for cyclists in conditions like these is to ride in the middle of the car lane, far from the parked cars. Here they will be clearly visible to vehicles behind them and also to traffic heading the other way.

Cyclists need to lobby their local councils for the removal of these unsafe cycle lanes or replacement with lanes that are safe.

Posted at: 20:46 | Category: Cycling | Permalink

Sat, 19 Jul 2008

Ocaml and Unix.select.

At the June meeting of FP-Syd, Tim Docker gave a presentation about his Tuple Space Server written in Haskell. This presentation rather intrigued me because I have had a long term interest in numerical analysis and numerical optimisation problems which lend themselves very well to parallel and distributed computing. I decided I should write a Tuple Space Server myself, in Ocaml.

Tim's Tuple Space server used threads and Software Transactional Memory (STM) to handle the connection of multiple masters and workers to the server itself. Although the Ocaml CoThreads library does have an STM module I thought there was probably an easier way.

In my day job I'm working on some C++ code that handles multiple network sockets and open file descriptors using the POSIX select system call. On Linux at least, there is a select tutorial man page which gives a example of using select written in C.

The beauty of select is that it allows a single process to multiplex multiple sockets and/or file descriptors without resorting to threads. However, the C example in the tutorial clearly demonstrates that this system call is a bit of a pain to use directly. Fortunately, for the project at work, I had some really great C++ base classes written by my colleague Peter to build on top of. These base classes hide all the nastiness of dealing with the system call itself by wrapping the select call into a daemon class and providing a simple base class which clients of the select call can inherit from.

For Ocaml there is a thin wrapper around the C library function in the Unix module and it has the following signature:

  val select :
    file_descr list -> file_descr list -> file_descr list -> float ->
      file_descr list * file_descr list * file_descr list

It takes three lists of file descriptors (one descriptor list for each of read, write and exceptions), a float value for a timeout and returns a tuple of three lists; one each for the file descriptors ready for reading, writing and exception handling.

Whereas the C++ solution had a daemon class, the Ocaml version instead has a daemon function. The daemon function operates on a set of tasks, with one file descriptor per task. Each file descriptor was embedded in a struct which I named task_t:

  type task_t =
  {   fd : Unix.file_descr ;
  
      mutable wake_time : float option ;
  
      mutable select_on : bool ;

      mutable process_read : task_t -> bool * task_t list ;
  
      mutable process_wake : task_t -> bool * task_t list ;
  
      finalize : task_t -> unit ;
      }

The fields of the struct are as follows:

• fd : This is the file descriptor or socket that this task is operating on.
• wake_time : This is an optional, per task timeout value. It is specified as either None for no timeout or Some with a value in seconds (ie Some 10.0 specifies a ten second timeout). A typical use might be when writing an network daemon where you want to close/drop any client connections which have been silent for more than a specified amount of time.
• select_on : If this is true, the daemon will add this task's file descriptor to the list of files waiting for read. This field (set to false) in conjunction with the process_wake can be used to implement deferred actions on a descriptor that ignore readiness for read.
• process_read : This is the function that is called when the file descriptor has data ready to be read. The function takes the task_t struct as a parameter and returns a tuple containing a bool and a task_t list, which may be empty. There's more on what happens with the return values of this function below.
• process_wake : This is the function that is called when a wake_time value has been set and there has be no activity on the socket for the specified amount of time. This function's return type is the same as for process_read.
• finalize : This function will be called when either of the above two functions notifies the daemon that the file descriptor should be closed.

The first thing to note in the above is the careful use of an immutable field for the file descriptor and mutable fields for process_read, process_wake and wake_time. The file descriptor is immutable so that any client code does not change its value behind the back of the daemon.

The others fields of the struct are purposely made to be mutable so that they can be changed on the fly. The functions process_read and process_wake both return their results in the same manner, a tuple containing two items:

• A bool which indicates whether the daemon should hold onto the task (if true) or if false, the daemon should run the task's finalize function and then drop it to be cleaned up by the garbage collector.
• A task_t list (possibly empty) of new tasks for the daemon to manage. Typically the daemon's server socket would be managed by the daemon function just like any other socket. When the server socket's process_read function accepts a new client connection it would return the new client task in this list.

The actual daemon run loop keeps the tasks in a hash table where the key is the file descriptor. Once the initial set of tasks is in the hash table, the loop basically does the following:

1. Find the file descriptors of all the tasks in the hash table which their select_on field set to true (uses Hashtbl.fold).
2. Find the minimum wake_time timeout of all the tasks (this is actually done on the same pass over all items in the hash tables as step 1.).
3. Pass the file descriptors from step 1. to the select with the timeout value found in 2. (The lists for writable and exception file descriptors are empty.)
4. When select returns a list or file descriptors ready to be read, map the file descriptor to a task using the hash table and then run the process_read function of each readable task.
5. For each task whose wake_time is exceeded, run its process_wake function.
6. For steps 4. and 5., if a task's process function returns false as the first element of the tuple it returns, remove the task from the hash table and run the task's finalize function. Also if the second element in the tuple is a non-empty list, then add the tasks to the hash table.

The above code was placed in a module named Daemon. Using this module, I've whipped up a simple demo program, an echo server the source code of which is available here. The tarball contains four files:

Makefile	The project's Makefile.
daemon.ml	The Daemon module.
echo-server.ml	The Echo server.
tcp.ml	A module of TCP/IP helper functions.

To compile this you will need the Ocaml native compiler which can be installed on Debian or Ubuntu using:

  sudo apt-get install ocaml-nox

The server can be built using make and when run, you can connect to the server using:

  telnet localhost 9301

All lines sent to the server will be immediately echoed back to you.

Posted at: 21:10 | Category: CodeHacking/Ocaml | Permalink

Fri, 18 Jul 2008

FP-Syd #5.

Last night was the 5th meeting of FP-Syd, the Sydney Functional Programming group. The turnout was down a bit (19 people) but I suspect that was due to World Imaginary Friend Day (which for some reason seems to run for a week) clogging up Sydney's streets and public transport system.

The first talk was given by the author of this blog and was titled "Ocaml and Unix.select". I'll be blogging this in more detail some time over the weekend.

Next up we had Ben Lippmeier, author of the Disciple compiler on his entry into the ICFP Programming Contest. In the long tradition of eating one's own dog food, Ben used his DDC compiler to write his entry and ran into a nasty intermittent bug in his compiler's run time. Despite the bug he still managed to put together a pretty decent solution to a rather interesting problem.

As usual, we had post-meeting drinks at the Redoak hotel. Thanks to Shane, James, the other Googlers and Google itself for use of Google's excellent facilities.

Posted at: 18:08 | Category: FP-Syd | Permalink

Wed, 02 Jul 2008

Tonight on the Telephone.

Its a little after 7pm and the phone rings. I answer.

Me : Hello.

Faint music on the other end.

Me : Hello!

Music continues.

Me : Hello !!!

Some Marketer : Yes, good morning, blah, blah, blah. Would you be interested in cheap holidays, blah, blah, blah.

Me : Tell me more.

Marketer : Well, you can go here and you can go there. Blah, blah, blah.

Me : That sounds interesting.

Marketer : Blah, blah, blah.

Me : That sounds good.

Marketer : Blah, blah, blah.

Pause.

Me : Can I bring my pony?

Marketer : Excuse me?

Me : Can I bring my pony?

Clunk, the marketer hangs up.

I managed to waste 1 minute 38 seconds of his time. I need to do better next time.

Posted at: 21:48 | Category: | Permalink

Sun, 29 Jun 2008

Goodbye Old Car.

For the last 8 or 9 years I've been driving this very nice 1985 BMW 323i.

However, the new apartment has off-street parking but not under cover parking. An old car like this would simply not survive being parked out in the open so I've had to sell it rather than see it rust away. It was sold a little over a week ago.

Goodbye old car.

Posted at: 14:29 | Category: NewHouse | Permalink

Mon, 16 Jun 2008

Goodbye Old House.

For the last eight years, my wife and I have enjoyed the room in this picture (with that view) as our main living area. This apartment was more than big enough for the two of us, but with the arrival of our daughter five and a half years ago, it has been increasingly cramped.

After a long, long search, we found and bought a new apartment a couple of months ago and we take possession of the new place this week. We'll have about 5 days to move between houses and then we'll have to say goodbye to this wonderful home which we have enjoyed so much.

Posted at: 19:22 | Category: NewHouse | Permalink

Fri, 06 Jun 2008

FP-Syd #4.

Last night was the 4th meeting of FP-Syd, the Sydney Functional Programming group. First up I'd like to thank Shane, James and the other Googlers who organised the use of one of Google's Sydney meeting rooms and laid on drinks and a bunch of tasty snacks. Another big thanks goes to our speakers for the evening and the 32 (!!!) people who showed up.

Unfortunately one of our planned speakers for the night had to drop out at the last minute. This meant that as people arrived I was asking them if anyone had a short talk they could give at very short notice. The thing that impressed me the most was that we got two offers.

First up we had Sean Seefried talk about using Ocaml at Nicta to write Nicta's Goanna static type checking tool for C and C++ code. It was interesting that they were using an existing commercial C and C++ parser and the converting the C abstract syntax tree (AST) produced to an Ocaml AST where they work on it further. Also interesting was that the chose Ocaml because it was a good language for writing compilers as well as having reasonably good and highly predictable performance. This was an excellent talk considering that Sean had about 10 minutes to prepare. Thanks Sean.

The next speaker was Scott Kilpatrick who also had about 10 minutes to prepare as well as being our first international speaker. Scott has just recently completed two B.S. degrees (one in Computer Science, the other in Mathematics) from the University of Texas at Austin and will be starting a Masters there in September. He's is currently interning at Google here in Sydney working on the Google Maps project.

Scott gave us a very interesting introduction to SUN's Fortress language. Scott is working on implementing the type checking system for the Fortress compiler rather than being a dedicated user of the language. He was however able to give us a brief overview of the language, which is designed to replace Fortran in high performance computation tasks. Fortress seems to have a huge array of interesting features and is definitely something to keep an eye on. Thanks Scott.

Finally we had Tim Docker speaking about writing a Tuple Server in Haskell using the State Transactional Memory monads in concurrent Haskell. Tim did a great job explaining what Tuple Servers are and what they are used for where he works at Maquarie Bank. He then talked about the monads in Haskell, in particular the STM monad and its use in his tuple space server. It was a great talk and I mid way through it, I realised that a tuple space server is a nice solution to an architectural problem I have. Thanks Tim.

After the talks we retired to the Redoak hotel for a few beers and a chat. Since a couple of people had missed the previous meeting, Ben Lippmeier, was kind enough to pull out his laptop and give his talk on "The Disciplined Disciple Compiler" again. Interestingly for me, he went into a little more detail on areas that he glossed over in the last meeting. DDC is a really interesting piece of work, being a strictly evaluated dialect of Haskell, with optional lazy evaluation (far more elegant than the way Ocaml does it) and with side effects and destructive updates looked after in the type system. The ideas behind DDC are really compelling. I look forward to seeing it progress. Thanks Ben.

Posted at: 22:18 | Category: FP-Syd | Permalink

Thu, 29 May 2008

Keeping Up With ....

Here's my pathetic attempt at keeping up with Steve Hanley, Jon Oxer and Mike Beattie.

From left to right:

1. My Dell Latitude X1 laptop running Enlightenment E17 on an Ubuntu Hardy base system. The desktop background is a picture of my lovely daughter.
2. A windows machine which is used mainly as a host for Firefox and the Tortoise SVN client. I spent the first six months in this job doing windows development. Fortunately that is all in the past.
3. My Linux desktop system, again running E17, but on an Ubuntu Gutsy base system.
4. One of our very cute bCODE scanners, which has just been PXE booted to a slightly modified Ubuntu Dapper installation, but before having over 100 of bCODE's own Debian packages installed on it.
5. A second bCODE scanner running its Adobe Flash GUI on top of an Ubuntu Dapper install done by cloning a disk image.

The Linux and windows machine share a keyboard and a mouse via the magic of Synergy2. The keyboard is my favourite Dell Enhanced USB Multimedia Keyboard.

The one thing missing from the above photo (two separate pictures taken and stitched together by my colleague Peter) is the whiteboard which is directly behind my seat. Yes, I have my own personal whiteboard and I love it.

Posted at: 21:54 | Category: Tech | Permalink

Sat, 24 May 2008

Objects vs Modules.

Although I've been using Ocaml for a several years now, I've not yet been in a situation where I've needed to write an Ocaml class to define a C++/Java/Python/Smalltalk/OO style object. I've found that most of the problems I encountered could be easily solved using functional code and that Ocaml's objects didn't provide an obviously better solution. Until now (or so I thought).

The problem was one of moving around the filesystem keeping track of the old directories so they were easy to return to. The obvious model for this was the pushd and popd built-ins in command shells like GNU Bash. This functionality can be easily wrapped up in an Ocaml object as in the following example and demo code (which needs to be linked to the Unix module):

  class dirstack = object
      val mutable stack = []

      method push dirname =
          (* Find the current working directory. *)
          let cwd = Unix.getcwd () in
          (* Change to the new directory. *)
          Unix.chdir dirname ;
          (* If successful, push old cwd onto the stack. *)
          stack <- cwd :: stack

      method pop () =
          match stack with
          |    [] -> failwith "Directory stack is empty."
          |    head :: tail ->
                  Unix.chdir head

  	end

  let () =
      print_endline (Unix.getcwd ()) ;
      let dstack = new dirstack in
      dstack#push "/tmp" ;
      print_endline (Unix.getcwd ()) ;
      dstack#push "/bin" ;
      print_endline (Unix.getcwd ()) ;
      dstack#pop () ;
      print_endline (Unix.getcwd ()) ;
      dstack#pop () ;
      print_endline (Unix.getcwd ())

However, there are some problems with the above code. Firstly, if the push and pop methods need to be used throughout the program, the dstack object needs to be made more widely accessible using one of the following three methods:

1. Being placed in the global scope.
2. Being made into a Singleton objecct.
3. Being passed around as a parameter to whatever function may need it.

Yuck! Yuck! Double yuck! Suddenly, this object oriented solution didn't look like such a great idea.

Then it struck me. This object can be easily transformed into an Ocaml module like this:

  module Dirstack = struct
      let stack = ref []

      let push dirname =
          (* Find the current working directory. *)
          let cwd = Unix.getcwd () in
          (* Change to the new directory. *)
          Unix.chdir dirname ;
          (* If successful, push old cwd onto the stack. *)
          stack := cwd :: !stack

      let pop () =
          match !stack with
          |    [] -> failwith "Directory stack is empty."
          |    head :: tail ->
                  stack := tail ;
                  Unix.chdir head

     end

  let () =
      print_endline (Unix.getcwd ()) ;
      Dirstack.push "/tmp" ;
      print_endline (Unix.getcwd ()) ;
      Dirstack.push "/bin" ;
      print_endline (Unix.getcwd ()) ;
      Dirstack.pop () ;
      print_endline (Unix.getcwd ()) ;
      Dirstack.pop () ;
      print_endline (Unix.getcwd ())

This solution using a module is much better than the one using an object. The Dirstack module itself is globally accessible and is already a singleton while the stack used to hold past directories is implemented as a list whose scope is limited to the module itself. (Furthermore, if Dirstack is implemented in its own file instead of using a module defined within a larger file, then the stack variable can be hidden completely by not listing it in the Dirstack interface file.)

So while I'm pleased with this solution, it does mean that I'll have to continue my hunt for a problem where an object provides a better solution than any other feature of the Ocaml language. This is particularly ironic because when choosing between two strict statically typed languages, Haskell and Ocaml, I chose Ocaml because I thought I needed objects. However, I stuck with Ocaml because of its pragmatism.

Posted at: 07:45 | Category: CodeHacking/Ocaml | Permalink

Sun, 20 Apr 2008

Cross Compiling for Legacy Win32 Systems (Part 2).

Cross compiling from Linux to Windows requires the installation of a couple of packages. On a Debian or Ubuntu system this can be done using:

  sudo apt-get install build-essential
  sudo apt-get install mingw32 mingw32-binutils mingw32-runtime wine

I'm running Ubuntu's Hardy Heron pre-release and the following is known to work with these versions:

  mingw32               4.2.1.dfsg-1ubuntu1
  mingw32-binutils      2.17.50-20070129.1-1
  mingw32-runtime       3.13-1
  wine                  0.9.59-0ubuntu5

For an example of a project which can be successfully cross-compiled, I have chosen libogg which is one of the two libraries required to encode and decode Ogg/Vorbis files. I also happen to know that the current libogg sources in the Xiph Foundation's SVN repository cross-compile from Linux to Windows correctly because I committed the patch to make it possible.

However, we need to look ahead a little. After we have cross compiled libogg we will also want to cross compile the associated libvorbis library which relies on libogg. We therefore need to configure libogg so that when we install it, it can be found by the libvorbis configure script.

For me that meant creating a MinGW32 directory in my home directory:

  mkdir $HOME/MinGW32

The next step to to grab the libogg source code from the Xiph SVN server. This can be achieved using the command:

  svn co http://svn.xiph.org/trunk/ogg libogg

Changing into the libogg directory, we are now ready to configure, test and install the library. That can be done using:

  ./autogen.sh
  ./configure --host=i586-mingw32msvc --target=i586-mingw32msvc \
      --build=i586-linux --prefix=$HOME/MinGW32
  make
  make check
  make install

The first command above, runs the auto tools to generate that configure script. The second command, configure is broken across two lines. It sets up the generated Makefiles to compile Windows binaries from a Linux host, with the install directory we set up before. The third line builds the windows version of libogg, the fourth line runs the test suite, with the windows executables being run under WINE and the final line installs everything in the MinGW32 directory created earlier.

All of the above commands should pass without errors. If they don't, check your versions of of the mingw cross compiler tools and/or WINE.

Posted at: 20:51 | Category: CodeHacking/MinGWCross | Permalink

Fri, 18 Apr 2008

FP-Syd #3.

Wow! Just Wow! I've just arrived home after another FP-Syd meeting.

First up we had Jeremy Apthorp with a short talk titled "My Favourite Bugs (that don't exist in functional languages)". Jeremy gave us all a good reminder of why we were all there. It was also a good warm up the second talk.

Ben Lippmeier, gave us a talk titled "The Disciplined Disciple Compiler" which is something he's working on as part of his PhD at Australian National University. Ben's DDC is a compiler for a Haskell dialect named Disciple that uses effect typing to deal with issues like side effects and destructive updates. Unlike Haskell, Disciple is strictly evaluated with optional, explicit lazy evaluation. This was a really thought provoking talk. Thanks Ben.

I would also like to send out a big thanks to Tom and Scott of Atlassian's Sydney office for providing the venue.

After the meeting we headed off the to Readoak hotel for beers and more FP discussion. By 10:30pm we were just about to leave the Redoak when I ran into Damana who used to work at bCODE and who had organized Sydney Geek Girl get together that same night.

Posted at: 00:44 | Category: FP-Syd | Permalink

Wed, 16 Apr 2008

Cross Compiling for Legacy Win32 Systems (Part 1).

My main two FOSS projects, libsndfile and libsamplerate have significant numbers of users that are tied to that particularly odious legacy system, Microsoft Windows. Since I don't normally use Windows myself, maintaining support for that OS has always been a huge pain in the neck.

Originally I shipped Microsoft project files for libsndfile, but that became unworkable because the different versions of the Microsoft tools (Visual C++ 5, Visual C++ 6, Visual Studio 2003, Visual Studio 2005 etc) used different and incompatible project file formats. I solved this by shipping a simple Makefile that used Microsoft's nmake and the command line compilers to build libsndfile. However, by about 2004, the Microsoft compiler's complete lack of support for the 1999 ISO C Standard made maintaining support too much trouble, so it was dropped.

Instead, I started using Cygwin and MinGW to compile libsndfile on Windows. Both of these tool-sets use a version of the GNU GCC compiler just like Linux and building libsndfile using these two tool-sets was trivial:

  ./configure
  make
  make check

Of course there were howls of protest from Windows users, but since they (with a small number of exceptions) had contributed so little, I didn't fell like I owed them anything. I also started releasing pre-compiled Windows binaries at the same time as the source code tarballs were released.

However, while the MinGW compiler was a huge improvement over the Microsoft one it was still a huge pain in the neck. I had to keep a Windows machine and keep it updated and patched against vulnerabilities. Furthermore, installing and updating MinGW was a painful manual process. Oh how I longed for a Debian/Ubuntu style apt-get command to look for and install updates. Finally, copying source code back and forth between Linux and Windows while debugging Windows issues was another pain point because version control systems like GNU Arch and bzr simply didn't work very well on Windows.

In about 2004, I tried the MinGW Linux to Windows cross compiler, a compiler that runs on Linux but generates binaries for Windows. This compiler worked, but left one rather large problem; how do I run libsndfile's rather large and comprehensive test suite? Compiling libsndfile without running the test suite is a waste of time. I did try to run the tests under WINE (the Windows emulator), but at the time tests were failing under WINE that didn't fail on Windows.

From that time on, I would try running the cross-compiled test suite under WINE once or twice a year. Then, some time in the last year or so, the number of problems with the test suite dropped to one, which was only a FIXME message. A little hacking on the WINE sources resulted in a patch that was sent to the WINE mailing list and has since been applied to the main WINE source tree.

With that bug fixed, I can now cross compile from Linux to Windows and run the full libsndfile test suite under WINE. That means that Windows has just become that little bit less relevant that it was before.

A future post will explain how to set up the cross compiler and WINE and walk through compiling and testing of a standard FOSS project.

Posted at: 23:12 | Category: CodeHacking/MinGWCross | Permalink

Fri, 11 Apr 2008

You Stupid Git!

As far as I can tell, the absolute, canonical, got-to-first documentation for the git distributed version control system (DVCS) can be found here:

http://www.kernel.org/pub/software/scm/git/docs/user-manual.html

This documentation seems comprehensive and well laid out. It explains commits, manipulating-branches, merging, collaborative development and the pretty damn interesting rebase and bisect commands. This documentation is called a user manual but it contains sufficient examples to make it a pretty damn fine tutorial.

Normally something like "here's a link to the documentation" would not be worthy of a blog post. However, failure to find the canonical user manual could lead a person (ie me) to post messages to mailing lists saying things like:

"I'm sure git is very clever and all, but its UI and documentation is probably the most user hateful thing I have seen [since] sendmail's cf files."

or, on finding a one hour long video screen-cast tutorial (apparently aimed at all those Ruby on Rails writing Mac OSX users):

"This makes me wonder, how fscked up does a DVCS have to be that you need tens of megabytes of video to show how it works when Bzr and many others can do it with less than ten kilobytes of html text?"

So while I was wrong about the documentation I still have huge reservations about git's user interface and stand by this statement:

"I am currently trying to learn git and I can see very clearly that git is designed by kernel programmers whose normal approach to a user interface is something like a Unix system call."

I'm sure git is a powerful tool and the rebase feature is something I've been wishing for in other systems for some time, but git's UI is already starting to grate.

Posted at: 20:02 | Category: CodeHacking | Permalink

Sun, 06 Apr 2008

Ocaml : Exception Back Traces in Native Code.

Some time ago I wrote a blog post about exception back traces which at the time of that post only existed for the Ocaml byte code compiler.

However, version 3.10 of the Ocaml compiler which was released about a year ago, included exception back traces for native code as well as byte code. With the imminent release of Ubuntu's Hardy Heron, version 3.10 of the compiler is about to become much more widely available .

Enabling exception back traces is as simple as adding the "-g" option to the ocamlopt command line and then setting a single environment variable as follows.

  export OCAMLRUNPARAM="b1"

Posted at: 12:48 | Category: CodeHacking/Ocaml | Permalink

Wed, 02 Apr 2008

Goodbye People Telecom.

I don't want to be accused of blogorrhea but I really need to get this off my chest.

I've been a customer of People Telecom since they took over Swiftel in 2004. Since this connection is paid for by revenue generated by SecretRabbitCode I have been on one of their most expensive and highest bandwidth plans. I have been a good customer; my bills were paid by direct debit from an account that always had sufficient funds, I rarely go anywhere near my download limits and as a moderately advanced Linux user rarely, if ever, call their technical support.

However, I am about to end my relationship with People Telecom and this is what led up to it:

1. My father needed a new ISP so I suggested that he ring People Telecom. He made the call on the morning of Tuesday, March 25th and signed up. He was told that someone would contact him within 4 days to finalize things.
2. On Monday March 30th, over 4 working days after the initial call he rang them again and was told that they had determined him to be a bad credit risk. Now firstly, it was just plain rude for them to have decided this and not told him and stupid for not informing him and trying to find an alternative payment solution. Secondly, my father is not a bad credit risk. He's a self funded retiree, paid off his mortgage over 15 years ago and has never had a credit card. As a credit risk, you don't get much better than this.
3. After hearing about this, I rang People Telecom at about 10am to see if there was some way to get my father connected. After negotiating their phone robot and listening to their on-hold music I finally spoke to someone and explained the situation. They said I needed to talk the credit department and switched me though, first to listen to the on-hold music and then to voice mail. I left my details and a message asking them to call me.
4. By 6pm that day I still hadn't received a call from the credit department so I called again, negotiated the phone robot, got transferred to the credit department, listened to the on-hold music and then got put through to voice mail. I left a second message.
5. At around 8pm I thought I might try and contact them via their web site to try and get this sorted out. Unfortunately, their web site was down and giving SQL errors.
6. On Tuesday April 1st at about 9:30 am, I rang again, negotiate the phone robot and got put on hold, but this time there is no music. Thinking their phone system was broken I hung up.
7. I try again at about 10am, negotiate the phone robot and got put on hold, but again, no music so again, I hung up. I then tried again to contact them via their web site and this time it worked. Since the web mail form only allowed a limited number of characters I just asked for them to contact me and left my mobile number.
8. By lunch time I still hadn't been contacted so I tried to ring again. I negotiated the phone robot, got put on hold, ignored the fact that there was nothing to indicate that I actually was on hold and finally got through to a person.
9. I explained the situation briefly and the tech support guy on the other end told me I need to talk to the credit department. I explained that I've tried to talk to the credit department and they didn't return my calls. The tech support guy suggested I talk to the billing department, promises I won't get put on hold, puts me though and I go on hold. Silence.
10. After a short period of time a woman picked up the phone and says she has read the case notes but there's not a lot she can do because its a issue with the credit department. I explain my problems with their credit department and ask to be put through to her manager. She agreed and I got put on hold again. Silence.
11. I finally got to talk to a manager named Eric. I explain the situation, explain that I have now spent over an hour of my very valuable time and suggest that its time for People Telecom to get this sorted out.
12. Eric, the manager suggests I put my fathers connection on my bill. I reply that this is not a good solution because my bill is a company bill, not my personal bill and having my father's bill on my company's bill would complicate accounting.
13. I asked if it was possible to direct debit my father's account and if for some reason his account isn't paid, they bill my personal credit card as a fallback. The manager responds that their system isn't able to do this.
14. Finally, I suggest that they need to come up with a decent solution to this or they lose me as a customer.
15. No solution followed so I told Eric the manager that some time over the next couple of months I will be changing ISPs and then I said goodbye.
16. Late Tuesday afternoon I received an email stating that the issue had been escalated to the Manager of the Credit Department and that I should receive a call tomorrow. By the end of Wednesday that call still hadn't come.

I am dumbfounded. I simply cannot remember ever having this kind of run-around and pig-headed intransigence from any other company ever.

I am now searching for a new ISP; one with something like my current 8000k down, 384k up link, about 10 gigs a month with a static IP address and reverse DNS.

Posted at: 20:09 | Category: Tech | Permalink

Sun, 30 Mar 2008

libsamplerate 0.1.3.

About a week ago I released a new version of SecretRabbitCode (aka libsamplerate).

The major change was that the new improved SINC based converters I blogged about here are now the default. There were also a couple of minor bug fixes.

The fine people at Infinitewave have now updated their test results to include the new converter and it shows Secret Rabbit Code comes very close to the best of the commercial converters in terms of quality.

Posted at: 15:11 | Category: CodeHacking/SecretRabbitCode | Permalink

Thu, 27 Mar 2008

FP-Syd #2.

Tonight was the second meeting of fp-syd, the Sydney functional programming group. We had a much more focused group this time with 24 attendees.

First up I gave a short talk on a data structure I had come up with for handling data in a text editor. This data structure was very much inspired by the book, "Purely Functional Data Structures" by Chris Okasaki. All data in the data structure is treated as immutable which means that it is very easy to store past state and to implement undo / redo.

The main talk was split into two parts. First up Manuel Chakravarty gave a talk titled "Monads are Not Scary" and he gave us all an excellent introduction to the topic which I'm pretty sure has convinced all the monad skeptics in the room that monads really aren't anything to be afraid of.

The second part of the main talk, was presented by André Pang who built on Manuel's monad introduction and covered Haskell's Parsec monadic parser combinator library. This too was an excellent talk which explained what seems to be a really elegant tool.

Since the group this week was well over 20 people, we are going to have to look for a bigger venue. Dave Peterson of Orbitec has been a great host for the last two meetings and we would sincerely like to thank him for making his offices available for this bunch of crazy people.

Posted at: 11:54 | Category: FP-Syd | Permalink

Mon, 24 Mar 2008

Cross Compiling with pkg-config.

I'm currently playing with the MinGW cross compiler versions of the GNU C and C++ compilers available via apt-get on Debian and Ubuntu systems. These cross compilers generate windows binaries from a Linux host system which is potentially a much less painful way turning FOSS code into binaries for that particularly odious legacy platform.

Most of the software I'm compiling uses the GNU tools; autoconf, automake, libtool and pkg-config for configuring the software before compiling. Autoconf already has good support for cross compiling and automake and libtool just do what autoconf tells them to do. Pkg-config however is the odd one out.

Pkg-config's job is to retrieve information about installed libraries so that the compiler can find the required header files for inclusion and libraries for linking. For instance, if you wanted compile a program that uses the gconf-2.0 library you could find out the required CFLAGS to be passed to the C compiler and required libraries for linking, by doing something like the following in the Makefile.

  GCONF_CFLAGS = $(shell pkg-config --cflags gconf-2.0)
  GCONF_LIBS = $(shell pkg-config --libs gconf-2.0)

In the above example, when pkg-config is run, it looks in the directory /usr/lib/pkg-config/ and reads information from the file gconf-2.0.pc (each installed library should have one or more of these pkg-config files) which then gets printed out. While the information given by pkg-config would be correct for a native build, it is unlikely to be correct for the cross compiling case.

This issue came up as early as 2003 and there is even a wiki page which suggests some quite extensive changes to pkg-config. Unfortunately I think these suggestions are somewhat fragile and pkg-config itself (I'm using version 0.22) already has features for a better solution.

Like many Unix programs, pkg-config's behaviour can be modified by manipulating certain environment variables. The pkg-config man page explains these variables very well. The first one is PKG_CONFIG_LIBDIR which modifies the default location where pkg-config looks for its per installed library config file. Secondly, the PKG_CONFIG_PATH variable can be set to allow additional pkg-config search paths.

Overriding these two variables results in a MinGW cross pkg-config bash script which I have named i586-mingw32msvc-pkg-config and which looks like this:

  #!/bin/bash

  # This file has no copyright assigned and is placed in the Public Domain.
  # No warranty is given.

  # When using the mingw32msvc cross compiler tools, the native Linux
  # pkg-config executable works fine as long as the default PKG_CONFIG_LIBDIR
  # is overridden.
  export PKG_CONFIG_LIBDIR=/usr/i586-mingw32msvc/lib/pkgconfig

  # Also want to override the standard user defined PKG_CONFIG_PATH with
  # a mingw32msvc specific one.
  export PKG_CONFIG_PATH=$PKG_CONFIG_PATH_MINGW32MSVC

  # Now just execute pkg-config with the given command line args.
  pkg-config $@

Now autoconf generated configure scripts that realise that the i586-mingw32msvc-gcc cross compiler is being used will run the above script and get suitable information for the cross compiler rather than the native compiler.

The only downside to this solution is that a separate script is required for each cross compiler which uses pkg-config. This however is a minor price to pay and it is unlikely that people will end up with huge numbers of XXXX-pkg-config scripts like was common before the widespread use of pkg-config.

Until a better solution becomes available, this is what I will be using.

Posted at: 13:24 | Category: CodeHacking/MinGWCross | Permalink

Sat, 08 Mar 2008

Progress on the Rabbit.

For over three years now, I have been working on (on and off, but mostly off) a new algorithm for doing audio sample rate conversion in Secret Rabbit Code. The idea for the new algorithm has been rattling around in my head for most of that time, but the problem was always the implementation. While I am making progress it has been slow.

However, a public comparison between a large collection of converters showed that while the conversion quality of Secret Rabbit Code was good, it was nowhere near state of the art.

In order to see if I could get Secret Rabbit Code closer to state of the art quickly, I decided to revisit the existing converter during the xmas/new-year break.

The existing converter had a set of digital filters whose coefficients were generated by a small program written in GNU Octave. My first task was to convert that program to Ocaml which has become my favourite language for technical computing. I then spent quite a bit of time finding and analyzing where the filter design program was loosing precision and finding work arounds. Finally, I spent even more time looking at how the different filter design parameters interact with one another and with the conversion algorithm itself.

Fortunately, all this work has paid off. The result is new versions of the SRC_SINC_MEDIUM_QUALITY and SRC_SINC_BEST_QUALITY converters. The old versions of these converters have been renamed to SRC_OLD_SINC_MEDIUM_QUALITY and SRC_OLD_SINC_BEST_QUALITY. The old versions will be removed once the new versions have been fully validated.

So far, the new converters seem to have significantly improved signal to noise ratio as can be seen from the following to spectrograms (using the methodology described here). It should be obvious from these plots that the new versions of the converters have significantly less artifacts (the purple and blue bits) than the old converters.

Obviously, conversion quality is not the only criterion to evaluate sample rate converters; conversion speed can also be important in some situations. In my preliminary testing, the updated Best SINC converter runs up to 25% slower than the old one. The new best converter also uses significantly more memory than the old one. Storage of filter coefficients has gone up by a factor of 20, which is now over a megabyte for best quality converter alone.

In the tables below I've listed the SNR, throughput speeds and bandwidths as measured by the test suite (the snr_bw_test and throughput_test programs) distributed with the code for a couple of different CPU types.

1.1 GHz Intel Pentium M (32 bit) with 2048 KB cache

Converter Name	SNR	Throughput	Bandwidth
SRC_OLD_SINC_MEDIUM_QUALITY	97.46 dB	648800 samples/sec	90.68 %
SRC_SINC_MEDIUM_QUALITY	121.33 dB	593673 samples/sec	90.55 %
SRC_OLD_SINC_BEST_QUALITY	97.35 dB	223025 samples/sec	96.96 %
SRC_SINC_BEST_QUALITY	145.68 dB	163735 samples/sec	96.08 %

1.8 GHz AMD Opteron 265 (64 bit) with 1024 KB cache

Converter Name	SNR	Throughput	Bandwidth
SRC_OLD_SINC_MEDIUM_QUALITY	97.46 dB	1088447 samples/sec	90.68 %
SRC_SINC_MEDIUM_QUALITY	121.33 dB	1088447 samples/sec	90.55 %
SRC_OLD_SINC_BEST_QUALITY	97.35 dB	179116 samples/sec	96.96 %
SRC_SINC_BEST_QUALITY	145.68 dB	187755 samples/sec	96.08 %

1.86GHz Intel Core Duo (32 bit) with 2048 KB cache

Converter Name	SNR	Throughput	Bandwidth
SRC_OLD_SINC_MEDIUM_QUALITY	97.46 dB	1167840 samples/sec	90.68 %
SRC_SINC_MEDIUM_QUALITY	121.33 dB	1042334 samples/sec	90.55 %
SRC_OLD_SINC_BEST_QUALITY	97.35 dB	395102 samples/sec	96.96 %
SRC_SINC_BEST_QUALITY	145.68 dB	302773 samples/sec	96.08 %

A pre-release containing these updated converters is available for download here. Once they have been tested a little more widely I intend to replace the old versions of the converters with the new, higher specification ones.

Anybody who wants to discuss this further should join the SRC mailing list and discuss it there.

Finally, once a version of Secret Rabbit Code with these new converters has been officially released I can get back to the new converter algorithm which should at least match the what I have here in terms of quality but run significantly faster and use at least an order of magnitude less RAM.

Posted at: 14:50 | Category: CodeHacking/SecretRabbitCode | Permalink

Thu, 28 Feb 2008

Inaugural Meeting of FP-Syd.

Tonight was the inaugural meeting of fp-syd, the Sydney functional programming group. Here's a somewhat blurry (sorry, my bad) picture of the attendees.

Across the back from left to right we have; Eric, Ben, Ondrej, Dave, Manuel, Rahul, Shane, André and Ben. At the front, left to right we have Sean, Scott, James and Erik.

I think the night was an absolutely rip roaring success. We met at about 6:30, we each gave an approximately 10 minute each intro of ourselves and our interests in functional program. The meeting then evolved into a general discussion and then at about 8:30 we moved on to Bazaar Beer Cafe for a beer.

Thanks to everyone who attended. It was a huge buzz and I look forward to meeting on a regular basis.

Posted at: 22:51 | Category: FP-Syd | Permalink

Sun, 24 Feb 2008

Functional Programming and Testing.

I read quite a lot of programming related blogs, but its rare for me to find one as muddle headed as this one titled "Quality Begs for Object-Orientation" on the O'Reilly network.

The author, Michael Feathers, starts the post by mentioning that he is dabbling in Ocaml and then makes the assertion that:

"I think that most functional programming languages are fundamentally broken with respect to the software lifecycle."

Now I'm not too sure why he brings up software lifecycle, because all he talks about is testing. However, he does give an example in Java involving testing and wraps up his post by saying that his Java solution is difficult to do in Ocaml, Haskell and Erlang.

Feathers gets two things wrong. Firstly he seems to be writing Java code using Ocaml's syntax and then complains that Ocaml is not enough like Java. His conclusion is hardly surprising. Ocaml is simply not designed for writing Java-like object oriented code.

The second problem is his claim that testing in functional languages is more difficult than with Java. While this may be true when writing Java code with Ocaml's syntax, it is not true for the more general case of writing idiomatic Ocaml or functional code.

So lets look at the testing of Object Oriented code in comparison to Functional code.

With the object orientated approach, a bunch of data fields are bundled up together in an object and methods defined some of which may mutate the state of the object's data fields. When testing objects with mutable fields, its important to test that the state transitions are correct under mutation.

By way of contrast, when doing functional programming, one attempts to write pure functions; functions which have no internal state and where outputs depend only on inputs and constants.

The really nice thing about pure functions is that they are so easy to test. The absence of internal state means that there are no state transitions to test. The only testing left is to collect a bunch of inputs that test for all the boundary conditions, pass each through the function under test and validate the output.

Since testing pure functions is easier that testing objects with mutable state, I would suggest that assuring quality using automated testing is easier for functional code than for object oriented code. This conclusion directly contradicts the title of Feathers' blog post: "Quality Begs for Object-Orientation".

The lesson to be learned here is that if anyone with a purely Java background wants to learn Ocaml or any other functional language, they have to be prepared for a rather large paradigm shift. Old habits and ways of thinking need to be discarded. For Ocaml, that means ignoring Ocaml's object oriented and imperative programming features for as long as possible and attempting to write nothing but pure stateless functions.

Update : 2008-02-26 17:04

Conrad Parker posted this to to reddit and the ensuing discussion was quite interesting.

Posted at: 23:26 | Category: CodeHacking/Ocaml | Permalink

Sat, 23 Feb 2008

Dear Dell

I am customer of yours. I own a Dell Latitude X1 laptop, my wife has an Inspiron 6400 and on my recommendation, my father bought a Dell Dimension 5150 desktop system.

Dell, I think its important for you to know that none of these machines run windows. All of them run Linux exclusively and have from the day they came into our possession. However for all of these systems we had to pay money to Microsoft for operating systems we have never used.

The reason I am writing this now is that the warranty on my Latitude X1 runs out in November of this year. That means that some time between now and November I will be purchasing a new laptop. One candidate for my new machine is the very, very attractive Dell XPS M1330:

When I buy my new laptop, I would like to buy it either with Linux pre-installed or without any OS at all. Since I know Linux very, very well indeed, I do not need Dell to support Linux, just the hardware.

For me as long time, devoted Linux user and developer (professionally as well as FOSS), having to pay money for a Microsoft operating system I don't use is rather offensive. Microsoft has run a long term campaign to thwart the progress of Linux and my money is being used to fund that campaign. For me, this situation in unconscionable.

Apart from the not being able to buy machines without a Microsoft operating system, my experience with Dell has been very good. If I was able to buy a machine from Dell without a Microsoft operating system, buying from Dell would be a no-brainer.

So Dell, here is my challenge for you. Offer me a good looking, high end laptop with lots of juicy options like the XPS with either Linux pre-installed or with no operating system and make sure that these options are reflected in the pricing (ie the Linux or no-OS options should be cheaper than the windows option). If you can do that you are almost guaranteed of keeping me as a customer. If not, I will be looking elsewhere for a company that can meet my requirements. If Dell loses this sale, it will likely lose other sales from my immediate family, friends and employers.

Dell, the ball is in your court. Thanks for listening.

Update : 16:47

Someone posted this to programming.reddit.com and it seems that if you are in the US at least, you can indeed get a Dell XPS with Ubuntu pre-installed. One good thing about the machine offered is that its got an Intel video chipset which I prefer to the Nvidia one. On the downside, its missing some of the options that I can get in the windows version like the solid state disk.

I've just looked again at the Dell Austalia website and I simply cannot find a page where I can get a similar machine. So Dell Australia or Asia/Pacific, what about making Ubuntu an option for the Dell XPS available here in Australia?

Posted at: 10:09 | Category: Tech | Permalink

Tue, 05 Feb 2008

Designing Library APIs.

At Linux.conf.au 2008 I gave a presentation titled "Designing Library APIs: How to Make Users Love Your Library". The presentation went reasonably well and the marvelous people on the LCA08 audio visual team have already posted the Ogg Video.

During the presentation I mentioned a couple of other resources but due to time limitations I never managed to get to that slide. Those resources are:

• Rusty Russell's Interface Simplicity Spectrum.
• Joshua Bloch's "How to Design a Good API and Why it Matters" which is available as a Google video and a set of slides. Although this presentation is geared towards Java, don't dismiss it because the principles are general enough to apply to all languages.

Finally, Brad Hards, one of the attendees at my presentation pointed me to this paper, Designing Qt-Style C++ APIs which seems to be pretty good.

Posted at: 21:40 | Category: | Permalink

Fri, 11 Jan 2008

Microsoft and the POSIX Standard.

POSIX defines a set of functions for retrieving information, such as file size, creation date, last modification date, ownership etc about files on disk. Two commonly used functions in this set are defined as follows:

   int stat (const char *path, struct stat *buf) ;

   int fstat (int filedes, struct stat *buf) ;

The first operates on the filename, and the second operates on a pre-existing file descriptor. Both functions interrogate the file system and fill in a bunch of fields of a struct supplied by the caller.

Microsoft also provides these functions in its C run time library and even documents them here and here. However, stat, the function that is probably more commonly used than fstat, does not always work as expected. In particular, the following scenario:

• Create a new file using the open function.
• Write N bytes to the file descriptor returned by open.
• Call stat on the file using the file name used during file creation.
• Call fstat on the file descriptor returned by open.

On any sane system, the file length field of the struct filled in by stat and fstat calls in the above scenario should be the same. On windows however, they are not, fstat returns the correct value N, while stat returns zero.

With the help from a kind volunteer on the libsndfile-devel mailing list this broken-ness was confirmed on Win2k, WinXP and Vista.

Microsoft; bringing new and improved versions of the DeathStation 9000 to the desk and laptop bag of every programmer. From that wiki page:

DeathStation 9000 (often abbreviated DS9K) is a fictional computer architecture often used as part of a discussion about the portability of computer code (often C code). It is imagined to be as obstructive and unhelpful as possible, whilst still conforming to any relevant standards, deliberately acting unexpectedly whenever possible.

Posted at: 23:21 | Category: Windiots | Permalink

Mon, 17 Dec 2007

Caught up with Anand and Bruce.

I was in London recently and managed to catch up with Anand Kumria and Bruce Badger for a pint and a curry in the Barbican area. I was rather jet-lagged but it was great to catch up with the guys and say hi!

The colors in this picture are a bit weird. Blame Bruce's camera .

Posted at: 22:02 | Category: | Permalink

Tue, 04 Dec 2007

Microsoft and the 1999 ISO C Standard.

Any programmer with a better than passing familiarity with the C programming language would know that the current standard for C is the ISO/IEC 9899:1999 standard published in 1999. If they program on Linux systems or use the GNU C compiler they might also know that while GCC is not fully C99 compliant, it does however come pretty damn close. In particular, many of the things listed as broken on the status page are mostly working (Complex, math.h, stdint.h, inline etc) and the things listed as missing are missing because there is probably not a huge demand for them.

In the world of Microsoft however, Visual Studio still has no serious attempt at support for C99 even eight years after the standard was released. Microsoft claims conformance with the 1989 C standard but seems to show little to no interest in even attempting to pursue conformance with the later standard.

In particular, the Microsoft compiler has the following C99 issues:

• snprintf. Microsoft does have a function _snprintf, but its behavior does not comply with the requirements of C99.
• C99 maths functions. The new standard introduced a bunch of new functions including the lrint family of functions.
• A couple of new header files including <stdint.h>. See here for a fix.
• The inline keyword. Microsoft of course has a non standard __inline keyword instead.
• Pre-defined macros like __func__. Again Microsoft has a non- standard __FUNCTION__ macro.
• Variable length arrays.
• Variadic macros.
• The long long type.

These short comings of the Microsoft compiler can make compiling Free Software written in standards conforming C rather difficult with that compiler.

The one that hits me worst with respect to compiling libsndfile and libsamplerate on windows is the lack of C99 maths functions, in particular lrint and lrintf. My current solution for these functions are inline assembler functions. However, Microsoft has just recently joined the world of 64 bit operating systems and their compiler for that platform supports neither the lrint family of functions nor inline assembler functions. WTF?

However, just today, I found out that the GNU GCC, GNU Binutils and MinGW teams have just released MinGW for windows 64. That should make things a little easier.

Posted at: 21:54 | Category: Windiots | Permalink

Sat, 24 Nov 2007

Ocaml Snippet : Sqlite3.

One of the really nice things about using Ocaml on Debian and Ubuntu is the large number of really well packaged third party libraries.

Most of these libraries are also well documented from doc strings extracted from the source code files using ocamldoc. However, the documentation for most ocaml libraries is purely reference documentation and its not always obvious how to use the library simply from reading the reference docs. What's really needed is example code to be read in conjunction with the reference docs.

I'm working on a program where I needed a small, fast easy to administer database. With those requitements, Sqlite is really hard to beat and best of all, someone has already written Ocaml bindings. On Debian or Ubuntu, the Ocaml Sqlite bindings can be installed using:

  sudo apt-get install libsqlite3-ocaml-dev

In order to get a feel for using it and take my first steps into the world of SQL (which I'd had very minimal exposure to before now), I wrote a small program to test out the features provided by the library.

The following stand alone program should be taken as an example of how to access a Sqlite database from Ocaml. Since I am not an SQL expert, the actual SQL usage should be taken with a grain of salt.

  exception E of string

  let create_tables db =
      (* Create two tables in the database. *)
      let tables =
      [    "people", "pkey INTEGER PRIMARY KEY, first TEXT, last TEXT, age INTEGER" ;
          "cars", "pkey INTEGER PRIMARY KEY, make TEXT, model TEXT" ;
          ]
      in
      let make_table (name, layout) =
          let stmt = Printf.sprintf "CREATE TABLE %s (%s);" name layout in
          match Sqlite3.exec db stmt with
          |    Sqlite3.Rc.OK -> Printf.printf "Table '%s' created.\n" name
          |    x -> raise (E (Sqlite3.Rc.to_string x))
      in
      List.iter make_table tables


  let insert_data db =
      (* Insert data in both the tables. *)
      let people_data =
      [    "John", "Smith", 23;
          "Helen", "Jones", 29 ;
          "Adam", "Von Schmitt", 32 ;
          ]
      in
      let car_data =
      [    "bugatti", "veyron" ;
          "porsche", "911" ;
          ]
      in
      let insert_people (first, last, age) =
          (* Use NULL for primary key and Sqlite will generate a unique key. *)
          let stmt = Printf.sprintf "INSERT INTO people values (NULL, '%s', '%s', %d);"
                                     first last age
          in
          match Sqlite3.exec db stmt with
          |    Sqlite3.Rc.OK -> ()
          |    x -> raise (E (Sqlite3.Rc.to_string x))
      in
      let insert_car (make, model) =
          let stmt = Printf.sprintf "INSERT INTO cars values (NULL, '%s', '%s');"
                                     make model
		  in
          match Sqlite3.exec db stmt with
          |    Sqlite3.Rc.OK -> ()
          |    x -> raise (E (Sqlite3.Rc.to_string x))
      in
      List.iter insert_people people_data ;
      List.iter insert_car car_data ;
      print_endline "Data inserted."


  let list_tables db =
      (* List the table names of the given database. *)
      let lister row headers =
          Printf.printf "    %s : '%s'\n" headers.(0) row.(0)
      in
      print_endline "Tables :" ;
      let code = Sqlite3.exec_not_null db ~cb:lister
                          "SELECT name FROM sqlite_master;"
      in
      (    match code with
          |    Sqlite3.Rc.OK -> ()
          |    x -> raise (E (Sqlite3.Rc.to_string x))
          ) ;
      print_endline "------------------------------------------------"


  let search_callback db =
      (* Perform a simple search using a callback. *)
      let print_headers = ref true in
      let lister row headers =
          if !print_headers then
          (    Array.iter (fun s -> Printf.printf "  %-12s" s) headers ;
              print_newline () ;
              print_headers := false
              ) ;
          Array.iter (Printf.printf "  %-12s") row ;
          print_newline ()
      in
      print_endline "People under 30 years of age :" ;
      let code = Sqlite3.exec_not_null db ~cb:lister
                                 "SELECT * FROM people WHERE age < 30;"
      in
      match code with
      |    Sqlite3.Rc.OK -> ()
      |    x -> raise (E (Sqlite3.Rc.to_string x))



  let search_iterator db =
      (* Perform a simple search. *)
      let str_of_rc rc =
          match rc with
          |    Sqlite3.Data.NONE -> "none"
          |    Sqlite3.Data.NULL -> "null"
          |    Sqlite3.Data.INT i -> Int64.to_string i
          |    Sqlite3.Data.FLOAT f -> string_of_float f
          |    Sqlite3.Data.TEXT s -> s
          |    Sqlite3.Data.BLOB _ -> "blob"
      in
      let dump_output s =
          Printf.printf "  Row   Col   ColName    Type       Value\n%!"  ;
          let row = ref 0 in
          while Sqlite3.step s = Sqlite3.Rc.ROW do
              for col = 0 to Sqlite3.data_count s - 1 do
                  let type_name = Sqlite3.column_decltype s col in
                  let val_str = str_of_rc (Sqlite3.column s col) in
                  let col_name = Sqlite3.column_name s col in
                  Printf.printf "  %2d  %4d    %-10s %-8s   %s\n%!"
                                 !row col col_name type_name val_str ;
                  done ;
              row := succ !row ;
              done
      in
      print_endline "People over 25 years of age :" ;
      let stmt = Sqlite3.prepare db "SELECT * FROM people WHERE age > 25;" in
      dump_output stmt    ;
      match Sqlite3.finalize stmt with
      |    Sqlite3.Rc.OK -> ()
      |    x -> raise (E (Sqlite3.Rc.to_string x))


  let update db =
      print_endline "Helen Jones has just turned 30, so update table." ;
      print_endline "Should now only be one person under 30." ;
      let stmt = "UPDATE people SET age = 30 WHERE " ^
                      "first = 'Helen' AND last = 'Jones';"
      in
      (    match Sqlite3.exec db stmt with
          |    Sqlite3.Rc.OK -> ()
          |    x -> raise (E (Sqlite3.Rc.to_string x))
          ) ;
      search_callback db


  let delete_from db =
      print_endline "Bugattis are too expensive, so drop that entry." ;
      let stmt = "DELETE FROM cars WHERE make = 'bugatti';" in
      match Sqlite3.exec db stmt with
      |    Sqlite3.Rc.OK -> ()
      |    x -> raise (E (Sqlite3.Rc.to_string x))


  let play_with_database db =
      print_endline "" ;
      create_tables db ;
      print_endline "------------------------------------------------" ;
      list_tables db ;
      insert_data db ;
      print_endline "------------------------------------------------" ;
      search_callback db ;
      print_endline "------------------------------------------------" ;
      search_iterator db ;
      print_endline "------------------------------------------------" ;
      update db ;
      print_endline "------------------------------------------------" ;
      delete_from db ;
      print_endline "------------------------------------------------"


  (* Program main. *)

  let () =
      (* The database is called test.db. Delete it if it already exists. *)
      let db_filename = "test.db" in
      (    try Unix.unlink db_filename
          with _ -> ()
          ) ;

      (* Create a new database. *)
      let db = Sqlite3.db_open db_filename in

      play_with_database db ;

      (* Close database when done. *)
      if Sqlite3.db_close db then print_endline "All done.\n"
      else print_endline "Cannot close database.\n"

The above code can be run as a script using:

  ocaml -I +sqlite3 sqlite3.cma unix.cma sqlite_test.ml

or compiled to a native binary using:

  ocamlopt -I +sqlite3 sqlite3.cmxa unix.cmxa sqlite_test.ml -o sqlite_test

When run, the output should look like this:

  Table 'people' created.
  Table 'cars' created.
  ------------------------------------------------
  Tables :
      name : 'people'
      name : 'cars'
  ------------------------------------------------
  Data inserted.
  ------------------------------------------------
  People under 30 years of age :
    pkey          first         last          age
    1             John          Smith         23
    2             Helen         Jones         29
  ------------------------------------------------
  People over 25 years of age :
    Row   Col   ColName    Type       Value
     0     0    pkey       INTEGER    2
     0     1    first      TEXT       Helen
     0     2    last       TEXT       Jones
     0     3    age        INTEGER    29
     1     0    pkey       INTEGER    3
     1     1    first      TEXT       Adam
     1     2    last       TEXT       Von Schmitt
     1     3    age        INTEGER    32
  ------------------------------------------------
  Helen Jones has just turned 30, so update table.
  Should now only be one person under 30.
  People under 30 years of age :
    pkey          first         last          age
    1             John          Smith         23
  ------------------------------------------------
  Bugattis are too expensive, so drop that entry.
  ------------------------------------------------
  All done.

Posted at: 14:20 | Category: CodeHacking/Ocaml | Permalink

Categories

• • Top (115)
  • CodeHacking (51)
  • • Erlang (1)
    • MinGWCross (3)
    • Ocaml (16)
    • SecretRabbitCode (4)
    • libsndfile (5)
  • Cycling (4)
  • FP-Syd (9)
  • LCA07 (2)
  • Music (3)
  • NewHouse (2)
  • OSDC06 (3)
  • Security (1)
  • Tech (11)
  • Tools (1)
  • WTF (4)
  • Windiots (7)

Syndication

• Atom
• RSS 2.0

Planets

• Planet Ocamlcore
• Planet SLUG
• Planet LA

Links

• Ocaml
• Erlang
• Hack A Day
• Create Digital Music
• Linux Rock Star
• The Daily WTF
• The Nedit Text Editor
• Make Blog
• Math Forge
• CSound Blog
• Reddit Science
• Audio Cookbook

• Jihad Watch
• Dhimmi Watch
• Gates of Vienna
• Jesus and Mo
• Kafir Canada
• Media Mythbusters
• Butterflies and Wheels
• The Fjordman Files

File Validators

• HTML Validator
• CSS Validator
• RSS Feed Validator