Fri, 05 Jun 2009

Jim Henson Co.

I received an email this morning: 


  From: David XXX <xxx@creatureshop.com>
  To: <erikd@xxx.com>
  Subject: libsndfile usage
  Date: Thu, 4 Jun 2009 10:52:02 -0700
  User-Agent: Mozilla-Thunderbird 2.0.0.19 (X11/20090103)

  Erik, just thought I would mention we have been using libsndfile
  internally at the Jim Henson Co. for a while, we recently got approval
  to open source some of our utilities which mostly use libsndfile to read
  in wav files for encoding into a quicktime file, using libquicktime. We
  released them as usage examples and extras for libquicktime.

  http://libquicktime.cvs.sourceforge.net/viewvc/libquicktime/lqt_utils/

  --
  David XXX
  Pipeline Tools Programmer
  Jim Henson Creature Shop
  xxx@creatureshop.com

This is so cool.

Of course Jim Henson and Company are probably best known for the Muppets and Sesame Street but a couple of years ago my then four year old daughter absolutely loved watching Bear in the Big Blue House.

In a subsequent email David explained that the Bear program was before his time, and that libsndfile was being used as part of the render process in the production of Sid the Science Kid which I don't think has reached this country yet.

Posted at: 21:06 | Category: CodeHacking/libsndfile | Permalink

Thu, 14 May 2009

libsndfile 1.0.20.

There's a new release of libsndfile available in the usual place. This is a security bug fix release which fixes a potential heap overflow in VOC files found and reported by Tobias Klein ( http://www.trapkit.de/ ) and another in the AIFF file parser found by me.

I am also making available patches for older versions of libsndfile:

Hopefully the next release will contain new features instead of just bug fixes.

Posted at: 20:53 | Category: CodeHacking/libsndfile | Permalink

Tue, 03 Mar 2009

libsndfile 1.0.19.

There's a new release of libsndfile available in the usual place. There is also a back story about this release.

At about the same time as my blog post entitled "Security Hyperventilating" I released version 1.0.18 of libsndfile. A couple of days later I received an email from Alin Rad Pop of Secunia Research (the company who got it right last time) informing me of a real, potentially exploitable bug in my newly released code. I've been told that this security vulnerability will be made public over the next couple of days as CVE-2009-0186.

While I certainly didn't believe my code was bug free I have worked very hard to reduce the bugs as much as possible. I have a full and rather comprehensive test suite, I run valgrind over the code regularly, I have learnt from past mistakes and I when I find one bug I nearly always take the time to search for other instances of the same class of bug in the code.

I even wrote a program to do automated Fuzz Testing. This program takes an existing sound file, modifies it, writes it to disk and then runs it through libsndfile. If libsndfile segfaults or does anything else wildly wrong, the problem file is saved for later review. That review usually results in a bug fix.

In spite of all this testing, there was still a security vulnerability. Thats mainly because libsndfile is one of those projects that needs to parse untrusted binary data. Remember all those exploitable bugs against libjpeg and libgif in the late 1990s? Well both of those projects only parse one file type; libsndfile parses more than a dozen. The only other project I can think of with large numbers of different things to parse is the Samba Project which needs to parse dozens of different kinds of messages in the CIFS protocol.

Of course there are other tools to for finding bugs; static analysis tools. In this field there are FOSS products like Sparse and Splint. The first is rather new and developed specifically to find bugs in Linux kernel source code and the second is basically unmaintained. Both are rather intrusive in that they require special annotations to help them ignore valid code and find buggy code.

Neither of these FOSS programs compare well against commercial offerings like Coverity's Prevent but Coverity is widely regarded as being rather expensive. However, for FOSS projects, Coverity does have a program where it scans FOSS projects and feeds the scan results back to the projects so they can fix bugs.

libsndfile was added to Coverity scan of FOSS projects well over a year ago. I fixed all of the issues (mostly minor) pretty much immediately and then asked to go on to the next rung of the ladder. Unfortunately, I was told that I had to wait for all the projects that were currently on rung 0 to fix their issues before that could happen. Eventually, that did happen, but I don't remember anyone contacting me about it. This slow progress was frustrating.

However, on hearing that there was a CVE about to be published against libsndfile I was fortunate enough to have someone offer to run libsndfile through one of the commercial static analysis tools. The result of that was a report containing 68 warnings, split into four roughly equally sized groups:

The interesting thing about the above test was that the bug that resulted in the CVE was still present in the code analysed by static analysis tool, but was not found. So while this tool did find a bunch of errors it is still not able to find every error. Obviously there is room for improvement here, both in my code and in the static analysis tool.

Posted at: 19:53 | Category: CodeHacking/libsndfile | Permalink

Sun, 15 Feb 2009

Ten Years of libsndfile.

Today, February 15th 2009, is the ten year anniversary of the first release of libsndfile.

Like most FOSS projects, libsndfile started off as an urge to scratch an itch. I was interested in Digital Signal Processing (DSP) and wanted an easy way to get digital audio into and out of software I was writing to try out various DSP algorithms. Secondly, I wanted to a sound file editor and one important part of such an editor is an ability to read and write various sound file formats. I did however look at a couple of existing sound file editors and found that most of them available at the time had buggy and incorrect WAV file handling. So I started out getting that part fixed. Nowadays, most sound file editors on Linux and many on other platforms use libsndfile for file I/O.

In its 10 years of existence, libsndfile has grown from some 5000 lines of code to over 45000 lines of code (not counting the test suite and the example programs). The earliest versions could read WAV, AIFF and AU file formats while the latest version supports 25 formats and is still growing.

It was originally written to run on Linux and other UNIX-like systems but soon ended up running on windows and Mac OS9 (the old non-Unix Apple Macintosh) operating system. Fortunately Mac OS9 has been assigned to the dustbin of history leaving windows as the only operating system that was difficult or painful to support. Recently, the windows development has moved to a system where the only way I support building of libsndfile for that OS is to cross compile from Linux, with the test suite being run under Wine. This has made my life significantly easier since I also release pre-compiled windows binaries.

One surprise for me was that a Wikipedia entry as added in 2006. The page says:

"libsndfile is a widely-used [citation needed] C library"

and I think that the ten year anniversary of the first release may be a good time to look at where libsndfile is actually being used. With a little research and some help from the libsndfile mailing lists, this is what I found (in no particular order):

On top of that there are language bindings for Fortran 77, Free Pascal, Haskell, Perl, Python ( PySndfile, AudioLab, libsndfile-python and possibly others ), Ruby and probably many others.

Overall its been a fun ten years. I've learnt a lot about writing reliable and portable cross platform code and become a much better coder for it. Having libsndfile as a hobby project has definitely helped my employment prospects and my career as a professional software engineer.

The next ten years of libsndfile will mainly be maintenance, but new file formats (I'm currently working on Ogg/Speex) and features will be added as needed.

Posted at: 11:31 | Category: CodeHacking/libsndfile | Permalink

Sat, 07 Feb 2009

libsndfile 1.0.18.

Over two years since the last one I've finally managed to do a new release of libsndfile. The changes are:

The Win32 installer and binaries have been tested under both Wine and Windows XP. The Win64 installer and binaries have received much less testing and should be considered alpha quality.

For those compiling from source, it should be noted that the configure script requires libvorbis version 1.2.1 or greater which is currently only available from Xiph SVN.

Posted at: 17:01 | Category: CodeHacking/libsndfile | Permalink

Thu, 31 Aug 2006

libsndfile 1.0.17.

I've just released a new version of libsndfile. The changes are:

Posted at: 22:19 | Category: CodeHacking/libsndfile | Permalink

Thu, 10 Aug 2006

C++ Wrapper for libsndfile, Part 4.

I got some email from Jaq who said:

"You don't actually give a good reason for not wanting a close() method on SndfileHandle; your reasons for not wanting open() are good, but surely a handle needs a way of closing itself? Or does the API overload object deletion with the closing of the handle too?"

So first of all, I should have stated that my current candidate header file is available here. The SndfileHandle object is designed to be used with the Resource Acquisition Is Initialization pattern. In particular, the object will close the file and release all allocations when it goes out of scope. For instance: 

  {
      SndfileHandle file ("foo.wav") ;

      // Do something with file which gets closed automatically
      // when file goes out of scope.
  }

So what's the problem with the close() method? Well, its very similar to the problems with the open() method. Lets look at an example: 

  SndfileHandle file1 ("foo.wav") ;

  // Make file2 == file1
  SndfileHandle file2 = file1 ;

  // Close file1
  file1.close ("bar.wav") ;

Obviously, the handle associated with file1 should be closed, but what about file2? Should that be closed or remain open?

The fact that its not obvious means that its best left out. If anyone really wants to make sure that a SndfileHandle is closed they can do: 

  SndfileHandle file1 ("foo.wav") ;

  // Make file2 == file1
  SndfileHandle file2 = file1 ;

  // Close file1
  file1 = SndfileHandle () ;

In this case its a little more obvious that file1 and file2 now refer to different handles.

Posted at: 20:54 | Category: CodeHacking/libsndfile | Permalink

Wed, 09 Aug 2006

C++ Wrapper for libsndfile, Part 3.

Thinking about the C++ wrapper continues. In the current candidate version, a SndfileHandle contains a pointer to a private reference counted struct which contains the actual data. The SndfileHandle class also has a copy constructor and an assignment operator.

In addition to the above, a number of people on the mailing list have asked for the SndfileHandle class to have open() and close() methods. This seems reasonable on the face of it, but Daniel Schmitt points out that the combination of copy/assign and open/close results in a rather strange ambiguity.

Daniel gives the following example using copy/assign (ie no open/close methods): 

  SndfileHandle file1 ("foo.wav") ;

  // Make file2 == file1
  SndfileHandle file2 = file1 ;

  // Now reuse file1
  file1 = SndfileHandle("bar.wav");

At the end of that block of code we now have file1 and file2 operating on different handles, which is exactly what any reasonable programmer would expect.

Now look at what happens if we have open/close methods: 

  SndfileHandle file1 ("foo.wav") ;

  // Make file2 == file1
  SndfileHandle file2 = file1 ;

  // Now reuse file1
  file1.open ("bar.wav") ;

The open method can be implemented in one of the following two ways :

After the block of code above, the two different implementations would result in the following state :

Obviously, the second implementation is completely wrong, but the first implementation is at least questionable. In terms of providing something which balances utility and consistency I'm tending to favor the idea of keeping the copy/assign operations and not providing open/close methods.

Posted at: 20:20 | Category: CodeHacking/libsndfile | Permalink

Mon, 31 Jul 2006

C++ Wrapper for libsndfile, Part 2.

After yesterday's blog post a guy in Germany, Daniel Schmitt, piped up on the libsndfile mailing lists and insisted I reconsider the C++ wrapper class' copy/assign issue. My big problem with copy/assign were that they would not behave the way people might reasonable expect them to.

Daniel's major contribution was renaming the class from Sndfile to SndfileHandle. Once that is done, having a copy constructor and an assignment operator using reference counting makes sense. With the class name containing the word "Handle", the name now fits the behavior. This is such a minor and seemingly trivial change but I simply didn't see it.

Thanks Daniel. Brilliant!

Posted at: 19:48 | Category: CodeHacking/libsndfile | Permalink

Sun, 30 Jul 2006

A C++ Wrapper for libsndfile.

Over the years I've received a bunch emails saying stuff like "why did you write libsndfile in that old fashioned C language instead of nice modern shiny C++?". Obviously anyone who even thinks something like this is too ignorant of C to be a good C++ programmer. A competent C++ programmer needs to know and be comfortable with the whole of the C language as well as the whole of C++.

At the time I started work on libsndfile in 1998 I was writing far more C++ code than C code. However, back then, the GNU C++ compiler was nowhere near as good as it is today and I thought a C library interface was a safer bet than C++. In retrospect, I believe the decision of using C was spot on, for the following reasons:

However, some people do prefer C++ to C and many of those would probably be writing their own C++ wrapper. Since the vast majority of these wrappers would largely the same, it makes sense for me to distribute a C++ wrapper with libsndfile.

I decided on the following set of criteria for the wrapper:

It does however use templates for the read/write/readf/writef methods: 

  template <typename T> sf_count_t read   (T *ptr, sf_count_t items) ;
  template <typename T> sf_count_t readf  (T *ptr, sf_count_t frames) ;
  template <typename T> sf_count_t write  (const T *ptr, sf_count_t items) ;
  template <typename T> sf_count_t writef (const T *ptr, sf_count_t frames) ;

with explicit specializations for types short, int, float and double.

It also explicitly makes the copy constructor and assignment operator private. The problem with these two is that two objects wrapping the same SNDFILE* pointer will not give the expected behavior. With the C version: 

  SNDFILE *file1 = sf_open (filename, ...);
  SNDFILE *file2 = file1 ;

anyone reading the code can see that file1 and file2 are two pointers pointing to the same object. The code reader knows what behavior to expect here.

Now contrast this with the C++ version: 

  Sndfile file1 (filename, ...) ;
  Sndfile file2 (file1) ;

The objects file1 and file2 look like two independent objects and should behave like two independent objects, but instead they behave like they do in the C version above. I believe that this is inconsistent.

The only solution that would maintain consistency would be to make the copy constructor do a deep copy but that is simply too much of a pain in the neck to implement.

The current version of the wrapper is available here or in verision 1.0.17 or later of libsndfile.

Posted at: 18:26 | Category: CodeHacking/libsndfile | Permalink