Computer Vision using Ruby and libJIT

The RubyConf 2009 took place near San Francisco. I gave a talk about Computer vision using Ruby and libJIT. Here is the abstract of the talk:

Ruby originated in Japan, the country which is world-leading in robotic research. It suggests itself to put the two together and to start using Ruby as a language to program robots. However at the moment the performance of available Ruby interpreters is not sufficient. It is hard to achieve performance comparable to compiled C++-code since manipulation of Ruby-integers and Ruby-arrays requires frequent bounds-checking. It can be shown that universal bounds-check elimination is actually impossible.

This talk presents HornetsEye which is a Ruby-extension facilitating the development of real-time machine vision algorithms for future robotic applications. HornetsEye offers I/O facilities to capture and display videos. HornetsEye also can be integrated into GUI-applications developed with Qt4-QtRuby. Furthermore there is a set of Ruby classes which provides the means to compose native datatypes and specify operations on them. The libJIT just-in-time compiler is used to achieve real-time performance. The project was inspired by NArray and ruby-libjit.

The software is called HornetsEye and it is available under the terms and conditions of the GPL.

Feel free to leave comments below.

Discussion

Lazy computation

Future work on HornetsEye could be about using lazy computation to avoid the creation of intermediate results when doing array operations. Here’s a small prototype implementation. Let me know if you have any suggestions.

class Promise
  def initialize( size, &action )
    @size, @action = size, action
  end
  def inspect
    "Promise(#{@size})"
  end
  def []( index )
    @action.call index
  end
  def demand
    Seq[ *( 0 ... @size ).collect { |i| @action.call i } ]
  end
  def -@
    Promise.new( @size ) { |i| -@action.call( i ) }
  end
  def fill!( value = 0 )
    @action = proc { |i| value }
    self
  end
end

class Seq
  class << self
    def []( *args )
      retval = new args.size
      retval.data = args
      retval
    end
    def new( size )
      unless Thread.current[ :lazy ]
        super size
      else
        Promise.new( size ) { 0 }
      end
    end
  end
  attr_accessor :data
  def initialize( size )
    @data = [ 0 ] * size
  end
  def inspect
    "Seq#{@data.inspect}"
  end
  def []( index )
    @data[ index ]
  end
  def -@
    Seq[ *@data.collect { |x| -x } ]
  end
  def fill!( value = 0 )
    @data = [ value ] * @data.size
    self
  end
end

def lazy
  previous_state = Thread.current[ :lazy ]
  Thread.current[ :lazy ] = true
  retval = yield
  Thread.current[ :lazy ] = previous_state
  if retval.is_a? Array
    retval.collect { |x| x.demand }
  else
    retval.demand
  end
end
# Ordinary computation
a = -Seq.new( 3 ).fill!( 1 )
# Simple lazy computation
b = lazy { -Seq.new( 3 ).fill!( 1 ) }
# ...

Reflection

Using the following code one can use reflection to interpret Ruby code in order to for example translate it to machine code later.

class Const
  attr_accessor :inspect
  alias_method :to_s, :inspect
  def initialize( s )
    @inspect = s.to_s
  end
  def method_missing( name, *args )
    str = "#{ self }.#{ name }"
    unless args.empty?
      str += "( #{args.join ', '} )"
    end
    Const.new str
  end
  def coerce( y )
    return Const.new( y ), self
  end
end
a = Const.new 'a'
# a
b = Const.new 'b'
# b
-a
# a.-@
a + b
# a.+( b )
a[ 2 ]
# a[]( 2 )
2 * a
# 2.*( a )
2 * a + b
# 2.*( a ).+( b )
2 * ( a + b )
# 2.*( a.+( b ) )

Unfortunately Ruby reflection does not capture creation of variables, assignment of values, and control structures such as while or if-then-else.

GPU

One could use “monkey patching” to add GPU support (using OpenCL) in a transparent way. Furthermore it would be desirable to make use of multi-core computers. Ruby 1.9 supports native threads but there is a global interpreter lock on extension calls. So multi-threading has to be implemented explicitely if it is to be used in such a Ruby extension.

Technical Issues

It was suggested to me to

• make the array classes available separately
• to use Rubygems for packaging the software and Gemcutter for publishing the software
• to use Git for version control and to host the repositories on Github
• to use ffi so that the extension works with different Ruby VMs (JRuby, Rubinius, …)

Update:

The video of the talk is now available!