Monday, February 7, 2011

A new GStreamer plugin

Some time ago, I bumped into a dead end when trying to implement the face-tracking overlays feature. The problem started when I tried to set the overlay width and height in run time.

The opencv facedetect gst-plugin sends bus messages with the x, y, width and height of the faces it detects. I captured the messages and set the rsvgoverlay element x and y properties without problems. But rsvgoverlay element lacks width and height properties.

This meant that if you moved your face away or close from the camera, the mask would still look the same size, or if the svg dimensions were huge or tiny, it would look that way instead of resizing to the face size.

I contacted rsvgoverlay maintainer Olivier Aubert to see if width and height could be added as properties for that element. He preferred not to and explained to me the reasons. (He pointed me to this bug where coincidentally Luciana had asked for something similar before our internships in GNOME Outreach program started, and she implemented a plugin for png overlays already, I haven't checked it yet).

Well, Olivier suggested I used the SVG data itself to set size and position. So I started implementing this:

case GST_MESSAGE_ELEMENT:
{
  if (strcmp (gst_structure_get_name (message->structure), "facedetect") == 0)
  {
    CheeseCameraPrivate *priv = CHEESE_CAMERA_GET_PRIVATE (camera);
    
    guint x, y, width, height;
    gchar *svg_data;
    const GstStructure *face;
    int face_count;
    
    /* TODO: get coordinates of multiple faces: 
       for (i=0; i < gst_value_list_get_size (gst_structure_get_value (message->structure, "faces")); i++) { ... }
    */
    face_count = gst_value_list_get_size (gst_structure_get_value (message->structure, "faces"));
    /* The last face in the list seems to be the right one, objects mistakenly 
     * detected as faces for a couple of frames seem to be in the list 
     * beginning. TODO: needs confirmation. */
    face = gst_value_get_structure (gst_value_list_get_value (gst_structure_get_value (message->structure, "faces"), face_count-1));
    gst_structure_get_uint (face, "x",      &x);
    gst_structure_get_uint (face, "y",      &y);
    gst_structure_get_uint (face, "width",  &width);
    gst_structure_get_uint (face, "height", &height);
             
    /* TODO: allow different positioning according to type of overlay
    if hat
    if beard
    x = the face right temple, so think a way to allow hats to go above, 
    beards to go midway below, etc.
    */
    
    /* TODO: ViewBox size affects anything? */
    svg_data = g_strdup_printf ("<svg viewBox=\"0 0 800 600\"><image x=\"%u\" y=\"%u\" width=\"%u\" height=\"%u\" xlink:href=\"%s\" /></svg>",
                                x, 
                                y, 
                                width, 
                                height, 
                                "/path/to/foo.svg");
    /* FIXME: Problem is right here ^^^^^^^^^^^^^^ I have no way of knowing the 
     * file path on run time with what we currently have. */
    
    /* The rsvgoverlay element in face recognition + svg overlay effects 
     * must be named "face_overlay" for this to work. Not very nice. 
     * TODO: iterate effect_filter bin elements looking for the rsvgoverlay 
     * element */
    GstElement *overlay_element = gst_bin_get_by_name (GST_BIN (priv->effect_filter), "face_overlay");                    
    g_object_set (overlay_element, "data", svg_data, NULL);
    
    g_object_unref (overlay_element);
    g_free(svg_data);
  }
}
break;


Then I realized: for this to work, I needed the SVG file path and we don't have a way to know it on runtime right now. It is parsed as part of the pipeline description from the .effect files in gnome-video-effects and the entire pipeline description is used to create the rsvgoverlay element. The same applies to all other effects. So to access that path, we would have to start knowing which effect we are handling at a specific time inside of cheese, and we don't want to do that.

Other option would be to parse the .effect files in a different way. That's something that is already being discussed here and it will allow for more functionality than I need for this.

So I started writing my very own GStreamer plugin, to have more control and manage several options and use cases better. Maybe I'll be using the existing rsvgoverlay and facedetect plugins somehow.

Last days I was following the steps in GStreamer Plugin Writer's Guide and climbing the vertical autotools learning curve. Hoo boy.


About Autotools:

Task #1: add a subdirectory named "pixmaps" with .svg images to gnome-video-effects to be installed in /usr/share/gnome-video-effects/pixmaps when running make install.

The files to edit were: gnome-video-effects/configure.ac and gnome-video-effects/Makefile.am

diff --git a/Makefile.am b/Makefile.am
index 8638fc8..b5b52fb 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1,4 +1,4 @@
-SUBDIRS = effects
+SUBDIRS = effects pixmaps
 
 DIST_SUBDIRS = $(SUBDIRS) po
 
diff --git a/configure.ac b/configure.ac
index 499b6f2..9d33ffb 100644
--- a/configure.ac
+++ b/configure.ac
@@ -23,5 +23,6 @@ AC_OUTPUT([
 Makefile
 gnome-video-effects.pc
 effects/Makefile
+pixmaps/Makefile
 po/Makefile.in
 ])

And a new Makefile.am needs to be added in the new pixmaps directory:

pixmapsdir = $(datadir)/gnome-video-effects/pixmaps

pixmaps_DATA = \
  kiss.svg \
  snow.svg \
  birthday-card.svg \
  microphone.svg
    
EXTRA_DIST = \
    $(pixmaps_DATA)

It looks simple now, but finding out how to do this, from which files to edit to what to add there, was a guessing-in-the-dark ordeal.

Task #2: include <cv.h> in my new plugin header file. Run "make" without errors.
Holy tap-dancing christ with crutches. This is so easy to do in Visual Studio. Just add the include path and the library to link against. But what to say about autotools that hasn't been said already? (With this I'm not saying that everything about programming in Windows is better, absolutely on the contrary: programming in linux is paradise compared to that. But creating a new little project to tinker and test things quickly? Yes, that is lacking in linux that I know of).

Files that needed to be edited: configure.ac in top-level directory, and Makefikle.am in src directory.

Here I include the entire Makefile.am file with added parts in blue:


# Note: plugindir is set in configure

##############################################################################
# TODO: change libgstplugin.la to something else, e.g. libmysomething.la     #
##############################################################################
plugin_LTLIBRARIES = libgstfaceoverlay.la

##############################################################################
# TODO: for the next set of variables, name the prefix if you named the .la, #
#  e.g. libmysomething.la => libmysomething_la_SOURCES                       #
#                            libmysomething_la_CFLAGS                        #
#                            libmysomething_la_LIBADD                        #
#                            libmysomething_la_LDFLAGS                       #
##############################################################################

# sources used to compile this plug-in
libgstfaceoverlay_la_SOURCES = gstfaceoverlay.c gstfaceoverlay.h

# compiler and linker flags used to compile this plugin, set in configure.ac
libgstfaceoverlay_la_CFLAGS = $(GST_CFLAGS) $(OPENCV_CFLAGS)
libgstfaceoverlay_la_LIBADD = $(GST_LIBS) $(OPENCV_LIBS) 
libgstfaceoverlay_la_LDFLAGS = $(GST_PLUGIN_LDFLAGS)
libgstfaceoverlay_la_LIBTOOLFLAGS = --tag=disable-static

# headers we need but don't want installed
noinst_HEADERS = gstfaceoverlay.h

Such a simple starter Makefile.am with instructions, such a small change to add. But to discover what had to be changed!

And to configure.ac I added:

dnl *** opencv ***
translit(dnm, m, l) AM_CONDITIONAL(USE_OPENCV, true)

  dnl we specify a max. version too because we set CV_NO_BACKWARD_COMPATIBILITY
  dnl and don't want the build to break when a new opencv version comes out.
  dnl Need to adjust this upwards once we know that our code compiles fine with
  dnl a new version and the no-backward-compatibility define. (There doesn't
  dnl seem to be a switch to suppress the warnings the cvcompat.h header
  dnl causes.)
  PKG_CHECK_MODULES(OPENCV, opencv >= 2.0.0 opencv <= 2.1.0 , [
    AC_PROG_CXX
    AC_LANG_CPLUSPLUS
    OLD_CPPFLAGS=$CPPFLAGS
    CPPFLAGS=$OPENCV_CFLAGS
    AC_CHECK_HEADER(highgui.h, HAVE_HIGHGUI="yes", HAVE_HIGHGUI="no")
    AC_CHECK_HEADER(cvaux.h, HAVE_CVAUX="yes", HAVE_CVAUX="no")
    CPPFLAGS=$OLD_CPPFLAGS
    AC_LANG_C
    if test "x$HAVE_HIGHGUI" = "xno"; then
      AC_MSG_RESULT(highgui.h could not be found.)
      HAVE_OPENCV="no"
    elif test "x$HAVE_CVAUX" = "xno"; then
      AC_MSG_RESULT(cvaux.h could not be found.)
      HAVE_OPENCV="no"
    else
      HAVE_OPENCV="yes" 
      AC_SUBST(OPENCV_CFLAGS)
      AC_SUBST(OPENCV_LIBS)  
    fi
  ], [
    HAVE_OPENCV="no"
    AC_MSG_RESULT(no)
  ])

And I don't even know if this last one is really ok because I placed the AC_SUBST macros inside an if and I don't know if that's valid. All this was made copying from already working projects with similar stuff and changing things until it worked, 9879864 "./autogen.sh &&./configure && make && make install" attempts later.



A programmer in fierce battle against GNU Autotools

How is it possible...? That it's so difficult? I cannot believe it. I felt that I needed to change a lightbulb; the documentation explained the history of electricity, the reference was as big as a manual to build your own nuclear reactor, and the examples explained how to light a candle.
Just this minute I found this: http://smalltalk.gnu.org/blog/bonzinip/all-you-should-really-know-about-autoconf-and-automake
I had to google "GNU Autotools hate" to find it.
And there is a lot more I'd like to say about autotools! But I'll let that for a RANT&RAGE type post, if I ever get to write about it ;)

13 comments:

Jan Moren said...

About autotools: I used to dislike them - no, I used to loathe them. It was just freaking impossible to get a grip on what I was actually doing.

At one point, though, I decided I needed to learn them properly (I was involved in a project that uses them a lot). After a good deal of googling I found this:

http://www.freesoftwaremagazine.com/books/agaal/brief_introduction_to_gnu_autotools

It's longish (longer than it needs to be, I think), but that document took me from loathing autotools to becoming a fan. That's the first source I've seen that makes you understand what is actually happening, and why.

The author, by the way, apparently has a book out now based on this text. It's going into my shopping card the next time I order books.

Anonymous said...

I was going to mention the same book as the comment above me, actually. So I'll just mention that the URL he gave is to chapter 1, but the introduction/preface/whatever you'd call it is at http://www.freesoftwaremagazine.com/books/autotools_a_guide_to_autoconf_automake_libtool.

Dinesh said...

I *totally* understand the frustration in using autotools...
nice picture of the dragons btw.

for my GSoC, it was generating configure script which was getting stuck in an infinite loop!! and at that stage i really didnt know where to look for or what to ask for!!

I spent more time getting things to build than working on the actual code!!

nice to know that i m not alone! :P

Diorama Yuanito said...

i can't believe it... an amazing article...

knocte said...

Agreed on the autotools part. For the love of god, they have to be buried very deep and forgotten.

p-Lo said...

I know they say that autotools are something that people hate only if they don't understand them... well I hate autotools, and I don't understand them, but I think I do have good reasons - so many silly problems I've had to debug while cross compiling various open source packages for an embedded system on heterogenous workstations.... such incredibly silly problems, like "check if file exists not valid for cross compiling" and abort. Thanks! and the libtool automake autoconf version incompatibilities. and the one where libtool would somehow emit subtly flawed command lines until I did "export echo=echo" - no joke, I could go on and on...

Stefan Sauer said...

Laura, you might be able to get the URI of the SVG by sending a uri-query.

Jon Nordby said...

You might interested in the generic cairo overlay element I did for gstreamer: https://bugzilla.gnome.org/show_bug.cgi?id=595520
(currently pending review)

Interestingly enough I actually picked up that task and did the patch due to a blogpost of yours where you talked about the rsvg element, because that element showed how simple it could be done.

Olivier said...

Well, with this context information (isolation of gstreamer pipelines), I can know see a use case where having these parameters actually matter. I wanted to keep the element interface simple, and delegate complexity to SVG, but anyway, most people will want to have these parameters (x/y/width/height, with -absolute and -relative variants for each).
I will add them next week or so.

Unknown said...

I gave up on autotools. For some time I really tried to learn it, until I realized there were alternatives :-) I have evaluated autotools, CMake, Waf and SCons. Now I use CMake and I think it's great! I recommend you take a look at it.

jjardon said...

Some good resources about autotools:

- http://www.lrde.epita.fr/~adl/autotools.html
- http://www.flameeyes.eu/autotools-mythbuster/
- http://www.openismus.com/documents/linux/automake/automake.shtml

Hope helps

Olivier said...

See https://bugzilla.gnome.org/show_bug.cgi?id=642116

Unknown said...

@OlivierAuber: that's so great! =D *highfive* that will make things a lot easier, thanks a lot!

@Janne Morén @rmunn @jjardon: thanks for the autotool links, will sure be useful. no other way than to arm ouserlves with patience and understand what's really going on.

@Jon Nordby: interesting! but I'm having some problems trying to test it

@Dinesh @p-Lo @knocte @D10 @bjornfor: we share the autotools sentiment!

@ensonic: we chatted already in #gstreamer :)