Last Saturday, I got up early and decided to whip up something quick yet useful. Roughly two hours later, I had released the first version of “Kanji”.

“Kanji” is a simple tool for reviewing Japanese characters or words. It reads in a UTF-8 encoded file and displays the entries within at random, showing a new entry every 30 seconds. It is intended to be run in “always on top” mode, in which case it can be used for passive studying of Japanese while working on other things.

This tool is minimalistic by design. There are no config files nor preferences. Everything is specified via the command line, after which, the program simply runs until closed.

As an example, the above screenshot was actually created via the following invocation (on Windows 7):

The code is available on Github for those who wish to try it: https://github.com/Vultaire/flashcards

Update: A new “card flipping” feature has been added, and cards are now defined in CSV files with one line per card and one “side” per column. The program has also been renamed to “flashcards” to reflect its more general intended usage.

Here’s the situation: I want to convert my FLAC collection to an MP3 collection so that I can upload my CD collection to Amazon. Further, I’d like to store the MP3s in a different folder, while maintaining the original subfolder structure. Finally, I want to maintain the metadata in my FLAC files; the flac command line utility only seems to support decoding to .wav without metadata.

Finally, I have 8 virtual cores, and I want to use them all to convert everything as quickly as possible.

Creating a mirrored directory structure, using xargs

First, let’s create the directories to store our MP3 files in:

find Music_CDs_FLAC/ -iname '*.flac' \
    | sed -re 's#^Music_CDs_FLAC/(.*)/[^/]*$#Music_CDs_MP3s/\1#' \
    | sort \
    | uniq \
    | xargs -r -P1 -n1 -d'\n' mkdir -pv

This command pulls all unique folder names under Music_CDs_FLAC and creates directories. In this case I used -P1 which only allocates 1 process. Of course, at this rate xargs is not necessary; the above could be accomplished with a simple bash for loop instead.

Creating a FLAC to MP3 script, using xargs

Next, let’s convert a single FLAC file to an MP3. This is a complex process since we can’t easily convert straight from FLAC to MP3. (Well, I’m sure there’s programs out there, but I don’t know what’s good and this serves as a good example.)

Oh, but wait, there’s more: lame expects tags on the command line, and what do we do about tags containing quotes or apostrophes? Do we escape them? I’d rather not bother here; let’s allow xargs to do the work for us.

Let’s make a script which takes an input and output file name, and which performs the needed conversion, using xargs for passing tags:

#!/bin/bash

# Author: Paul Goins
# License: Public Domain
# File: flac2mp3.sh
#
# Script to convert FLAC files to MP3, using xargs for passing tag
# arguments to lame.
#
# Why do we do this?  It's for the metatags which may contain
# characters requiring shell quotes.  This gets around having to do
# any escaping.
#
# Note: This script "works for me"; it may not keep all metadata from
# your FLAC files.  It does keep most of the data from my FLAC files
# which were generated via abcde.

set -o nounset
set -o errexit

infile="$1"
outfile="$2"
argfile="$(mktemp)"

# Extract metadata and push tags to argfile
metafile="$(mktemp)"
metaflac --export-tags-to "$metafile" "$infile"

arg_regex='s/^[^=]*=(.*)$/\1/'
ARTIST=$(grep "$metafile" -e 'ARTIST=' | sed -re "$arg_regex")
if [ ! -z "$ARTIST" ]; then
    echo -e "--ta\n$ARTIST" >> "$argfile"
fi
ALBUM=$(grep "$metafile" -e 'ALBUM=' | sed -re "$arg_regex")
if [ ! -z "$ALBUM" ]; then
    echo -e "--tl\n$ALBUM" >> "$argfile"
fi
TITLE=$(grep "$metafile" -e 'TITLE=' | sed -re "$arg_regex")
if [ ! -z "$TITLE" ]; then
    echo -e "--tt\n$TITLE" >> "$argfile"
fi
DATE=$(grep "$metafile" -e 'DATE=' | sed -re "$arg_regex")
if [ ! -z "$DATE" ]; then
    echo -e "--ty\n$DATE" >> "$argfile"
fi
GENRE=$(grep "$metafile" -e 'GENRE=' | sed -re "$arg_regex")
if [ ! -z "$GENRE" ]; then
    echo -e "--tg\n$GENRE" >> "$argfile"
fi
TRACKNUMBER=$(grep "$metafile" -e 'TRACKNUMBER=' | sed -re "$arg_regex")
if [ ! -z "$TRACKNUMBER" ]; then
    echo -e "--tn\n$TRACKNUMBER" >> "$argfile"
fi

rm "$metafile"

# Push infile/outfile as final args
echo "-" >> "$argfile"
echo "$outfile" >> "$argfile"

# Perform the conversion
flac --decode --stdout --silent "$infile" \
    | xargs --arg-file="$argfile" -r -d'\n' lame --quiet -b320

rm "$argfile"

Okay, this bit of code does have some real benefit from using xargs. However, it’s not what really makes me excited.

A sidenote: xargs and multiple commands

xargs runs a single command. For complex behavior involving pipes, it is possible to run something like the following (contrived) example:

# Contrived example!  Yay!!
cat /dev/null | xargs -n1 bash -c "echo foo | grep -e \"oo\""

For complex behavior like the above file conversion script, the benefits of writing a dedicated script are rather obvious. However, even when piping a couple of commands together, it may be worth writing a quick and dirty script and calling it via xargs, rather than messing with shell quoting. This is especially true if you’re dealing with lots of data which could contain characters requiring special quoting.

Using xargs for parallelization: fully utilizing 8 virtual cores

This brings us to the final part: to automatically convert all my FLAC files to MP3, and to do so in parallel. I run a Core i7, which means I have 4 physical cores with hyperthreading, coming out to 8 virtual cores. I’m going to run 12 parallel processes to perform the conversion, ensuring my processor stays as busy as possible.

Here’s the script:

find Music_CDs_FLAC -iname '*.flac' \
    | sed -re 's#^Music_CDs_FLAC/(.*)\.flac$#Music_CDs_FLAC/\1.flac\nMusic_CDs_MP3s/\1.mp3#' \
    | xargs -t -r -P12 -n2 -d'\n' ~/scripts/flac2mp3.sh

There we go. The above will print out each command as it’s executed, and it will quickly take up nearly 100% of all my CPU cores. It may hit a bottleneck when writes need to be flushed to disk, but this should (hopefully) take less time overall than doing things one-by-one.

A quick description of what I did:

• I found all .flac files and printed out one per line. (find command)
• I took each .flac file and output two lines: the original .flac and the target .mp3. (sed command)
• I launched concurrent conversion scripts, up to 12 at a time, taking two arguments per script call (input/output files), separating input args on newlines. The commands were echoed to the console as they were run. (xargs command)

Final note: after actually running the above snippet, converting 36 GiB of FLACs (1283 files) clocked in as follows:

real    41m56.118s
user    321m43.014s
sys     3m18.788s

Note: the real time was only 42 minutes, yet the user time was 5 hours and 22 minutes! Without xargs, using just a for loop to work through all the files, this would have taken much longer. This workload obviously benefitted from parallelization via xargs.

For the record, on my system I am using kernel version 2.6.39-bpo.2-amd64.

I forget the exact forum page I found, but it seems using DKMS is now the preferred way of installing the official drivers in Debian. http://wiki.debian.org/NvidiaGraphicsDrivers seems to confirm this, although it is not immediately obvious.

Anyway, basically the procedure should be as simple as:

apt-get install -t squeeze-backports dkms nvidia-kernel-dkms

However, if you look carefully, an error likely occurred. If you missed the error, you should be able to reproduce it via:

dpkg-reconfigure nvidia-kernel-dkms

The output should be something like:

------------------------------
Deleting module version: 195.36.31
completely from the DKMS tree.
------------------------------
Done.
Loading new nvidia-195.36.31 DKMS files...
Building only for 2.6.39-bpo.2-amd64
Building initial module for 2.6.39-bpo.2-amd64

Error! Bad return status for module build on kernel: 2.6.39-bpo.2-amd64 (x86_64)
Consult the make.log in the build directory
/var/lib/dkms/nvidia/195.36.31/build/ for more information.

Looking at /var/lib/dkms/nvidia/195.36.31/build/make.log, I found the following errors:

$ grep -ie error /var/lib/dkms/nvidia/195.36.31/build/make.log
/var/lib/dkms/nvidia/195.36.31/build/nv-linux.h:22:28: error: linux/autoconf.h: No such file or directory
/var/lib/dkms/nvidia/195.36.31/build/nv-linux.h:92:75: error: linux/smp_lock.h: No such file or directory
make[4]: *** [/var/lib/dkms/nvidia/195.36.31/build/nv.o] Error 1
make[3]: *** [_module_/var/lib/dkms/nvidia/195.36.31/build] Error 2
make[2]: *** [sub-make] Error 2
make[1]: *** [all] Error 2
make: *** [modules] Error 2

Okay, we’re missing some include files. It seems these are files which are not included in recent kernel versions.

Reference for smp_lock.h: http://idratherhack.blogspot.com/2011/04/nvidia-drivers-fail-without-smplockh.html

References for autoconf.h:

1. https://bugs.launchpad.net/ubuntu/+source/dahdi-linux/+bug/591375
2. http://alxrem.blogspot.com/2011/05/nvidia-drivers-on-debian-squeeze-with.html

Basically, smp_lock.h is for the now obsolete Big Kernel Lock, and autoconf.h got moved.

So, how do we kludge together a fix? The following is what I could come up with. Copy/paste this into a bash shell:

(
    set -o errexit

    die () {
        echo "$@" > /dev/stderr
        exit 1
    }

    if (($(id -u) != 0)); then
        die "This snippet must be run as root."
    fi

    # Install prerequisites
    apt-get install -t squeeze-backports dkms nvidia-kernel-dkms

    # Create fake smp_lock.h
    cd /lib/modules/$(uname -r)/source/include/linux
    if [ ! -e "smp_lock.h" ]; then
        touch smp_lock.h
        echo "Created dummy smp_lock.h."
    fi

    # Link to relocated autoconf.h
    cd /lib/modules/$(uname -r)/build/include/linux
    ln -sv ../generated/autoconf.h

    # Patch header file in nvidia sources
    # (Patch based on: http://alxrem.blogspot.com/2011/05/nvidia-drivers-on-debian-squeeze-with.html)
    cd /usr/src/nvidia-195.36.31
    cp -v nv-linux.h{,.bak}
    patchfile=$(tempfile)
    cat <<"EOF" > $patchfile
--- a/nv-linux.h     2011-05-11 12:26:22.206190191 +0400
+++ b/nv-linux.h  2011-05-04 15:36:27.163263047 +0400
@@ -975,8 +975,13 @@
 #endif

 #if defined(NV_ACQUIRE_CONSOLE_SEM_PRESENT)
-#define NV_ACQUIRE_CONSOLE_SEM() acquire_console_sem()
-#define NV_RELEASE_CONSOLE_SEM() release_console_sem()
+# if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 38)
+#  define NV_ACQUIRE_CONSOLE_SEM() console_lock()
+#  define NV_RELEASE_CONSOLE_SEM() console_unlock()
+# else
+#  define NV_ACQUIRE_CONSOLE_SEM() acquire_console_sem()
+#  define NV_RELEASE_CONSOLE_SEM() release_console_sem()
+# endif
 #else
 #define NV_ACQUIRE_CONSOLE_SEM()
 #define NV_RELEASE_CONSOLE_SEM()
EOF
    patch -Np1 -i $patchfile
    rm $patchfile

    # Reconfigure kernel sources (which will automatically build the nvidia DKMS module)
    dpkg-reconfigure linux-headers-$(uname -r)

    # Explicitly reconfigure the nvidia bits (sometimes this seems to be done automatically, sometimes not...)
    dpkg-reconfigure nvidia-kernel-dkms
)

Okay, that was a little long, but the above should successfully install the kernel module. (No warranty, but I developed and tested this on my two amd64 Debian Squeeze machines.)

For full disclosure, side effects of the above snippet are as follows:

1. New empty header: /usr/src/linux-headers-$(uname -r)/include/linux/smp_lock.h
2. New symlink: /usr/src/linux-headers-$(uname -r)/include/linux/autoconf.h (-> ../generated/autoconf.h)
3. Modified header: /usr/src/nvidia-195.36.31/nv-linux.h (original is backed up as /usr/src/nvidia-195.36.31/nv-linux.h.bak)

From here, what typically remains is:

apt-get install -y nvidia-glx  # or one of the nvidia-glx-legacy-* modules
apt-get install -y nvidia-glx-ia32  # optional
apt-get install -y nvidia-xconfig nvidia-settings
if [ -e "/etc/X11/xorg.conf" ]; then
    echo "/etc/X11/xorg.conf exists; please change the display driver to "nvidia".
else
    nvidia-xconfig
fi

At this point, reboot, and you should be up and running!

Note that none of what I’ve done here is really original; I’ve just assembled notes from others and put them together in a few bash snippets. Just hope it’ll help someone get up and running faster than it took me. =)

Sidenote #1: On the Debian wiki page, if you don’t know about DKMS and try to read this page, you’ll likely end up assuming that module-assistant is the way to go, but I found that this would not work in my case, probably due to the backported kernel. (I don’t exactly remember the error.)

Sidenote #2: Installing nvidia-kernel-dkms adds a blacklist rule for the nouveau module, but Xorg’s defaults will cause nouveau to be loaded anyway unless an appropriate Xorg.conf file exists.

Sidenote #3: Found someone who had nearly an identical problem: http://alxrem.blogspot.com/2011/05/nvidia-drivers-on-debian-squeeze-with.html The notes are almost a complete solution, minus the smp_lock.h fix. I’m guessing it was written before the BKL was completely removed.

UPDATE #1: Unfortunately WordPress decided to butcher my source code and replace HTML special characters with character entities. I’ve fixed the code; hopefully the character replacements were the only changes it “helped” me with. Please let me know if this works.

UPDATE #2: Thank you everyone for your feedback. I was pretty late getting back to this but have finally updated the procedure based on your comments. Thanks!

UPDATE #3: Just a quick note, as I haven’t had time to fully apply and test this myself. I’ve noticed an update is necessary for this to work with the new backported 3.2 kernel. At a minimum, the following patch looks to be needed. (Note: the sourcecode WordPress plugin is not preserving tabs in the patch, so please apply manually.)

--- /usr/src/nvidia-195.36.31/Makefile.bak	2012-02-14 20:31:03.000000000 -0800
+++ /usr/src/nvidia-195.36.31/Makefile	2012-02-14 22:57:22.000000000 -0800
@@ -155,8 +155,9 @@

 # Check for kernel versions that we don't support.

-BELOW26 := $(shell if [ $(VERSION) -lt 2 -o $(PATCHLEVEL) -lt 6 ]; then \
-		echo y; fi)
+#BELOW26 := $(shell if [ $(VERSION) -lt 2 -o $(PATCHLEVEL) -lt 6 ]; then \
+#		echo y; fi)
+BELOW26 :=

 ifeq ($(BELOW26),y)
 $(error Only 2.6.x and later kernels are supported \

Yes, I know this doesn’t “fix” the if condition; however it should allow the compilation to proceed. I’ll post more once I have details.

I recently re-installed Debian Squeeze on my main machine. I’m using the 64-bit version along with 64-bit Chrome.

However, the 64-bit Chrome does not come with a built-in Flash player. Instead, it seems to be finding the plugin by searching the Mozilla plugin folder (/usr/lib/mozilla/plugins/), where it finds flash-mozilla.so, which is a symlink to /usr/lib/gnash/libgnashplugin.so, the Gnash flash plugin.

Don’t get me wrong; I’m happy to see how Gnash is looking, and I’m happy that a free software alternative exists. However, I’m not a FLOSS puritan; I’ll use the best tool for the job which, in my opinion, remains as the official Flash plugin.

So, how to install the Flash plugin?

Adobe says to refer to the browser documentation. Google’s docs mention Flash as being built-in for 32-bit, but I couldn’t find mention of how to install for 64-bit.

Here’s how I did it: since Chrome seems to be using the symlink in the mozilla folder, which is managed by Debian’s “alternatives” system, I installed the plugin and added it as an alternative.

Step 1: Download the plugin from http://get.adobe.com/flashplayer/. (This procedure explains the .tar.gz installation method; Adobe now offers an Apt-oriented solution as well but I have not tested this.)

Step 2: Install the plugin as follows:

sudo mkdir -pv /opt/flashplugin
cd /opt/flashplugin
sudo tar -xf <path/to/flashplugin.tar.gz>

Step 3: Install the plugin as a flash-mozilla.so alternative:

sudo update-alternatives --install \
    /usr/lib/mozilla/plugins/flash-mozilla.so \
    flash-mozilla.so \
    /opt/flashplugin/libflashplayer.so \
    10

Step 4: Select the Adobe plugin for use as the active plugin:

sudo update-alternatives --set flash-mozilla.so /opt/flashplugin/libflashplayer.so

That’s it! Flash should now work in both Chrome and Firefox. If it does not work immediately, restart your browser.

A brief aside about my background for newcomers: I’m experienced with Python and C++ (although admittedly rusty in the latter), and I’ve written straight C as well (in college, for work, for GNU Typist). However, I feel my C skills are not as strong as they could be.

Basically, I know how to program and am not interested in reinventing the wheel. My focus is on developing my skills with practical C development, using any open source available libraries which I can along with, if feasible/possible, the MinGW/MSys toolchain.

This first article is a tutorial of how to use GLib with the autotools on Windows via MinGW/MSys.

MinGW/MSys were installed via the automatic installer. Relevant versions of installed programs include gcc 4.5.0, automake 1.11.1, and autoconf 2.67.

The GTK+ bundle in use is gtk+-bundle_2.22.0-20101016_win32.zip. I’m not actually using GTK+ in this project; however, the bundle includes GLib, pkg-config and the needed autoconf macros.

Unzip the GTK+ bundle somewhere on your system. Add the bin folder to your PATH, likely at the beginning to avoid conflicts with other tools.

The m4 macros in the share/aclocal folder of the GTK+ bundle should be copied to the folder identified by aclocal --print-ac-dir. On my system, this is /mingw/share/aclocal via the MSys shell.

Before going further, you should test pkg-config and make sure it works as expected:

pkg-config --cflags --libs glib-2.0

# Output on my machine:
#   -mms-bitfields -Ic:/gtk_bundle/include/glib-2.0 -Ic:/gtk_bundle/lib/glib-2.0/include  -Lc:/gtk_bundle/lib -lglib-2.0 -lintl

If this works, then create the following source file:

glib_test.c:

#include <glib.h>
#include <stdio.h>

int main(int argc, char **argv) {
	printf("GLib version: %d.%d.%d\n",
		GLIB_MAJOR_VERSION,
		GLIB_MINOR_VERSION,
		GLIB_MICRO_VERSION);
	return 0;
}

Run autoscan to create configure.scan. It should look like this initially:

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

AC_PREREQ([2.67])
AC_INIT([FULL-PACKAGE-NAME], [VERSION], [BUG-REPORT-ADDRESS])
AC_CONFIG_SRCDIR([glib_test.c])
AC_CONFIG_HEADERS([config.h])

# Checks for programs.
AC_PROG_CC

# Checks for libraries.

# Checks for header files.

# Checks for typedefs, structures, and compiler characteristics.

# Checks for library functions.

AC_OUTPUT

We need to make a few changes to configure.scan to make a working configure.ac.

Autoconf will actually work without automake. So, if we want to use automake as well, we need to add a macro for it:

AM_INIT_AUTOMAKE([foreign])

Also, we’re using GLib, which requires special linker and compiler flags provided by the pkg-config tool. When using autoconf, we link these in via some special macros.

The following will check for pkg-config, get the needed flags from pkg-config for GLib, and assign them to two variables usable in automake:

PKG_PROG_PKG_CONFIG([0.23])  # Replace with target pkg-config version (I just used "pkg-config --version")
PKG_CHECK_MODULES([GLIB], [glib-2.0 >= 2.26.0])  # Replace with target GTK version number
AC_SUBST([GLIB_CFLAGS])
AC_SUBST([GLIB_LIBS])

Finally, we need to specify the makefiles we want to generate via ./configure. For this example, we just need one: Makefile.

AC_CONFIG_FILES([Makefile])

While we’re working on this, go ahead and fill in some values for FULL-PACKAGE-NAME, VERSION and BUG-REPORT-ADDRESS.

Finally, if we merge all the above changes together and save as configure.ac, we should have something like the following:

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

AC_PREREQ([2.67])
AC_INIT([glibtest], [0.1], [fakeaddress@foo.net])
AM_INIT_AUTOMAKE([foreign])
AC_CONFIG_SRCDIR([glib_test.c])
AC_CONFIG_HEADERS([config.h])

# Checks for programs.
AC_PROG_CC
PKG_PROG_PKG_CONFIG([0.23])

# Checks for libraries.
PKG_CHECK_MODULES([GLIB], [glib-2.0 >= 2.26.0])
AC_SUBST([GLIB_CFLAGS])
AC_SUBST([GLIB_LIBS])

# Checks for header files.

# Checks for typedefs, structures, and compiler characteristics.

# Checks for library functions.

AC_CONFIG_FILES([Makefile])
AC_OUTPUT

Now that configure.ac is out of the way, we need a Makefile.am. This will use the GLIB_CFLAGS and GLIB_LIBS variables made in the autoconf.ac.

Makefile.am:

bin_PROGRAMS = glib_test

glib_test_SOURCES = glib_test.c
glib_test_CFLAGS = $(GLIB_CFLAGS)
glib_test_LDADD = $(GLIB_LIBS)

The next step is to run autoreconf -i, which run all the necessary autotools and creates a configure file.

Run ./configure followed by make. This should compile the program.

Test the program: glib_test.exe. The MSys shell output should be something like the following:

Vultaire@core /c/code/projects/glibtest
$ glib_test.exe
GLib version: 2.26.0

If you got this far, excellent. You’re ready to start hacking with GLib.

An aside: why GLib? Because it provides a number of the data structures which would be covered by the STL in C++, or as part of the standard library in many higher level languages. It seems like a good foundation library to use for general development purposes, even if GTK+ is never used in the project.

Update, 2011-Jan-19:

Minor fix: Replaced glib_test_LIBS with glib_test_LDADD. The former was a mistake; the latter is the proper variable for specifying extra libraries which need to be linked.

Also, some documentation references:

http://www.flameeyes.eu/autotools-mythbuster/pkgconfig/pkg_check_modules.html: My original source for the pkg-config macros.
http://people.freedesktop.org/~dbn/pkg-config-guide.html#faq: Documents simple usage of the pkg-config macros with automake.
http://sources.redhat.com/automake/automake.html: Automake manual (confirms that glib_test_LDADD is where we should put pkg-config input.)