Friday, 5 December 2014

Working in a separate prefix

I've been surprised in the past to discover that even some seasoned engineers didn't know how to use the autotools prefix feature. A sign they've been lucky enough and didn't have to deal with Autotools too much. Here's my attempt to provide some introduction to ./configure --prefix.

Working with or in "a separate prefix" is working with libraries and binaries (well, anything produced by 'make install' in an autotooled project really) installed in a different directory than the system-wide ones (/usr or even /usr/local that can become quite messy). It is the preferred way to hack on a full stack without polluting your base distribution and has several advantages:
  • One can hack on the whole stack without the fear of not being able to run your desktop environment you're working with if something goes wrong,
  • More often than not, one needs a relatively recent library that your distribution doesn't ship with (say a recent libdrm). When working with the dependencies in a prefix, it's just a matter of recompiling it.
Let's take an example to make the discussion easier:
  •  We want to compile libdrm and intel-gpu-tools (because intel-gpu-needs needs a more recent libdrm than the one coming with your distribution),
  •  We want to use the ~/gfx directory for our work,
  • git trees with be cloned in ~/gfx/sources,
  • ~/gfx/install is chosen as the prefix.
First, let's clone the needed git repositories:

$ mkdir -p ~/gfx/sources ~/gfx/install
$ cd ~/gfx/sources
$ git clone git://anongit.freedesktop.org/mesa/drm libdrm
$ git clone git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

Then you need to source a script that will set-up your environment with a few variables to tell the system to use the prefix (both at run-time and compile-time). A minimal version of that script for our example is (I store my per-project setup scripts to source at the root of the project, in our case ~/gfx):


$ cat ~/gfx/setup-env
PROJECT=~/gfx
export PATH=$PROJECT/install/bin:$PATH
export LD_LIBRARY_PATH=$PROJECT/install/lib:$LD_LIBRARY_PATH
export PKG_CONFIG_PATH=$PROJECT/install/lib/pkgconfig:$PKG_CONFIG_PATH
export ACLOCAL_FLAGS="-I $PROJECT/install/share/aclocal $ACLOCAL_FLAG"
$ source ~/gfx/setup-env

Then it's time to compile libdrm, telling the configure script that we want to install it in in our prefix:


$ cd ~/gfx/sources/libdrm
$ ./autogen.sh --prefix=/home/damien/gfx/install
$ make
$ make install

Note that you don't need to run "sudo make install" since we'll be installing in our prefix directory that is writeable by the current user.

Now it's time to compile i-g-t:


$ cd ~/gfx/sources/intel-gpu-tools
$ ./autogen.sh --prefix=/home/damien/gfx/install
$ make
$ make install

The configure script may complain about dependencies (eg. cairo, SWIG,...). Different ways to solve those:
  • For dependencies not directly linked with the graphics stack (like SWIG), it's recommended to use the development package provided by the distribution
  • For old enough dependencies that don't change very often (like cairo) you can use the distribution development package or compile them in your prefix
  • For dependencies more recent than your distribution ones, you need to install them in the chosen prefix.

Saturday, 20 September 2014

git commit --fixup and git rebase -i --autosquash

It's not unusual that I need to fix previous commits up when working  on a branch or in the review phase. Until now I used a regular commit with some special marker to remember which commit to squash it with and then git rebase -i to reorder the patches and squash the fixup commits with their corresponding "parent" commits.

Turns out, git can handle quite a few of those manual manipulations for you. git commit --fixup <commit> allows you to commit work, marking it as a fixup of a previous commit. git rebase -i --autosquash will then present the usual git rebase -i screen but with the fixup commits moved just after their parents and ready to be squashed without any extra manipulation.

For instance, I had a couple of changes to a commit buried 100 patches away from HEAD (yes, a big topic branch!):
$ git diff
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 29f3813..08ea851 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -2695,6 +2695,11 @@ static void skylake_update_primary_plane(struct drm_crtc *crtc,

intel_fb = to_intel_framebuffer(fb);
obj = intel_fb->obj;
+
+ /*
+ * The stride is expressed either as a multiple of 64 bytes chunks for
+ * linear buffers or in number of tiles for tiled buffers.
+ */
switch (obj->tiling_mode) {
case I915_TILING_NONE:
stride = fb->pitches[0] >> 6;
@@ -2707,7 +2712,6 @@ static void skylake_update_primary_plane(struct drm_crtc *crtc,
BUG();
}

- plane_ctl &= ~PLANE_CTL_TRICKLE_FEED_DISABLE;
plane_ctl |= PLANE_CTL_PLANE_GAMMA_DISABLE;

I915_WRITE(PLANE_CTL(pipe, 0), plane_ctl);

And I wanted to squash those changes with commit 2021785
$ git commit -a --fixup 2021785

git will then go ahead and create a new commit with the subject taken from the referenced commit and prefixed with <code>fixup!<code>
commit d2d278ffbe87d232369b028d0c9ee9e6ecd0ba20
Author: Damien Lespiau <damien.lespiau@intel.com>
Date: Sat Sep 20 11:09:15 2014 +0100

fixup! drm/i915/skl: Implement thew new update_plane() for primary planes

Then when using the interactive rebase with autosquash:
$ git rebase -i --autosquash drm-intel/drm-intel-nightly

The fixup will be next after the reference commit
pick 2021785 drm/i915/skl: Implement thew new update_plane() for primary planes
fixup d2d278ff fixup! drm/i915/skl: Implement thew new update_plane() for primary planes

validating the proposed change (by in my case leaving vim) will squash the fixup commits. Definitely what I'll be using from now on!

Oh, and there's a config option to have git rebase automatically autosquash if there are some fixup commits:
$ git config --global rebase.autosquash true

Wednesday, 2 October 2013

HDMI stereo 3D & KMS

If everything goes according to plan, KMS in linux 3.13 should have stereo 3D support. Should one be interested in scanning out a stereo frame buffer to a 3D capable HDMI sink, here's a rough description of how those modes are exposed to user space and how to use them.

A reader not well acquainted with the DRM sub-system and its mode setting API (Aka Kernel Mode Setting, KMS) could start by watching the first part of Laurent Pinchart's Anatomy of an Embedded KMS Driver or read David Herrmann's heavily documented mode setting example code.

Stereo modes work by sending a left eye and right eye picture per frame to the monitor. It's then up to the monitor to use those 2 pictures to display a 3D frame and the technology there varies.

There are different ways to organise the 2 pictures inside a bigger frame buffer. For HDMI, those layouts are described in the HDMI 1.4 specification. Provided you give them your contact details, it's possible to download the stereo 3D part of the HDMI 1.4 spec from hdmi.org.

As one inevitably knows, modes supported by a monitor can be retrieved out of the KMS connector object in the form of drmModeModeInfo structures (when using libdrm, it's also possible to write your own wrappers around the KMS ioctls, should you want to):
typedef struct _drmModeModeInfo {
uint32_t clock;
uint16_t hdisplay, hsync_start, hsync_end, htotal, hskew;
uint16_t vdisplay, vsync_start, vsync_end, vtotal, vscan;

uint32_t vrefresh;

uint32_t flags;
uint32_t type;
char name[...];
} drmModeModeInfo, *drmModeModeInfoPtr;

To keep existing software blissfully unaware of those modes, a DRM client interested in having stereo modes listed starts by telling the kernel to expose them:
drmSetClientCap(drm_fd, DRM_CLIENT_CAP_STEREO_3D, 1);

Stereo modes use the flags field to advertise which layout the mode requires:
uint32_t layout = mode->flags & DRM_MODE_FLAG_3D_MASK;

This will give you a non zero value when the mode is a stereo mode, value among:
DRM_MODE_FLAG_3D_FRAME_PACKING
DRM_MODE_FLAG_3D_FIELD_ALTERNATIVE
DRM_MODE_FLAG_3D_LINE_ALTERNATIVE
DRM_MODE_FLAG_3D_SIDE_BY_SIDE_FULL
DRM_MODE_FLAG_3D_L_DEPTH
DRM_MODE_FLAG_3D_L_DEPTH_GFX_GFX_DEPTH
DRM_MODE_FLAG_3D_TOP_AND_BOTTOM
DRM_MODE_FLAG_3D_SIDE_BY_SIDE_HALF

User space is then responsible for choosing which stereo mode to use and to prepare a buffer that matches the size and left/right placement requirements of that layout. For instance, when choosing Side by Side (half), the frame buffer is the same size as its 2D equivalent (that is hdisplay x vdisplay) with the left and right images sub-sampled by 2 horizontally:

[caption id="attachment_398" align="aligncenter" width="300"]sbsh Side by Side (half)[/caption]

Other modes need a bigger buffer than hdisplay x vdisplay. This is the case with frame packing, where each eye has the the full 2D resolution, separated by the number of vblank lines:

[caption id="attachment_399" align="aligncenter" width="300"]Frame Packing Frame Packing[/caption]

Of course, anything can be used to draw into the stereo frame buffer, including OpenGL. Further work should enable Mesa to directly render into such buffers, say with the EGL/gbm winsys for a wayland compositor to use. Of course, fun profit would the last step:

[caption id="attachment_374" align="aligncenter" width="261"]PS3_3D2 A 720p frame packing buffer from the game WipeOut[/caption]

Behind the scene, the kernel's job is to parse the EDID to discover which stereo modes the HDMI sink supports and, once user-space instructs to use a stereo mode, to send infoframes (metadata sent during the vblank interval) with the information about which 3D mode is being sent.

A good place to start for anyone wanting to use this API is testdisplay, part of the Intel GPU tools test suite. testdisplay can list the available modes with:
$ sudo ./tests/testdisplay -3 -i
[...]
name refresh (Hz) hdisp hss hse htot vdisp vss vse vtot flags type clock
[0] 1920x1080 60 1920 2008 2052 2200 1080 1084 1089 1125 0x5 0x48 148500
[1] 1920x1080 60 1920 2008 2052 2200 1080 1084 1089 1125 0x5 0x40 148352
[2] 1920x1080i 60 1920 2008 2052 2200 1080 1084 1094 1125 0x15 0x40 74250
[3] 1920x1080i 60 1920 2008 2052 2200 1080 1084 1094 1125 0x20015 0x40 74250 (3D:SBSH)
[4] 1920x1080i 60 1920 2008 2052 2200 1080 1084 1094 1125 0x15 0x40 74176
[5] 1920x1080i 60 1920 2008 2052 2200 1080 1084 1094 1125 0x20015 0x40 74176 (3D:SBSH)
[6] 1920x1080 50 1920 2448 2492 2640 1080 1084 1089 1125 0x5 0x40 148500
[7] 1920x1080i 50 1920 2448 2492 2640 1080 1084 1094 1125 0x15 0x40 74250
[8] 1920x1080i 50 1920 2448 2492 2640 1080 1084 1094 1125 0x20015 0x40 74250 (3D:SBSH)
[9] 1920x1080 24 1920 2558 2602 2750 1080 1084 1089 1125 0x5 0x40 74250
[10] 1920x1080 24 1920 2558 2602 2750 1080 1084 1089 1125 0x1c005 0x40 74250 (3D:TB)
[11] 1920x1080 24 1920 2558 2602 2750 1080 1084 1089 1125 0x4005 0x40 74250 (3D:FP)
[...]

To test a specific mode:
$ sudo ./tests/testdisplay -3 -o 17,10
1920x1080 24 1920 2558 2602 2750 1080 1084 1089 1125 0x1c005 0x40 74250 (3D:TB)

To cycle through all the supported stereo modes:
$ sudo ./tests/testdisplay -3

testdisplay uses cairo to compose the final frame buffer from two separate left and right test images.

Monday, 3 June 2013

Working on more than one line with sed's 'N' command

Yesterday I was asked to help solving a small sed problem. Considering that file (don't look too closely on the engineering of the defined elements):

<root>
<key>key0</key>
<string>value0</string>
<key>key1</key>
<string>value1</string>
<key>key2</key>
<string>value2</string>
</root>


The problem was: How to change value1 to VALUE!. The problem here is that you can't blindly execute a s command matching <string>.*</string>.

Sed maintains a buffer called the "pattern space" and processes commands on this buffer. From the GNU sed manual:

sed operates by performing the following cycle on each line of input: first, sed reads one line from the input stream, removes any trailing newline, and places it in the pattern space. Then commands are executed; each command can have an address associated to it: addresses are a kind of condition code, and a command is only executed if the condition is verified before the command is to be executed.

When the end of the script [(list of sed commands)] is reached, unless the -n option is in use, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed.3 Then the next cycle starts for the next input line.


So the idea is to first, use a /pattern/ address to select the the right <key> line, append the next line to the pattern space (with the N command) and finally run a s command on the buffer now containing both lines:

  <key>key1</key>
<string>value1</string>


And so we end up with:
$ cat input 
<root>
<key>key0</key>
<string>value0</string>
<key>key1</key>
<string>value1</string>
<key>key2</key>
<string>value2</string>
</root>
$ sed -e '/<key>key1<\/key>/{N;s#<string>.*<\/string>#<string>VALUE!<\/string#;}' < input
<root>
<key>key0</key>
<string>value0</string>
<key>key1</key>
<string>VALUE!</string
<key>key2</key>
<string>value2</string>
</root>

Sunday, 13 January 2013

A git pre-commit hook to check the year of copyright notices

Like every year, touching a source file means you also need to update the year of the copyright notice you should have at the top of the file. I always end up forgetting about them, this is where a git pre-commit hook would be ultra-useful, so I wrote one:


#
# Check if copyright statements include the current year
#
files=`git diff --cached --name-only`
year=`date +"%Y"`

for f in $files; do
head -10 $f | grep -i copyright 2>&1 1>/dev/null || continue

if ! grep -i -e "copyright.*$year" $f 2>&1 1>/dev/null; then
missing_copyright_files="$missing_copyright_files $f"
fi
done

if [ -n "$missing_copyright_files" ]; then
echo "$year is missing in the copyright notice of the following files:"
for f in $missing_copyright_files; do
echo " $f"
done
exit 1
fi


Hope this helps!

Tuesday, 10 April 2012

Extracting part of files with sed

For reference for my future self, a few handy sed commands. Let's consider this file:
$ cat test-sed
First line
Second line
--
Another line
Last line

We can extract the lines from the start of the file to the marker by deleting the rest:
$ sed '/--/,$d' test-sed 
First line
Second line

a,b is the range the command, here d(elete), applies to. a and b can be, among others, line numbers, regular expressions or $ for end of the file. We can also extract the lines from the marker to the end of the file with:
$ sed -n '/--/,$p' test-sed 
--
Another line
Last line

This one is slightly more complicated. By default sed spits all the lines it receives as input, '-n' is there to tell sed not to do that. The rest of the expression is to p(rint) the lines between -- and the end of the file.

That's all folks!

Friday, 16 September 2011

A simple autotool template

Every now and then, you feel a big urge to start hacking on a small thingy and need to create Makefiles for it. Turns out that the autotools won't be that intrusive when we are talking about small programs and you get do a reasonable job with a few lines, first the configure.ac file:
# autoconf
AC_PREREQ(2.59)
AC_INIT([fart], [0.0.1], [damien.lespiau@gmail.com])
AC_CONFIG_MACRO_DIR([build])
AC_CONFIG_AUX_DIR([build])
AC_CONFIG_SRCDIR([fart.c])
AC_CONFIG_HEADERS([config.h])

# automake
AM_INIT_AUTOMAKE([1.11 -Wall foreign no-define])
AM_SILENT_RULES([yes])

# Check for programs
AC_PROG_CC

# Check for header files
AC_HEADER_STDC

AS_COMPILER_FLAGS([WARNING_CFLAGS],
["-Wall -Wshadow -Wcast-align -Wno-uninitialized
-Wno-strict-aliasing -Wempty-body -Wformat -Wformat-security
-Winit-self -Wdeclaration-after-statement -Wvla
-Wpointer-arith"])

PKG_CHECK_MODULES([GLIB], [glib-2.0 >= 2.24])

AC_OUTPUT([
Makefile
])

and then Makefile.am:
ACLOCAL_AMFLAGS = -I build ${ACLOCAL_FLAGS}

bin_PROGRAMS = fart

fart_SOURCES = fart.c
fart_CFLAGS = $(WARNING_CFLAGS) $(GLIB_CFLAGS)
fart_LDADD = $(GLIB_LIBS)

After that, it's just a matter of running autoreconf
$ autoreconf -i

and you are all set!

So, what do you get for this amount of lines?

  • The usual set of automake targets, handy! ("make tags" is so under used!) and bonus features (out of tree builds, extra rules to reconfigure/rebuild the Makefiles on changes in configure.ac/Makefile.an, ...)

  • Trying to make the autoconf/automake discreet (putting auxiliary files out of the way, silence mode, automake for non GNU projects)

  • Some decent warning flags (tweak to your liking!)

  • autoreconf cooperating with aclocal thanks to ACLOCAL_AMFLAGS and coping with non standard locations for system m4 macros


I'll maintain a git tree to help bootstrap my next small hacks, feel free to use it as well!