jd:/dev/blog - Tag - rebuildd

rebuildd demo

jd — Tue, 04 Dec 2007 22:36:00 +0100

I've set up a quick rebuildd demo. It may allow you to rebuild one of your package on etch/lenny/sid and for amd64/i386 as you wish. You will get build log by email.

I've just wrote the PHP page in 2 hours. I just hate Web interfaces and PHP is crap, seriously. For the record, I've tried to use DataGrid, and after having lost 1h to get it work with no PHP/PEAR error printed or available, I managed to have it working with a big pear upgrade-all. OMFG.

rebuildd 0.3.0 released

jd — Mon, 03 Dec 2007 15:06:00 +0100

Everything seems to work, so just releasing it. I think it's becoming an amazing tool to do archive rebuild.

It just misses a good Web interface.

Testing out rebuildd 0.3.0

jd — Sun, 02 Dec 2007 10:04:00 +0100

awesome beiing in rc state, that leaves me a small amount of spare time. I decided to spend it on rebuildd. I had started the 0.3.0 version back in August, but never touched it again until now.

I was afraid because I did not remember in which state I left the code repository. And, good surprise, the code was in a quite good shape!

I just implemented and fixed some stuff that were reported on the BTS, and then, I only had to test it out.

For that I used the servers kindly provided by TuxFamily.org:

whisky: 6 x Pentium III @ 700 MHz/L2 1 MB + 1.7 GB RAM
octave: 8 x Pentium III @ 500 MHz/L2 2 MO + 2 GB RAM

I stored the MySQL database on whisky. filled it with rebuildd-job add-quinn-diff (new feature in 0.3.0) and then started rebuildd on each node. And then I saw the 2 hosts starting to grab, lock jobs, build packages on each side. Wonderful. That just works.

rebuildd-job stats says that 20 % of the archive is now built in less than 24h, seems fast, cool.

Bad point is that I'm building and trashing logs into /dev/null because I've no QA idea to implement right now.

Ubuntu: not contributing to Debian, and even doing bad work

jd — Tue, 02 Oct 2007 20:34:00 +0200

Googling around using the rebuildd keyword to find if people were talking about it, I found that patch from Ubuntu.

This patch is used to ship rebuildd Debian package in Ubuntu.

Wahou. First surprise! So, Ubuntu is fixing bugs and not telling me. I though that people saying "Ubuntu does not contribute back to Debian!" were exaggerating. Seems I was wrong.

Nevermind: I take the patch. I read the patch. I apply the pat… oh wait: the fix is wrong.

lol.

Debian archive rebuild for nostrip finished

jd — Thu, 16 Aug 2007 17:02:00 +0200

So after rebuilding the whole archive with octave and rebuildd during 8 days, here are the results.

Some stats:

813 bugs
95 bugs already resolved
A lot of FTBFS (some due to my buildd running out of free space)
One DD grunting at me
Some DD giving good advices
A lot of thanks

Direct side effect: a lot of packages updated with more fixes thans the nostrip ones: triggering a rebuild is also triggering DD's work.

I'll do a rebuild in some weeks to check if fixed packages are really fixed and to rebuild packages that FTBFS and wil be fixed at that time.

And thanks to Mike Hommey for this idea.

rebuildd 0.2.1 released

jd — Wed, 08 Aug 2007 17:17:00 +0200

Running rebuildd for real since 48 hours, I've found some nasty bugs I have been able to fix. One of the biggest was me trying to do things that Python understood in a different maner, so it was quite interesting.

I'm now very happy about that project. From my point of view, I really managed to write a build handler daemon in a very better and more modern way that buildd used to do. It really fits my needs, and it's a real pleasure to use it. Ok, ok. stopping the self-congrats stuff.

I've updated the Web page to be more close to the 0.2.1 version.

Rebuilding on octave

jd — Tue, 07 Aug 2007 16:28:00 +0200

Due to the generosity of the administrator team from TuxFamily.org, I can now run rebuildd on octave, their (spare) HP NetServer LXr 8500 (8xPIII 500 MHz / 2 GB RAM). Which is quite faster that the P4 1.4 GHz I used for my first tests.

I can now run 24 builds in parallel.

Implementing Debian policy check with rebuildd

jd — Mon, 06 Aug 2007 16:47:00 +0200

After reading Mike Hommey's idea about buildd network for QA testing, I decided to try to implement the 10.1 policy recommendations check using rebuildd.

Here is how I proceeded.

Install and configure rebuildd

This is done by apt-get install rebuildd. Then, init database (I use sqlite) with rebuildd init.

Then I initialized build system with rebuildd-init-build-system. I had to add nostrip to DEB_BUILD_OPTIONS, so I decided to do quick and dirty and I edited /usr/bin/dpkg-build-package adding the good export info in the first lines.

Configuration file and post build cmd

My configuration file is simple and the one by default: I only changed email addresses and add a post_build_cmd script.

Here's the code for the script:


#!/bin/sh
dist=$1
pkg=$2
version=$3
arch=`dpkg --print-architecture`
RET=0
cd /var/cache/pbuilder/result || exit 1

for file in $(grep '.deb$' ${pkg}_${version}_${arch}.changes |cut -d\  -f6)
do
	echo Checking $file for stripped binaries...
	echo ---------------------------------------
	mkdir /tmp/$file
	dpkg -x $file /tmp/$file
	find /tmp/$file -type f | (while read line
	do 
		L=$(file $line | egrep 'ELF.*, stripped')
		if test ! -z "$L"
		then
			echo $L
			RET=1
		fi
	done; exit $RET)
	RET=$?
	rm -rf /tmp/$file
done
echo
echo ...done

echo Removing result files...
for file in $(grep-dctrl -sFiles . ${pkg}_${version}_${arch}.changes | sed '1d' | cut -d\  -f6)
do
	rm -f $file
	rm -f /var/cache/rebuildd/build/$file
done
rm -f ${pkg}_${version}_${arch}.changes
echo ...done

exit $RET

Fille the db

Then I fill the database with:

grep-dctrl -sPackage,Version .  /var/lib/apt/lists/ftp.fr.debian.org_debian_dists_sid_main_source_Sources | cut -d: -f2- | tr '\n' ' ' | sed 's/   /\n/g'| sed 's/  / /' | xargs -L1 -ixxxxxx echo xxxxxx sid | rebuildd-job add

And finally I just have to start rebuildd and let it do the job.

Argh, already 3 build logs received with stripped binaries…

Update: Bug list can be found here

rebuildd 0.2.0 released

jd — Thu, 19 Jul 2007 08:52:00 +0200

I just released rebuildd 0.2.0 and uploaded it to NEW queue.

Lot of changes expected, and got lot of changes. Database is quite stable now, but not compatible with previous release (but who cares). The main change is the rewrite of the Web server, which is now standalone (and so optional) and use a new framework, webpy, which is finally quite cool.

I'd like to enhance a bit more the Web interface, but it's getting on my nerves so I think I'm going to let others do it.

Some stats between 0.1.0 and 0.2.0:

 30 files changed, 1032 insertions(+), 355 deletions(-)

rebuildd Web 2.0 features

jd — Sun, 15 Jul 2007 17:04:00 +0200

Ahaha, no sorry, no 2.0 features for rebuildd yet.

I'm a bit lazy about implementing the Web interface, The problem is that everything is quite ready for it as you can see on demo server, but I can't convince myself to do HTML. I'm bad and even using mako templates and looking on different Python Web frameworks can't help me. I can't bear doing HTML pages, I don't know how to do clean things with CSS. And I don't want to learn.

A friend pointed me to Paste which should do a lot of things for me, but I'm too lazy to try to read this.

Actually, everything on the job-management side is quite ok. I'm just lacking of a Web interface to have a nicer overview of what's going on.

Anyway, I still like to do useless coding on client eye candy:

By the way, any help would be welcome.

About unit tests

jd — Fri, 13 Jul 2007 14:36:00 +0200

I just wrote something like 20 unit tests for rebuildd. Writing them, I found at least 2 critical bugs.

Either I'm bad, or unit tests are very useful.

Or both.

Big update for rebuildd

jd — Mon, 09 Jul 2007 11:45:00 +0200

During the last five days, I made a bunch of updates to rebuildd:

35 files changed, 709 insertions(+), 190 deletions(-)

That's mainly small enhancements, and a brand new Debian packaging. It's now very usable from my point of view. Moreover, the repository is now on Alioth.

There's still some code improvement to do and some features to add, but I'm quite happy with it.

Consequently, I just released the first version, numbered 0.1.0.

rebuildd online preview

jd — Wed, 04 Jul 2007 11:22:00 +0200

I just set up a small rebuildd test daemon on a dev box. You can see it here.

Five days of coding: rebuildd

jd — Tue, 03 Jul 2007 19:14:00 +0200

Last week I decided to begin a new project, whose name is now rebuildd.

If the name is not explicit to you, let me explain: it's like buildd, but its first purpose is to rebuild packages.

If you want to know more, you can take a look on the Web page I just set up.