Here at Nias Digital we produce a number of products which appear as USB/serial devices.  Each time you plug one in Windows comes over all helpful and caches that particular device’s VID/PID/serial tuple against the COM number so next time you plug it in it appears in the same place.

The problem comes from the fact that many programs, including the Windows hotplug notification mechanism, only provide 7 or 8 bits of storage for the COM number – as soon as you plug in your 128th device some things break and by the 256th nothing works all that well any more.  The most common response to this problem is to manually change the COM number in the Advanced tab of the port settings dialog however all the numbers which Windows has cached as above are marked as unavailable – with the right device history there can be no ports left available in the working range!

To get around this some Registry hacking is in order.  First, clear out the unneeded VID/PID/serial cachings.  These are dependent on the manufacturer of the USB/serial bridge hardware but for FTDI parts you’ll find the mappings in
HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Enum/FTDIBUS
Delete all the keys inside that path and you’ll clear the mappings.  The actual COM numbers though will still be marked as unavailable.  To clear this, find the key
HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/COM Name Arbiter
(yes it’s the only key I know of with spaces in the same) That key has a binary value called “ComDB” which is a bitmask of all reserved COM port numbers.  For example, if it reads
ff f0 00 00...
Then then COM1 through COM12 will be marked as reserved.  If you’re running in to the same problems as me then at least the first 127 bits in the string will be set.  All you have to do is go through and 0 out bits corresponding to devices you don’t care about any more.  In my case I’ve got 10 internal serial ports (mostly on PCI serial expanders) so I edit my ComDB field to start with ff c0 and the rest 0s.  Be a little careful if you’ve got Bluetooth serial ports, they take up big chunks and removing their entries can break things.

Once that’s done, rejoice!  Plugging in new USB/serial adaptors (or old ones whose cachings you’ve removed) will now have their COM numbering starting down low again.

NOTE: Tested on XP and seems like it should work on later versions; if you have an addendum, post as comments

So, guess what, a blog post that’s not only useful, but totally not to do with VCS!  And you didn’t think it was possible…

Instead, today we’ll be installing the Player/Stage robotics server and simulation platform on MacOS 10.6.2 Snow Leopard.

Player

Use macports to install the basic tools of the trade; cmake and pkgconfig.  Don’t use it to install libtool, make sure you’re using the version supplied with the XCode tools (use “which libtool” to verify this.

Unpack the player source (I used 3.0.1), make a “build” subdirectory, change to it and run
export CXXFLAGS="-arch x86_64"
cmake ../
ccmake ../
In the ccmake menu, set “CMAKE_OSX_ARCHITECTURES” to “x86_64″.  Player doesn’t seem to automatically install to the SDK so either you can install it to, eg, “/usr” and copy it to the SDK later or just set the install path to the SDK path with “usr” tacked on the end.

If any of the later stages fail, you might find you need to disable the python bindings as well.

Press ‘c’ to configure and ‘g’ to generate.  If ‘g’ doesn’t come up immediately, keep pressing ‘c’ until it does (or until it gives a sensible error).  For some reason it doesn’t always work first time.

Once done,
make
sudo make install
If you’ve installed to your SDK then move on, otherwise
sudo cp /usr/lib/libplayer* <SDK DIR>/usr

Stage

I’ve heard tell that macports can install a 64-bit version of FLTK for you.  If that works in your case, go for it.  For myself, I had to compile FLTK manually.  The only version I can get to compile on 10.6.2 is the SVN snapshot of the 1.3.x branch.  2.0.x won’t work, neither do earlier versions from the 1.y branch.

Unpack that and
./configure --prefix=<your favorite prefix>
make
sudo make install
I found I had to change src/fl_draw_pixmap.cxx::339 to add an explicit cast, from:
U32* colors = d.byte1[*p++];
to
U32* colors = (U32*)d.byte1[*p++];
Once again, if you didn’t use your SDK as your prefix,
sudo cp /usr/lib/libfltk* <SDK PATH>/usr
Now Stage is ready to be built.  Make sure CXXFLAGS is still set to force a 64-bit build (otherwise just export it again), unpack Stage, make a build subdir, change to it and run
cmake ../
ccmake ../
Once again, set architecture and install paths, configure, generate and exit.  If the configuration doesn’t find the player installation, make sure that PKG_CONFIG_PATH includes the player installation location’s pkgconfig dir; eg if you installed to /usr, make sure PKG_CONFIG_PATH includes /usr/lib/pkgconfig.

Once Stage configuration is complete,
make
sudo make install
For some reason, Stage doesn’t give the player plugin the right name.  If running Player fails having not found the libstageplugin, you’ll need to
sudo ln -s /usr/lib/stageplugin.so /usr/lib/libstageplugin.so
(assuming you installed to /usr/lib)

You should be good!  Well, worked for me, YMMV.

G’luck

Now for the great time-saver of this series of tutes.  You’re going to spend a lot of time specifying revisions when working with GIT, but it does a great job of easing this for you.

The way most people get revision specifiers is by using git-log to find the commit they’re looking for and cut-and-pasting the SHA1 hash associated with it.  This hash is the basis of all git specification methods.

Note that the examples I give here relate to a particular command but will work with pretty much any of them.  Most GIT commands pipe their given specifiers through git-rev-list to get the commit IDs out and the docco for that command is a good place to get more info on what I’m going to say.

Tags

The most common form of easy revision specification is tagging.  Tag a version simply by

git tag -a tag_name [SHA1 hash of target commit]

or just

git tag -a tag_name

if you’ve currently got the target commit checked out.

You can use -u <key-id> instead of -a to sign the tag and -v to verify a signed tag.  Once you’ve got a revision tagged, that tag name simply acts as a synonym for the SHA1 has it’s linked to.

Relative

Relative specification is useful if you’re working with commits very close to HEAD.  Drop back 1 or 2 revisions with

git log HEAD^

or

git log HEAD^^

When going further back this can of course be a pain so instead of

git log HEAD^^^^^^^

you can use

git log HEAD~7

to go back 7 commits at once.

Time-based

Now GIT starts to get cool.  It keeps a track of at what your tree looked like at different points in time and you can access this information in a jiff.  Say you’ve just pulled in a bunch of changes and want to look at everything which wasn’t there when you left work; git does a pretty good job of parsing English-like time specifications so you can simply do

git log @{6pm.yesterday}

How about what was on some other branch the day before yesterday but not in your branch now

gitk other_branch@{day.before.yesterday} ^HEAD

Note that a carat before a revision name is logical not, a carat after is one commit before.

As you can see, the @{} specification is very powerful, allowing you to get the status of a tree at a particular time either in normal date specifications or more English-like phrasing.  This is not, however, the same thing as the commits since a particular time.  For example, if you pull in a big old bunch of commits, @{5.minutes.ago} will point to the tree before the pull, not a tree with all but the commits timestamped in the last 5 minutes.  For this you need the –since specifier

gitk --since="5th October"

Other neat specifiers

There are other things you can use to select commits too.

gitk --author="Ben Nizette" --committer="Haavard Skinnemoen"

will show all the patches I wrote which went through Haavard.

gitk --since="yesterday" --grep="gpio"

Will show all patches in the last day which claimed to have something to do with gpio.  We can limit this further

gitk --since="yesterday" drivers/gpio/

will simply show the last day’s commits which touched something in the drivers/gpio/ directory.

gitk --extended-regexp="^foo.*[0-9]*$"

Well that does pretty much as you’d expect really.

gitk --author="Linus Torvalds" --no-merges

Shows you how much actual coding (non-merge) work Linus actually does.

The dots

There can be some confusion when using GIT as to what a series of dots actually does.  2 dots (“..”) is a range specifier

git log HEAD~5..HEAD~2

shows all commits between the 5th and 2nd newest.  It’s a shorthand for

git log HEAD~2 ^HEAD~5

3 dots (“…”) is a symmetric difference, that is

git log HEAD~5...HEAD~2

will show all commits that are in HEAD~5 or HEAD~2 but not both.  In the case where they’re just different points on the same branch 2 and 3 dots do the same thing, but suppose you’re about to merge a pair of branches and you want to see what might conflict,

gitk merge_1...merge_2

shows the commits which are in one branch or the other but not both – i.e. it just lists the commits which could possibly conflict.

You can use this during a merge as well to narrow down which commits introduced the conflict, read their changelog and hopefully get a better understanding of the correct conflict resolution.  Say you’re out at sea with a conflict in complex/subsystem.c; simply

gitk merge_1...merge_2 complex/subsystem.c

To view all patches which could possibly cause that conflict.

Anyway, that’s a quick overview of the specifiers that make my life easier.  You can refer to “man git-rev-list” for other cool options to try out too.

Rebase is generally used to move a set of changes started against one base to another. 99% of the time, you’re developing something against an upstream branch, that branch moves and you want to get the latest and greatest. You can either merge upstream with you and loose your clean history, or rebase. Rebase has grown to be a fairly general way to change history though; you can use it to reorder, change and drop commits and move your development not just further up the same branch but across to somewhere else entirely.

Before I get going, please have a read of the last entry which has some tips about working with rebase; the biggest of these is simply Keep Rebased Trees Private.

GIT tree storage

A lot of rebase semantics make more sense if you realize that a GIT commit ID doesn’t really refer to a commit but rather to the state of your source tree immediately after that commit.  Many people expect

git diff abcdef1234

to give a unified diff of the changes introduced by that commit, but that commit ID, as above, doesn’t represent a commit as such but a tree.  What’s the correct output when you ask for the diff of a tree?  It just doesn’t make sense.

Branch names really just represent a specific commit ID, the ID of the most recent commit on that branch.  A branch name has no internal concept of where it’s currently based; this is one of the most common errors when rebasing – telling GIT the branch to operate on is based on the wrong commit.

Rebase

As I mentioned above, the most common use of rebase is simply to shift your commits further up an upstream branch.  If you’re following my advice about tracking upstream in the “master” branch and doing work somewhere else (call it “my_branch”) then this is accomplished by

git rebase master my_branch

or just

git rebase master

if you’ve got my_branch checked out.  Because GIT only deals with commits, you have to have committed current work before you run.

But remember branch names are really just commit IDs, so this is read by GIT as “take all the commits in my_branch which aren’t in master and move them to the current HEAD of master”; or else “squish all the master history in before any my_branch history”.

Rebase interactive

This was described in the previous entry.  By appending the –interactive flag to the rebase (but otherwise using the same syntax) you get an opportunity to change the commits in my_branch as they get moved to the destination.  Note that the destination head can actually be exactly where my_branch is already based.  While this makes the rebase a no-op usually, in interactive mode you can use it to change commit history at any time.

onto

The –onto switch allows you to move a set of commits from one branch of your repo to somewhere completely different.  Say you’re developing feature_2 which is based on feature_1 but feature_1 is not yet in the upstream branch.  After a bit of a brainwave you realize feature_2 can be rearranged to not have that dependency and is ready to move upstream by itself.  Do

git rebase --onto master feature_1 feature_2

This is read by GIT as “take all the history from feature_2 which isn’t in feature_1 and stick it on the end of master”.  Because the –onto switch has to come before the other 2 branch names the syntax seems backwards; just remember that the last 2 branch IDs are the same no matter what version of the rebase command you’re using (just “base” and “head” branches respectively).

Another feature of –onto rebasing is an alternative way to drop commits from the middle of history.  Suppose you’re on branch “my_branch” and the last 5 commits have names “commit1″ to “commit5″ (I’ll be writing an entry soon on the best ways to specify actual revisions).  Then

git rebase --onto commit2 commit4 my_branch

Will take the stretch of commits between commit4 and the branch head and move them to be based at commit2 – i.e. you’ve just dropped commit3 from your tree.  (note the sloppy naming here, as I’ve been at pains to point out commit names identify trees, not commits, so actually calling them commitN isn’t ideal )

When all goes wrong

During a rebase, you may of course get a number of conflicts between your code and the tree on to which you’re trying to move it.  All such conflicts will be marked with standard merge markers (“<<<< >>>>”) which you can grep for, use git diff to see or just read the error log.  Once you’ve got rid of all these sites, mark them as fixed by

git add my-fixed-file.c

but instead of committing, run

git rebase --continue

If you’re stuck and want to get out of there, run

git rebase --abort

to undo all rebasing actions.  Finally, if a commit is causing conflicts because it’s no longer needed,

git rebase --skip

will skip that particular commit.  Note that this will loose that commit, be careful!

Finally, git rebase is smart enough to recognize commits which introduce the same changes as each other but have different descriptions and skip them.  That is, if you’ve got a commit in the tree you’re rebasing but has already been accepted in to the new base, git will automatically skip that patch.

Cool, huh?!  That’s it for now, next time I’ll show you the fastest and coolest ways to specify the commits you care about.

Version control seems to be one of my favorite subjects for blogging; here’s a slightly new thing – a tutorial style thing showing off the coolest bits of GIT I’ve come across.  This is part 1, offering tips for easy GIT usage especially for people coming from a dissimilar VCS.

Tip 1: Commit Often.  Really Often.

Git offers a great number of features, but only for commits.  Unlike other VCSs though, a commit isn’t final; you can still play with the ordering, combine commits, drop them, move them however you see fit.  There is no down-side to committing often, if you end up with a big commit mess, use interactive rebasing to organise it all again.

Interactive Rebase

Suppose you’re developing your stuff on a branch called my_branch which is based on the master branch shared with others.  You’ve taken tip 1 and you’ve ended up with a big mess of commits with no logical ordering or cohesion.  Interactive rebasing allows you to reorder, combine and drop commits.
git rebase -i master my_branch
Will bring up $EDITOR with a list of all commits between master and my_branch.  If you change the order of commits in this list, it will change the order of the commits in your branch.  Replace the “pick” keyword with “s” to squish a commit in to the previous one.  Delete a commit from that list and it will be deleted from your branch.  Save and exit and GIT takes care of the rest.

In this way you can organise, say, a new feature in to a series of logically separated actions for easy review by your peers.

Tip 2: Pro: Rebase changes history.  Con: Rebase changes history.

OK so rebasing is great.  You can move your changes on top of a newer base or, like above, you can reorganize your commits to be more useful.  Be very very careful though as you must never rebase something that someone else has access to.

Rebasing changes history but it doesn’t leave a marker anywhere saying what it has done.  If someone has pulled your branch and you rebase it, their version won’t be updated to reflect this.  They will have to remove their copy of your branch and pull it again from scratch.

Tip 3: Branches are cheap; use them for everything

Unlike some other VCSs, branching is very easy to do and merging back is just as easy.  Separating your working tree in to different branches for each feature for example can allow you to test each feature on a stable base.  You can leave the “master” branch for tracking upstream and working in private branches that others aren’t going to be annoyed if you break.

In particular, if you’re starting out with GIT and aren’t sure how to do something, branch off and try it there.  If it works, merge it back, if it fails kill the branch and start again.

Like a lot of things though, use this tip with care.  If you end up with a massive branch, getting it merged back in to the mainline gets harder and harder.  Both because there’s more likely to be conflict and also because the final version is going to be hard to review.  In short: Big branches eventually have the same problems as any patch bombs; be careful.

OK so maybe it’s time I write something useful.

Gitosis:

• Pros: A fantastic program which simplifies the task of securely hosting git repositories.
• Cons: It doesn’t do read-only unsecured shares

Getting Gitosis

There are gitosis packages for mos major distros but I haven’t had much luck getting them to work, best to get the latest source:
git clone git://eagain.net/gitosis.git
cd gitosis
python setup.py install

Don’t forget to install python setuptools.

Preparing the System

Now you have to set up a user for gitosis to run as.  The hosted repos will be stored in this user’s home dir, so choose wisely.  On my system I did
sudo adduser \
--system \
--shell /bin/sh \
--gecos 'git version control' \
--group \
--disabled-password \
--home /home/git \
git
Now, every user who interacts with Gitosis will need an SSH key.  Check ~/.ssh, if you’ve got id_rsa.pub in there you’re good otherwise generate one by
ssh-keygen -t rsa
Now to initialize a gitosis instance for your created user with your personal user authorised to administer. Copy your public key to /tmp (so you don’t have permission problems later on) then
sudo -H -u git gitosis-init < /tmp/id_rsa.pub
Now comes the sweet bit of Gitosis – administration isn’t actually done on the server, all gitosis hosting configuration is itself hosted by gitosis. On your personal machine then, clone the admin repository
git clone git@YOUR_SERVER_HOSTNAME:gitosis-admin.git
cd gitosis-admin
Now the administration workflow is

1. Edit gitosis-admin/gitosis.conf
2. Add user’s public keys to gitosis-admin/keydir
3. git add keydir/*
4. git commit -a -m 'Describe your changes'
5. git push

Creating a new Repo

OK so now you’re set up and authorised for administration, sweet.  Let’s set up our first repository.  Open gitosis-admin/gitosis.conf and add a section like
[group groupname]
members = usernames
writable = new-repo

Where of course group names should be descriptive, new-repo is the name of, you guessed it, your repository and usernames is a space-separated list of fully qualified usernames allowed to write. That is, “user1@somecomp user2@anothercomp” etc.

Save this and exit.  If there are users other than yourself in the list, copy their keys to gitosis-admin/keydir.  The keyfile filenames should be the fully qualified username with a .pub extension, eg user1@somecomp.pub.

Add and commit any changes and push them back to the server like
git add keydir/*
git commit -a -m 'Added new-repo with users user1, user2'
git push


And that’s the configuration done. Now you of course need to give the repo content. We’re going to make the repo locally and push it up, so

cd ~/code/my-repo
git init
#make some changes, commit
git remote add origin git@SERVER_HOSTNAME:new-repo.git
git push origin master:refs/heads/master
Note that the push will fail unless you have at least 1 commit on the master branch. Once you’ve done this once, git will remember the local and remote refs so you’ll only need to
git commit [OPTIONS]
git push

And that’s it! There are a fair few config options I’ve left out, like using group names as user names in writable lists, read-only access (but not public access) etc. Check out example.conf in the directory where you originally cloned gitosis for more info.

The excuse: Work’s got massive so I haven’t had a good chance to work on Aevercore in ages.  However I’m still sold on the idea and have a bunch of almost-working scripts so it isn’t dead, just sleeping.  Stay tuned

The browser these days is the centre of a hells of a lot of desktop activities.  I mean, how often do you actually spend “browsing” the internet?  I know I spend bugger all.  I go to Facebook, Twitter, LWN, LinuxForDevices, XKCD and a handful of other websites all the time but fairly rarely actually sit down and browse.  And I suspect you’re the same.

If the regularly visited sites are apps the browser is a complete secondary desktop environment with tabs instead of windowing and a whole lot besides.  Why do I want another desktop when GNOME/KDE work so well?  Why shouldn’t web apps be Revealed, ALT-Tab’d and resized like anything else?

There are solutions – both major Linux desktops have widgets which can bring some sites to my desktop and Twitter’s well defined API means that I can just use Gwibber.  All well and good where it works but I want to treat every web app this way.

In steps uzbl, a browser conforming to the UNIX philosophy that one program should do one thing and do it well.  In fact, browser may even be overstating it – let’s try interactive web renderer.  It uses the WebKit renderer so you get CSS, Javascript, Flash, all the things you need but nothing you don’t.

Using uzbl and a little bit of BASH we get Page Apps.  Normal apps which can be launched from GNOME-DO, run dialogs or menus but are nothing more or less than the web apps we all know and love.

page apps in action

OK so maybe people won’t get it.  What’s the problem with 1 extra mouse click or a few extra key strokes?  But they’re missing the point, it isn’t about that, it’s about promoting web apps to first class citizens.

By default uzbl is modal, VIM-like, so as yet my PageApps don’t quite act as you’d expect.  But they will.  You can get uzbl at http://www.uzbl.org/ and I’ll release my PageApps scripts just as soon as they’re ready.

Well it’s been a while since my last post. Back to it!

Over the last few weeks I’ve been playing with the idea of doing my own Linux distribution based, in concept, on Tinycore Linux. It will target (guess!) the AVR32 family of uCs in the first instance hopefully ARM shortly thereafter.

The working name is Aevercore or avc for short. For those who don’t know Tinycore, the idea is that the entire distribution is run from RAM for a few reasons.
1) Increased speed.
2) No bitrot or residue – the system is at exactly the same state each startup.

Of course not everyone sees 2) as an advantage and indeed some times it isn’t. To that end there is a persistence list – at each shutdown the items on the list are written to persistent storage and at each boot, they’re extracted again. In embedded space, we also have
3) Running from RAM means less flash wear.

Tinycore, and avc as well, also support running from persistent storage in the usual manner. Anything to be run from persistent storage is loopback mounted and symlinked in to place. This means that all writebacks still hit RAM and you get all of the cleanliness advantages without eating more RAM than needed.

To this end, I’ve been experimenting with build systems. Ideal would be T2. After a good few days hitting it though, I can’t get the darn thing to build for AVR32 and can’t be bothered learning what’s going wrong. I know a fair bit about Buildroot so I’ll probably run with that in the first instance.

So, what I need first. Of course the first thing is just time. Once a current project at Nias is done I should have a bit more of that. Second is good compile hardware. I certainly can’t go compiling a full operating system on my Laptop and the desktop is creaking arthritically too.

Who knows, if I can break my habit of not following good idea through, I’ll have an alpha out in 6 weeks or so.

Just saw this interesting little piece of wisdom from the guys at the Sun datacentre in Fyshwick

Cool eh?  I wonder how he discovered that particular effect…