• All tables have an “id” field which serves as its key.
• The database is used through an ORM which uses this to retrieve/update/delete records.
• These systems discourage or disallow multi-column primary keys.

The upside is that this method requires little thought, and is often marketed as “easy”, “simple”, etc. This approach may require little work at first, but it can bite you later. I hope to show how such a structure can become hard to maintain and make it hard to ensure data quality through a few examples.

ID-per-table design example

We want to record competitors in a series of dance competitions. We also want to have a simple checkbox to say they’ve paid entry fees

CREATE TABLE competition (
  competition_id INTEGER PRIMARY KEY,
  friendly_name TEXT NOT NULL
);

CREATE TABLE competitor (
  competitor_id INTEGER PRIMARY KEY,
  first_name TEXT NOT NULL,
  surname TEXT NOT NULL,
  phone_number TEXT NOT NULL
);

CREATE TABLE entrance (
  entrance_id INTEGER PRIMARY KEY,
  competition_id INTEGER NOT NULL REFERENCES competition (competition_id),
  competitor_id INTEGER NOT NULL REFERENCES competitor (competitor_id),
  paid_fee BOOLEAN NOT NULL
);

We’ve already got our first issue – what does it mean for a competitor to be entered in the competition twice? We probably don’t want to allow that (or if we do need to be clear about that). We can stop that fairly easily – slap a UNIQUE constraint on those fields:

CREATE TABLE entrance (
  entrance_id INTEGER PRIMARY KEY,
  competition_id INTEGER NOT NULL REFERENCES competition (competition_id),
  competitor_id INTEGER NOT NULL REFERENCES competitor (competitor_id),
  paid_fee BOOLEAN NOT NULL,
  UNIQUE (competition_id, competitor_id)
);

Now we want to track the different divisions within the competition and awards:

CREATE TABLE division (
  division_id INTEGER PRIMARY KEY,
  competition_id INTEGER NOT NULL REFERENCES competition (competition_id),
  division_name TEXT,
  UNIQUE (competition_id, division_name)
);

CREATE TABLE award (
  award_id INTEGER PRIMARY KEY,
  division_id INTEGER NOT NULL REFERENCES division (division_id),
  competitor_id INTEGER NOT NULL REFERENCES competitor (competitor_id),
  place INTEGER NOT NULL,
  UNIQUE (division_id, place),
  UNIQUE (competitor_id, division_id)
);

What’s wrong with this you may ask? If systems get out of sync then someone could have an award record without being entered in the competition.

You can try to check for this in application code, but unless you check everything everywhere and very carefully manage locking of database rows (SELECT … FOR UPDATE and such) then you can still get in trouble. One process could kick someone out of the competition for failing to pay their fees shortly before another records an award. This is especially dangerous when information gets stale over time, for example because it comes from a printout, is copied to an offline system or a record is cached. We should really want to catch this!

A bad fix

We could make award point to the entrance record instead of directly to the competitor, containing an entrance_id . We can enforce such a constraint by changing our IDs to go through another record, but then we’ll be using an entrance_id on the award instead of a competitor_id.

CREATE TABLE award (
  award_id INTEGER PRIMARY KEY,
  division_id INTEGER NOT NULL REFERENCES division (division_id),
  entrance_id INTEGER NOT NULL REFERENCES entrance (entrance_id),
  place INTEGER NOT NULL,
  UNIQUE (division_id, place),
  UNIQUE (entrance_id, division_id)
);

I would call the entrance_id here an opaque identifier – it means nothing without looking at the table it refers to. Every query has to link through that table even if it uses none of the non-key data – performance degrades and opportunities for mistakes increase. It’s also plain counter-intuitive – we issue an award to a person, not their entrance record.

Imagine also that you can enter freely, but can only claim your award if you submit blood samples for a drug check. The award needs to reference both the drug check record and the entrance record to check they exist.

CREATE TABLE award (
  division_id INTEGER PRIMARY KEY,
  entrance_id INTEGER NOT NULL REFERENCES entrance (entrance_id),
  drug_check_id INTEGER NOT NULL REFERENCES drug_check (drug_check_id),
  place INTEGER NOT NULL,
  UNIQUE (division_id, place),
  UNIQUE (entrance_id, division_id)
);

Do we know that the entrance record and the drug check record point to the same person? Same goes for competitions – was the drug check done at the same competition we’re issuing the award for? Probably going to be OK in this case, but we have one record used mostly for checking fees and another used mostly for rules enforcement. The people managing those are probably different and work differently so we’d be better off making sure things can’t get out of sync.

How would we fix this with compound keys?

To clarify, this is not a rant against using any artificial keys. Sometimes there are two Rob Smiths and you need to ask “are you the Rob Smith from …” instead of making the system assume they’re the same. I’m going to keep the competitors and competitions identified by numbers, because people are hard to key (not everyone even has two names) and we could want to reuse competition names.

Firstly, as is often the case, the places where we’ve been using UNIQUE are actually the natural identifiers of the table. Let’s make them the keys:

CREATE TABLE entrance (
  competition_id INTEGER NOT NULL REFERENCES competition (competition_id),
  competitor_id INTEGER NOT NULL REFERENCES competitor (competitor_id),
  paid_fee BOOLEAN NOT NULL,
  PRIMARY KEY (competition_id, competitor_id)
);

CREATE TABLE division (
  competition_id INTEGER NOT NULL REFERENCES competition (competition_id),
  division_name TEXT NOT NULL,
  PRIMARY KEY (competition_id, division_name)
);

Now on to the award record:

CREATE TABLE award (
  competition_id INTEGER NOT NULL,
  division_name TEXT NOT NULL,
  competitor_id  INTEGER NOT NULL,
  place INTEGER NOT NULL,
  PRIMARY KEY (competition_id, division_name, place),
  FOREIGN KEY (competition_id, division_name) REFERENCES division (competition_id, division_name),
  FOREIGN KEY (competition_id, competitor_id) REFERENCES entrance (competition_id, competitor_id)
);

We have all the columns in this table that we need to be able to enforce the constraint that the competitor is entered in the competition – we have some record of whether they’ve paid before we give them any trophies. The database will enforce these constraints between transactions and cannot store data which breaks that rule.

We can extend this – require that people actually danced in that division and add in the drug check example from the bad fix:

CREATE TABLE entered_division (
  competition_id INTEGER NOT NULL,
  division_name TEXT NOT NULL,
  competitor_id  INTEGER NOT NULL,
  judging_number INTEGER NOT NULL,
  PRIMARY KEY (competition_id, division_name, competitor_id),
  UNIQUE (competition_id, division_name, judging_number),
  FOREIGN KEY (competition_id, division_name) REFERENCES division,
  FOREIGN KEY (competition_id, competitor_id) REFERENCES entrance
);

CREATE TABLE drug_check (
  competition_id INTEGER NOT NULL,
  competitor_id INTEGER NOT NULL,
  lab_sample_number INTEGER NOT NULL,
  PRIMARY KEY (competition_id, competitor_id),
  UNIQUE (competition_id, lab_sample_number),
  FOREIGN KEY (competition_id, competitor_id) REFERENCES entrance (competition_id, competitor_id)
);

CREATE TABLE award (
  competition_id INTEGER NOT NULL,
  division_name TEXT NOT NULL,
  competitor_id  INTEGER NOT NULL,
  place INTEGER NOT NULL,
  PRIMARY KEY (competition_id, division_name, place),
  FOREIGN KEY (competition_id, division_name, competitor_id) REFERENCES entered_division,
  FOREIGN KEY (competition_id, competitor_id) REFERENCES drug_check
);

(P.S. REFERENCES by default targets the primary key columns – can often leave off the column list)

This means that:

• You have to enter the competition to enter any divisions.
• You have to enter a division, being allocated a judging number (number pinned to back/leg – alternate key used by the judges) in that division before you can win an award.
• We can never give an award to somebody who hasn’t had a drug check. The drug check record we refer to can’t be for a different person or a different competition – we use the same competitor_id and competition_id in multiple foreign keys.

The tables have many duplicated columns, but the ability to enforce these constraints is valuable. As I’ll discuss next, we also gain some performance advantages by duplicating these columns.

What about performance?

In relational database engines these sorts of designs will also tend to use simpler queries which perform much better – we don’t need to join through lots of tables to find who or what a record relates to. An example task here would be to find the winners of a competition and enter them in an invitation-only event.

Everybody who placed in competition 999 (any division) is automatically entered in competition 1000:

INSERT INTO competition (competition_id, friendly_name)
    VALUES (1000, 'Invitational Jam 2020');

INSERT INTO entrance (competition_id, competitor_id)
    SELECT DISTINCT 1000, competitor_id
    FROM award
    WHERE competition_id = 999 AND place < 4;

We could easily create an index on award(competition_id) or even award(competition_id, place) to make those queries cut right down to the records they’re likely to need quickly. With the id-per-table style you might have:

INSERT INTO competition (competition_id, friendly_name)
    VALUES (1000, 'Invitational Jam 2020');

INSERT INTO entrance (competition_id, competitor_id)
    SELECT DISTINCT 1000, competitor_id
    FROM entrance
    JOIN division_entrance ON (entrance.entrance_id = division_entrance.entrance_id)
    JOIN award ON (award.division_entrance_id = division_entrance.division_entrance_id)
    WHERE competition_id = 999 AND place < 4;

There are a couple of ways the database system could evaluate that, and some indexes that would help, but no way of avoiding joining those tables to each other. It will be a lot slower!

Conclusion

The id-per-table design pattern may reduce the amount you have to think about when making your database, but it can make it much harder to enforce constraints in the database. While there is often a way to do so, it can make enforcing uniqueness of records or requiring records to share a common relationship difficult. This tends to force integrity checking in to application code, where it may require additional queries and requires careful locking to make these checks safe between transactions.

It also often has serious performance implications. Creating opaque identifiers for tables requires us to link through those tables even when not using their data, just to get to the tables we actually want. This can turn queries in to twisted monsters after a few rounds of iterative development.

I strongly encourage people to pick up some of the “traditional” theory about databases and use compound keys on any record which implies or represents a relationship between other entities.

Javascript, while capable of much more, was originally used to add small pieces of intelligent behaviour to web sites. It might be used to validate a form as you fill it in or to hide a block until you click a button. Nowadays, Javascript is used for a lot more, but is still saddled with being treated as a language for “decorating” web sites rather than serious work.

All too often, Javascript is still treated as if it’s just doing those little bits and pieces, when we ask much more of it. Code meant for validating forms is adapted to send the form via AJAX and completely override the normal form submission mechanism. Instead of a web site having a couple of pieces of Javascript from the site itself, we import Javascript from advertising networks, social media sites, analytics companies and other third parties and end up with many more, possibly interfering scripts.

Over the last few years I’ve gone from doing your standard copy-and-paste Javascript to really learning to treat it as a real language.

Javascript should be modular

One classic example of where the copy-paste script falls apart is when we have clashes between those snippets. An example I’ve seen in real life is when one script tried to set the global value “random” to a function for generating random numbers, while another script set the global value “random” to be an actual random number. One script was then trying to use a number as a function, which didn’t work very well.

It went something like:

In one file…

function createAdCall(location, params) {
  random = Math.floor(Math.random() * 10000000)
  ...
  ad.innerHTML = "<script type=\"text/javascript\" src=\"http://ad.server/give/me/ad?cache_buster=" + random +"\"></sc" + "ript>"
}

In another file…

function random() {
  return 3;
}
function blah(...) {
  ...
  var foo = random();
  ...
}

Whenever the createAdCall was run it would stomp on “random”, replacing the function. Both parties were to blame. The party setting random globally should have been giving it a local scope, but the other party shouldn’t be using the global “random” either – it’s a very common term.

In almost every other language, it’s taken for granted that using global variables or common names in the global namespace is an excellent way to tell your peers that you’re clueless, but people do this in Javascript all the time.

In Javascript it’s extremely easy to create modules of functions. Here are two, different ways of creating modules:

In one file…

ExtraAwesomeAdNetwork = {}
ExtraAwesomeAdNetwork.ads_requested = 0
ExtraAwesomeAdNetwork.createAdCall = function(location, params) {
  var random = Math.floor(Math.random() * 10000000)
  ExtraAwesomeAdNetwork.ads_requested++
  ...
  ad.innerHTML = "<script type=\"text/javascript\" src=\"http://ad.server/give/me/ad?cache_buster=" + random +"\"></sc" + "ript>"
}

In the other file…

(function() { // Start private scope by wrapping code in a function
  // Create our random function. Will be preferred over global "random" by
  // anything in this block, and invisible outside it.
  function random() {
    return 3
  }
  function blah() {
    ...
    var foo = random();
    ...
  }
  // Export functions
  MyAnalyticsToolkit = {
    "blah": blah
  }
})() // End private scope - calls the function immediately

We now call the functions in the form ExtraAwesomeAdNetwork.createAdCall(…) and MyAnalyticsToolkit.blah(…) . If both define functions with the same name, we’re keeping them separate. Each module can store its own private data without fear that anything else will trample on it.

Javascript supports objects

Javascript is entirely capable of doing full OO-style development. This can be used to enhance the modular approach above:

function Car(fuel_type) {
  this.fuel_type = fuel_type
}

Car.prototype.start = function() {
  if (this.fuel_type == 'petrol') {
    alert("Brrrrrrooom...")
  } else if (this.fuel_type == 'diesel') {
    alert("BrrrrrroooOOOOOOOOOOOOOOOOOOOOOOMMMMMMM!")
  } else {
    alert("Your " + this.fuel_type + " powered car is too hard to start.")
  }
}

var petrol_car = new Car('petrol')
petrol_car.start()
var diesel_car = new Car('diesel')
diesel_car.start()
var pedal_car = new Car('pedal')
pedal_car.start()

Especially when combined with modularisation as above, this can lead to much cleaner code.

Events

Javascript used to be run by putting code in HTML attributes, but this has a few issues:

<form onSubmit="return validateName(document.getElementById(\"name\").value)">
<input id="name" type="text" value="Tony" />
</form>

A few issues in that but really it’s just plain ugly. It mixes your Javascript logic in with the HTML and content.

Much better to attach an event listener. You can do this in native Javascript using attachEventListener, but I do prefer to use a library to simplify this. Here’s an example with jQuery:

<form id="name_form">
  <input id="name_form_name" type="text" value="Tony" />
</form>

In your Javascript file:

$(document).ready(function() {
  $('#name_form').submit(function() {
    return validateName($('#name_form_name').val())
  })
})

Keeping references

I get annoyed when I see code creating HTML with IDs then relying on them having some structure to find them again:

<script>
  function highlightRow() {
    var id = this.id.replace(/^highlight/, '')
    $("#row" + id).style('background', 'red')
  }

  nextRow = 0
  function createRow(container) {
    var myRowId = nextRow++
    var row = $('<div id="row' + myRowId + '"> ... <button id="highlight' + myRowId + '">Highlight</button></div>');
    $('#highlight' + myRowId).click(highlightRow)
  }
  
  $(document).ready(function() {
    for (var i = 0; i < 25; i++) {
      createRow($('#rowContainer'))
    }
  })
</script>
...
<div id="rowContainer"></div>

Things are simpler and more reliable when you keep references to the DOM nodes around and use them. Javascript functions are closures – they can keep accessing variables from the scope they were created in.

<script>
  function highlight(node) {
    node.style('background', 'red')
  }
  
  function createRow(container) {
    var block = $('<div id="row' + myRowId + '"> ... <button class="highlight">Highlight</button></div>')
    block.find('.highlight').click(function() {
      highlight(block)
    })
    container.append(block)
  }

  $(document).ready(function() {
    var rowContainer = $('#rowContainer')
    for (var i = 0; i < 25; i++) {
      createRow(rowContainer)
    }
  })
</script>
...
<div id="rowContainer"></div>

Final words

Javascript in the wild tends to be a messy, hideous beast, but it doesn’t have to be that way. Modularise your code rather than polluting the global namespace, use objects to store persistent state, include a library to simplify DOM access, attach event handlers in JS rather than embed in the HTML, keep references to DOM elements rather than abuse IDs and above all, treat Javascript as a real language. Trust me, it’ll soon be a lot less painful.

When a router can’t forward traffic as quickly as it’s getting it, it starts dropping packets. TCP takes advantage of this to measure link capacity – it increases the amount of traffic being sent until it starts losing packets then reduces the speed again. This generally works very well, but has an issue with packet loss not related to the data being sent. Many networks just have noise or lose packets randomly, and TCP will tend to back right off to a trickle when this happens completely unnecessarily.

I’ve been having a play with some simple erasure correction over networks. The result so far is a messy ruby script which will set up a tunnel with some erasure tolerance between linux boxes. EDIT: script updated. Was insanely buggy and kept losing link when client changed source port. Now repairs connection when that happens, editing this now over the link.. It breaks each packet it gets on the tun interface in to 3 UDP packets, and can reassemble any 2 of those in to the original. It takes just over half as much again traffic (adds some headers) but is a bit more resilient to random packet loss.

You can set up a server instance like:

you@server$ sudo ruby fectun.rb 0.0.0.0 4943 server PASSWORD

It will bind to the IP and port given and wait for a client to connect. The word “server” is magic – no decent command line options yet. The password should be chosen and kept the same between them – the server can only handle one client at once and any client with the password can take over the connection.

To connect with a client:

you@client$ sudo ruby fectun.rb 0.0.0.0 9000 my.other.computer 4943 PASSWORD

If all goes well, the server end should tell you it’s switching its destination to the client that just connected. The port may be mangled if the client is behind a NAT router.

Each end now has a “tun” interface connected to each other. These need to be given IP addresses:

you@server$ sudo ifconfig tun0 10.93.0.1/24

you@client$ sudo ifconfig tun0 10.93.0.2/24

Then from the client you should be able to:

you@client$ ping 10.93.0.1

or set up a SOCKS proxy by running SSH over the link:

you@client$ ssh you@10.0.0.1 -D 8080

It’s all a little sketchy at the moment but have managed to get a link to my server going from home. I hope to extend this to batch packets together within the erasure code so that it doesn’t send 3 times as many packets.

P.S. Please use this nicely. Could be quite anti-social on busy networks if you sent a lot of traffic over it.

Requires:

• Metal framing wire – a very thin wire.
• Tacks to attach wire to frame.
• Battery from 6 to 12 volts – lower generally better as gives you a bit more time and reduces risk of cutting foundation in to pieces. I used a 7.2V NiCD battery pack intended for RC cars.
• Clips to attach battery to a nail or wire. “Crocodile clips” would be perfect.
• Board or block that is about the size of a sheet of foundation. Must be possible to fit this through the frame, but must also be close to the full length of the foundation and at least as wide as the spacing between the top and bottom wires. Have used a pair of hive-width runners successfully.

1. Nail tacks half-way in where the wire will start and finish going through the holes of the frame.
2. Wire the frame with thin metal “framing wire”, wrapping around the tacks. Add tension by unwrapping from each end, pulling tight and wrapping around the tack again.
3. Once wire is tensioned, nail the tacks fully in. If possible don’t clip off the left-over wire at each end yet – having a tail there makes it easier to clip.
4. Place the foundation sheet in the groove in the frame. Leave the sheet on one side of the wires – don’t try to weave it through.
5. Place frame on the block so that the wires are on top and the foundation sheet is below. The foundation sheet should rest on the block, and the frame around it should be held up by the wires running over the foundation.
6. Attach one terminal of the battery with a clip to one of the tacks or the tail sticking out from it.
7. Attach the clip for the other terminal to the other tack or tail. The wire will heat up – be ready to unclip after a few seconds!
8. The weight of the frame pushes the wire in to the foundation. If you have any loose wire, may need to push down on it to get it to embed. At lower voltages you can touch the wire without burning yourself to help this along.
9. When the wax surrounds the wire (a few seconds), unclip the nearest clip from the frame. This generally creates a visible band of clearer wax where the foundation has been heated – can judge visually quite easily when to disconnect.

This allowed me to do my first 20 frames quite quickly. Had a few dodgy embeddings at first due to wires in the first frames I wired being too loose, but the majority came out with no exposed wire.

You can either make the changes in the same place and not be able to separate them down the track, or you can do the work in different branches and have to integrate them down the line. The more you split your work up, the more you have to merge and more likely it is that conflicts will happen. In one of the projects I work on, we’ve come up with a few interesting ways of keeping work separate and resolving these conflicts when they occur.

This project has a lot of concurrent work going on at any given point. These are, in Catalyst’s normal way, entered in to our work management system as “requests”. Each request might be for a piece of work which takes a couple of hours, a day or a week or more.

We use git for source control, but the techniques here apply to any VCS with usable branching and merging. To manage these, we use a branch for each request and work whenever possible to keep the changes for requests separate. By keeping changes apart, we allow requests to proceed through stages of development, testing and eventually release independently of each other, so that an issue with one request doesn’t hold others up.

The project has one branch which represents the code running on the “production” servers:

--o production

Whenever we start work on a request, it starts at the latest “production” code and changes are added to it:

--o production
   \
     ---o request 1234

The developer keeps all the changes for that ticket in its own branch. Come time to test, a bunch of requests are merged together to create a testing version:

request 9999 --------o
request 1111 -----o   \
request 1234 --o   \   \
                \   \   \
                 o---o---o---o testing

This is tested by our QA people, and the tickets are individually passed or failed. If we have tickets 1111, 1234 and 9999 then each ticket could be released, or not, at any given time. We can elect to include 9999 and 1234 in a build, but decide that 1111 isn’t stable enough yet.

Where this gets tricky is where two requests alter the same piece of code. Our attempt to merge this then generates a conflict. If you just resolve that conflict when it happens, you’d have to do it every time – the “resolved” combination isn’t reusable since the testing branch contains dozens of requests’ changes, any of which could be withdrawn at any time.

When a conflict happens with a ticket, our standard approach is to:

• Merge in the latest “production” code, ensure it doesn’t clash with that.
• Look for other branches modifying the files where the conflict is happening, find which branch or branches are clashing with it.
• Create an “integration branch” to tell the VCS how to integrate them.

The integration branch is yet another branch, the whole purpose of which is to tell the VCS how to resolve a conflict. If requests 1234 and 5678 are clashing, we would create a branch containing both of them:

request 5678 -----o
request 1234 --o   \
                \   \
                 o---X---o int_1234_5678

When we merge in the second one, we’ll get a conflict, which will have to be resolved by hand. Once we’ve done this, however, we can avoid further conflicts when merging branches. We merge in the integration branch first, then any subsequent changes on the individual requests:

request 5678  --------o
request 1234  -----o   \
int_1234_5678 --o   \   \
                 \   \   \
                  o---o---o---o testing

Because int_1234_5678 already contains the clashing changes from both, merging this first avoids the conflict reappearing. We still try to merge the individual requests in, because they may contain further changes since the integration branch was created.

This approach works pretty well for clashes between two branches, but it does get messy quickly if three branches change the same code at the same time. For that level of complexity, you would have to create 4 integration branches to allow any combination. At this point we tend to give up on this approach and combine changes irreversibly.

http://www.instructables.com/id/Making-a-Small-Knitting-Loom/

The only materials needed were some 16 gauge wire, a crochet hook and some wool. I picked up the wire and made a couple of these – one small one and a slightly larger one.

A bit wonky but they seem to work. One of the great things here is that they have a gap in them which allows them to snap around a post/pipe/cable without needing to slip them over an end. These can be used just with some wool and a crochet hook.

My first project with this was to decorate a hat stand at work, with the aim of producing something interesting and different and confusing colleagues – I wanted this just to appear overnight and obviously be impossible to slip on or off. I came back after work with the crochet hook and some multi-coloured wool and started shortly before 7pm. With a brief break to grab dinner, this fairly large first attempt took me until 12:30 to complete.

Apparently the goal of puzzling colleagues succeeded, but I was tired enough that I got in to work today at the last possible moment and missed all the reactions.

I definitely sped up as I went – wouldn’t take as long again, but working on something like that that solidly is one of the most tiring things I’ve done. I highly recommend trying this technique – maybe not in such a rush though!

Another approach to this is to create and use a “materialised path” – a record on each category which lists all of its ancestors. In Postgres, it seems that materialised paths are quite easy to use and lend themselves naturally to more conventional indexing and querying. First, let’s create an example data structure:

CREATE TABLE public.category (
  category_id SERIAL PRIMARY KEY,
  parent_id INTEGER NULL REFERENCES category(category_id),
  category_name TEXT NOT NULL,
  UNIQUE(parent_id, category_name)
);

This describes a tree of named categories. Each category contains a reference to its parent category. It also has a name, where no two sections under the same parent share a name. This can be visualised like a directory structure – under each category we have zero or more uniquely named sub-categories. There can be any number of “root” categories – designated by setting the parent_id field to NULL.

The problem we face is that doing queries on the whole tree is difficult because each entry only contains local context, not overall position in the tree. We have to iterate over large portions of the tree to perform many queries. A very good solution to this is to have a materialised path:

ALTER TABLE category ADD COLUMN path INTEGER[] NULL;

This will contain an array of category id values from the root down to the current one. For example, if we have category 1234 and its parent is 1000 and 1000’s parent is 500 and 500’s parent is 10 and 10 is a root, our path will be {10,500,1000,1234} . This structure allows us straight away to do queries over large portions of the tree. To get all descendants of category 3 we could run the query:

SELECT * FROM category WHERE 3 = ANY(path);

This is obviously convenient, but the path has to be kept up to date for this to work. The most reliable and efficient way to do so is by writing some functions in the database.

CREATE FUNCTION public.find_category_path(start_id INTEGER) RETURNS INTEGER[] AS $$
  DECLARE
    current_id INTEGER;
    result INTEGER[];
  BEGIN
    IF NOT EXISTS(SELECT 1 FROM public.category WHERE category_id = start_id) THEN
      -- No category exists with that ID. We will return NULL - could also raise error.
      RETURN NULL;
    END IF;
    
    current_id := start_id;
    WHILE current_id IS NOT NULL LOOP
      result = current_id || result;
      SELECT INTO current_id parent_id FROM public.category WHERE category_id = current_id;
      
      IF current_id = ANY(result) THEN
        -- We have a loop in parent IDs
        RETURN NULL;
      END IF;
    END LOOP;
    RETURN result;
  END;
$$ LANGUAGE 'plpgsql' RETURNS NULL ON NULL INPUT;

This will find the path for any given section, provided a valid ID is given and we don’t have a loop of categories all referring to each other as parents. This may sound like a strange thing to have, but I’ve seen it happening.

This is fine for new categories, but if we update the parent of an existing category then we also have to update its children, and their children, and so on. We need a method to find its children and do this efficiently:

-- This does most of the work but shouldn't be called directly.
-- Use the wrapper below to call this with the proper starting values.
CREATE FUNCTION
  public._update_category_path(target_id INTEGER, path_to INTEGER[], set_path INTEGER[])
RETURNS VOID AS $$
  DECLARE
  BEGIN
    UPDATE public.category SET path = set_path WHERE category_id = target_id;
    IF set_path IS NULL THEN
      -- This means that we have a loop in parent IDs
      -- We have to set all children to also have a NULL path
      PERFORM public._update_category_path(category_id, path_to || category_id, NULL)
        FROM public.category WHERE parent_id = target_id AND NOT category_id = ANY(path_to);
    ELSE
      -- Update our children
      PERFORM public._update_category_path(category_id, path_to || category_id, set_path || category_id)
        FROM public.category WHERE parent_id = target_id AND NOT category_id = ANY(path_to);
    END IF;
  END;
$$ LANGUAGE 'plpgsql' CALLED ON NULL INPUT;

-- This is a wrapper around the above function, starting it off with the right path.
CREATE FUNCTION public.update_category_path(INTEGER) RETURNS VOID AS $$
  SELECT
    _update_category_path(
      $1,
      public.find_category_path($1),
      public.find_category_path($1)
    )
  ;
$$ LANGUAGE 'sql';

This floods downwards, updating each child based on its parent’s new path. The separate parameters for path to it and value to set are to cover the possibility of having a loop in the parent references, in which case we set all their paths to NULL – they don’t have a meaningful path.

Finally, we get these to be run automatically when a category is created or updated:

CREATE FUNCTION public.category_path_trigger() RETURNS TRIGGER AS $$
  DECLARE
  BEGIN
    IF TG_OP = 'INSERT' THEN
      PERFORM public.update_category_path(NEW.category_id);
    ELSIF (NEW.parent_id IS NULL AND OLD.parent_id IS NOT NULL)
      OR (OLD.parent_id IS NULL AND NEW.parent_id IS NOT NULL)
      OR (NEW.parent_id != OLD.parent_id)
    THEN
      PERFORM public.update_category_path(NEW.category_id);
    END IF;
    RETURN NEW;
  END;
$$ LANGUAGE 'plpgsql';

CREATE TRIGGER maintain_category_path
  AFTER INSERT OR UPDATE ON public.category
  FOR EACH ROW EXECUTE PROCEDURE public.category_path_trigger();

Finally we update any existing categories:

SELECT public.update_category_path(category_id)
  FROM category
  WHERE parent_id IS NULL;

So now we have a materialised path – what can we do with it? One of the best features around this in PostgreSQL is that we can build an index on the path!

CREATE INDEX category_path_index ON public.category(path);

Not only do we now have a list of ancestors on each category, but we can now do some really efficient queries.

We can display the category tree:

SELECT repeat('  ', array_upper(path, 1) - 1) || category_name, category_id
  FROM category
  ORDER BY path;

To get all descendant categories:

-- Add one to the last element of an array
CREATE FUNCTION public.increment_last_array_component(INTEGER[]) RETURNS INTEGER[] AS
  'SELECT $1[1:(array_upper($1, 1) - 1)] || ($1[array_upper($1, 1)] + 1)' 
LANGUAGE 'sql' IMMUTABLE RETURNS NULL ON NULL INPUT;

SELECT sub.* FROM category sub, category top
  WHERE sub.path > top.path
  AND sub.path < increment_last_array_component(top.path)
  AND top.category_id = ?;

This is equivalent to the descendants query from earlier, but is capable of using the index. This works because if our category has the path {1, 2, 3} then everything under that, such as {1, 2, 3, 12} will come before {1, 2, 4}. With a helper function to add one to the last element of the array, we can turn this in to a range query which the index can accelerate. All of Postgres’ index functionality works here – getting part of the tree in order can be done with an index scan.

Materialised paths in PostgreSQL are both easy to create and well supported by an index on the field. As well as the queries here you could:

• Create a second path using the names. This would allow you to quickly look up categories based on a path or URL (depending on application).
• Allow for easier collection of aggregate statistics – find all descendant categories and add/average information from them.
• Convert complicated application-layer tree traversal in to single queries.
• Allow triggers and routines altering children to also easily update their ancestors.

P.S. If you use this technique, please let me know by posting a comment. It’d be great to know the different uses people find for this.

¹ http://dev.mysql.com/tech-resources/articles/hierarchical-data.html

UPDATE: There was an error in the trigger when the parent_id used to be or was becoming NULL (comparison issues). Fixed now.

Setting up the connection at home was interesting – I’m using OpenWRT on the router (highly recommended!) and set up tunneling using 6to4, one of the transition mechanisms in place to help people get IPv6 enabled. OpenWRT requires a few packages for IPv6 networking:

opkg install kmod-ipv6 kmod-ip6tables ip6tables kmod-sit kmod-iptunnel6

Telstraclear, who I connect with, run a local 6to4 relay. 6to4 sends IPv6 traffic encapsulated inside IPv4 packets to the closest 6to4 relay. If you have a good and well connected 6to4 relay nearby, this is a really good option. In my case, with a router on my ISP’s network, this seems to be the best option.

My router’s script to set up the tunnel, based on a sample from the OpenWRT wiki, is available here . One neat thing is that with 6to4, you get allocated a whole /48 of address space. In other words, your local network could support 1,208,925,819,614,629,174,706,176 public addresses². This may seem a little wasteful, but it allows all sorts of cool things to happen, and IPv6 has space to spare.

Once you have IPv6 on the router, the next task is to extend it to the network. On Linux routers you can do this by installing and running radvd. Under a recent OpenWRT this is packaged and easy to install. This sends out advertisements to the LAN, telling them the IP range used by the network. If the IP range is the right size – a 64 bit prefix – then the clients on the network will assign themselves an address within it based on their MAC address. This is the “stateless autoconfiguration” method and is pretty cool – much better than DHCP. To make the network larger, just repeat the router advertisements – you’ll never have collisions.

Unfortunately it seems the Telstraclear 6to4 server isn’t quite as well connected as I would hope – I’ve had trouble connecting to a few sites. One of the issues with setting up an IPv6 network is that (I’m guessing) about 5% of sites which advertise IPv6 support are actually inaccessible, either because the site is misconfigured or because the routes are much sparser than for IPv4. Getting to servers at my office is really ridiculous – it goes all the way to Los Angeles and back and some of the links seem unreliable. Since it doesn’t fall back to IPv4 properly, these become hard to get at. This hasn’t proven too problematic so far, but I’ll be keeping an eye on it.

One of the results of setting up IPv6 in this way is that suddenly all the computers on the network have a global address. This is actually an interesting consequence, as a lot of computers probably shouldn’t be directly connected to the net without a firewall. My flatmates in particular run Windows boxes and have been known to not have an antivirus installed, yet alone understand firewalls and services. I didn’t want to expose them, so actually blocked access from outside unless you’re on a whitelist of addresses. To make it easier to configure this, I set up a ruby script³ which runs as a CGI script on the router and allows any computer on the network to add itself to this whitelist. It also allows a computer to reserve some IPv4 ports to forward, for example to run peer-to-peer software on.

My server required a completely different approach. The 6to4 server near it is really bad and sometimes can’t be reached at all. As a result, I had to use a tunnel to the Hurricane Electric tunnel broker . They provide free tunnels to their routers and, being a major IPv6 backbone operator, are very well connected. This works in a really similar way to 6to4, but rather than tunneling to the closest 6to4 server you tunnel to one of their routers, and rather than using a /48 of address space derived from your IP you use a /64 allocated by the tunnel broker.

An interesting side benefit here was that Hurricane Electric also provide a “certification” system with a series of tests. Each test involves actual practical set-up and actually tests that you can set up both IPv6 clients and servers. It does this by making you prove that you control a site, then testing that it supports particular features. You can follow through them as you set up a server with IPv6 and they’ll act as a pretty good guide of the steps to get fully IPv6 compatible. The “badges” are a bit naff and the instructions could use a brush up, but it helps to answer the “what next?” question.

One thing I found odd was that I needed to make the server respond to the addresses, but you can’t add them to the tunnel itself. I ended up setting up a bridge as a dummy device and adding the addresses to that. This made the server accept connections to those addresses rather than sending back the “no route” response. It feels really nasty and dirty – there’s a bit of that still involved I’m afraid.

Once you have IPv6 set up on a server and some services configured to bind to IPv6 addresses the next thing is to get DNS going. Whereas IPv4 addresses of servers are in the “A” record, you need to add an “AAAA” record with the IPv6 address. Be sure to test that you can access the machine using IPv6 from outside first – if you add the DNS entry but the IPv6 connection fails, clients won’t fall back to IPv4. If you can get reverse DNS delegation (Hurricane Electric will do this on their tunnels) then setting up reverse DNS is also a good idea and isn’t any harder than forward DNS. Just make sure you get the reverse DNS name right – the easiest way is to dig -x on your IP and take it from the output.

The conclusion of all this is that it took a while, but I now have IPv6 at the flat and on my server. For those who do run web sites, I highly encourage doing all this early, before you have to for work. It’ll take a while and isn’t easy (I’ve brushed over the hours of wandering around dead ends) but in doing so you’ll be more prepared for the imminent rise in use. I’m sure they’ll find some way to slow the address space exhaustion, but even so people will be forced to update within half a decade.

¹ The main site, my grandmother’s site, mail server, name servers.

² OK, some of those are reserved.

³ The main script and an included file, firewall setup for OpenWRT, and firewall updater for the reserved ports.

This is only really a test of multiple indistinguishable elements, which is harder to work with than it would appear. It doesn’t store close to the actual capacity – it only lets you use the amount of information that it could encode with worst-case selection. The result of this is that encoding with two decks at once only gives us as much information as encoding with 2 decks individually. It also probably leaks information about the starting order and is likely to be distinguishable from a random shuffling since there will be some amount of unusable capacity.

Still, this does have the advantage of being able to scale it up, or work with orderings of small numbers of elements. I’ll be thinking about whether this can be done more efficiently.

Script: permute_multi.rb
Series file with 2 decks: cards_2decks.series

The reason you might want to do this is that a deck of cards appears entirely innocent and is expected to be in some unpredictable order. The message is also quite easy to destroy – shuffling a deck can be done quickly and doesn’t look as suspicious as burning a slip of paper. The question was how much information can you store in the ordering of a deck of cards and how you can get it in and out.

It turns out that you can actually fit a small message in the order of a deck, though it would be extremely hard to design a means of encoding and decoding by hand. You would be limited in this case to a well defined series of signals – e.g. the 3 of spades as the 8th card could signal that the secret police were on to you.

With a computer on either end, we can achieve close to the theoretical information capacity of the ordering – enough to hold 28 bytes of data in a standard deck of cards. I’ve written a script that can encode a series of bytes in to the ordering of a series of elements and decode on the other end. It’s not limited to cards, but both ends need to know the series of elements they’re re-ordering.

My script is available here . The script should run on any system with a Ruby interpreter. The examples given below are for use on *nix systems with the bash shell – may need some changes to the commands to run on a different system.

You start off with a series file. This should contain a number of unique entries, one per line. For example, each line could represent a card from the deck. The minimum length is 6 entries – enough to encode a single byte. Both ends will need the same series file to encode and decode. An example file with cards is available as cards.series . I’ve used “C” for clubs, “H” for hearts, “D” for diamonds, and “S” for spades, followed by the card within that suit.

To see how much information can be stored in the ordering of those elements, you can ask it for the capacity:

ruby permute.rb -c -s cards.series

In the case of the cards in a deck, we can store 28 bytes. If both jokers were added, we could store 29.

To use it you type something like:

ruby permute.rb -e -s cards.series <<< “My message” > encoded

Then arrange the cards in the order in the file “encoded”. Put the deck in a box, pass it along to the recipient, and get the person on the other side to enter the entries in to a file in that order. They must exactly reproduce the original “encoded” order and create a file with one entry per line identical to the original encoded file. They can then decode:

ruby permute.rb -d -s cards.series < encoded

The original message should be reproduced, padded with null bytes to the capacity of the series. At the moment there isn’t a way of storing the length – people can add their own convention if they’re happy to sacrifice the storage space.

Questions to be pursued from here:

• If we use a randomly shuffled deck as the order of the shared base series, how well does this protect the content of the message?
• If we encrypt the data so that the information appears random before encoding, does this process leave any evidence that the deck contains a message? In other words, can you demonstrate that the deck is not just in a random order? Any unused part-byte of capacity will be fixed at the moment – should I randomise those bits?