I hate make

Actually, I hate make and shell (all forms) and sed and basically, the *nix command structure.

The main reason: escapes. I just can’t seem to ever get them right.  In Makefiles they are a nightmare!

Just try issuing a sed -e command after a dependency — AFAICT§ there are three levels of escape to consider: 

1. the make level (where $ is special, and you double it to get a real one), 
2. the shell level (where $ is special outside of single quotes) and 
3. the sed pattern level, where $ is special and you “escape” it with a backslash to make it not special again.

Here is the line (almost) I had to get right:

sed -e 's:^ABC=$:ABC=${VAR}:' file >file.tmp

Now, execute this on the command line (bash shell) and it nearly works — it should replace a complete line “ABC=” by the complete line “ABC=${VAR}” in the file called file.

It doesn’t actually work because of the second $.  I have used (single) quotes so that the shell doesn’t try to substitute the (apparent) shell variable ${VAR} but I forgot that $ means ‘end-of-line’ to sed (even though I used it to mean that in the first part of the substitution string). So actually, it ought to be:

sed -e 's:^ABC=$:ABC=\${VAR}:' file >file.tmp

using a backslash escape to tell sed to treat the second $ literally.

So this second version goes in the Makefile, and of course it fails.  (And, it takes me some time to find this out, since the make is executed in the heart of some long-winded build process, and if I don’t execute it there it will not have the context to make it work — bórza!)

The reason is that single quotes don’t protect expressions in make, so it tries to substitute the ${VAR}.  “Aha,” I cried, “I’ll have to escape the backslash and the $ for make.”  Like this:

sed -e 's:^ABC=$:ABC=\\$${VAR}:' file >file.tmp

But I was wrong, the backslash isn’t sufficiently special in make — well it is, but double backslash is not “escaped” to backslash: instead it leaves it as double backslash, and the single quotes still offer no protection.  So we end up with “ABC=\${VAR}” in the output.

Round I go again:

sed -e 's:^ABC=$:ABC=\$${VAR}:' file >file.tmp

and this works.  Finally.

Why all this fuss?

Because *nix systems and their utilities have internalized all this escapery, inherited from an old-fashioned and stupid macro-language-style command structure, and we are now stuck with it.  Everywhere.  And of course, *nix being the democratic organisation it is, everyone does it differently, with different rules and with different exceptions.  Ugh.

So... I think we should start to unravel it.  We could start with make, for example. Let’s create a version of make where all the expressions are evaluated with referential transparency, which is to say that a literal string, once established, is never rescanned and re-interpreted again, even if I pass it around in a variable and it appears in another expression.  Substitutions are only made once if I say they should happen.  If I write an expression with a $ in it, it cannot be interpreted as a variable indicator anywhere else.

Then, the utilities that we call from make should have an ‘uninterpreted’ interface. That is, it should be possible to pass them a literal string without worrying about the contents of the string.  If we are a utility passing strings we haven’t checked, we no longer have to worry about what they might contain and scan them for things to escape.  That way, we also don’t have to worry about what the utility is that we are calling, nor where it might send this string.

I guess we would have to tackle the shell, too.

There would be consequences: some of our tricks (for example: escaping the right number of times so that the cascade of string passings remove the escapes and trigger interpretation at just the right level; or, varying the name of a reference using the value of another) would be harder to achieve — but let’s face it, who needs these really? And when you do, how long does it take to get it right?  And have you ever got it (really) right?

Yup, a proper referentially transparent shell would be a first step.  Or maybe a grep....

hmm...

§ I also hate all these cheesy acronyms — it’s just a linguistic barrier to ensure the prols can’t easily join the club.

PS: Did you notice that the ‘footnote’ didn’t link properly? I don’t know how to do this in this form — how do I link to an anchor in the same page?

Spec thought for the day

Browsing around StackOverflow I came across a question about specifying binary files. One of the answers said that the Java spec for class files was a good example of how to do it.

So I went there to look and found this:

  ClassFile {
   …
   u2 constant_pool_count;
   cp_info constant_pool[constant_pool_count-1];
   …
  }

and the following description:

constant_pool_count

The value of the constant_pool_count item is equal to the number of entries in the constant_pool table plus one. A constant_pool index is considered valid if it is greater than zero and less than constant_pool_count, with the exception for constants of type long and double noted in §4.4.5.

constant_pool[]

The constant_pool is a table of structures (§4.4) representing various string constants, class and interface names, field names, and other constants that are referred to within the ClassFile structure and its substructures. The format of each constant_pool table entry is indicated by its first "tag" byte. The constant_pool table is indexed from 1 to constant_pool_count-1.

and I wondered if this is a good spec what does a horrible one look like?

Granted, this is quite chatty, but when you read it for sense you get more and more confused.

It looks like a C structure, so let's read it like one… 

Why is the count one more than the number of elements in the array?  The array index starts at 0 when I declare one.  Oh yeah: the valid indexes are greater than zero (why omit the zero index?)—except for some long and double stuff, life's too short—and then the valid indexes are from 1 to count-1.

Whoa! where do they go? The declaration a[3] normally allocates three elements (indexes 0, 1 and 2), but here we allocate count-1 and index from 1 to count-1.  The last index would be invalid!??!

Aha!  We must be basing our arrays from one and not zero!! 

…and the count entry isn't a count really.

Ugh!

If this is a good spec my name is Edsgar Dijkstra.

Install updates?

It's a few weeks now since I last posted about my new job.

Spring (in my step) source

    One of the nicest things that ever happened to me, being in a new (smaller and friendlier) company has boosted my energy, optimism and self-esteem. That, and free membership to the local gym, has made me feel younger! It is simply amazing what a challenge and a change (with, it has to be said, a wise and people-savvy company) can make to a person. [I still look the same age, though--you can't have everything.]

A few weeks ago (about six, now) we adopted Scrum working. (See picture below.)

This, in case you didn't know, is an iterative development structure that encourages short, development spurts, with clearly defined and effort-sized deliverable outcomes. These spurts are called Sprints and in our case are three weeks (elapsed time). [Of course, in my old place we couldn't have just `adopted' Scrum. This would have been hard to do without several levels of sign-off: and the running and scheduling of Sprint planning meetings would be very difficult indeed.]

Progress

    We just completed our second (full) sprint. (That's how I knew it was six weeks -- see?) I have to say that, normally cynical about these much-lauded `methodologies' (and having experienced--and even taught--a fair few in my time), I have found this one to be quite positive.

    Since the internal workings of a sprint are really up to the team (and they are relatively free to do as they please) we have been guided (by the team leader) rather than driven, and are still feeling our way. However, the nature of the beast is that teams are expected to learn for themselves what is good and bad about the process -- including how to estimate the team's tasks beforehand. 

    For us, that means we are still tinkering with how we deal with change: most of these are the usual--design changes, external dependency lapses, unexpected and persistent bugs, unplanned absence; and the answers are the usual ones--postpone design disruption (until the end of the sprint), re-order tasks to allow more external time (and plan tasks to negotiate with third parties), rally the `troops' to crack hard bugs once and for all, pull together to keep the sprint plan on target.

    In the case of a short spurt (the sprint), and well-defined and agreed tasks for that spurt, we find it is easier to apply those standard solutions. In every case the small tasks and sprint structure makes it easier:

• postponing tasks is easier since there are enough small tasks planned to rearrange and even re-assign;
  
• re-ordering is easy, even if there are inter-dependencies, since the short sprint meant that not too many were scheduled at once, and the dependencies are manageable;
• rallying the troops is not hard, because only a week or so ago all the team members took joint responsibility for the deliverables--there is no shortage of offers of help.

After a number of these, too, the members of the team have worked closely and regularly and this makes joint responsibility easier to execute: we learn each others' skills. New guys learn more quickly and have a sense of achievement earlier. They become old guys sooner (and I mean that in the nicest possible way).

Distress

It is prudent of me to talk about the not-so-good things, or things for which the jury is still out:

• the expectations of the product owner can be skewed, and not allow enough team experience before expecting (or trusting) tight estimates -- hopefully this is something that can be ironed out over time;
  
• the team members have to work at close co-operation during the sprint (we are lucky to be all in the same room--although this could be a problem for a larger team);
• learning what `hours worked', `task complete', `ideal hours' mean requires several sprints to iron out--and this can easily be disrupted by team personnel changes;
• it is hard to do difficult things--especially if they require sustained, co-ordinated effort.

There are other niggles, but I have to be careful not to confuse genuine concerns with my natural tendency to be a Grumpy Old Man.

    More on the progress of this `experiment' later.

Meanwhile...

    Back at the (old) ranch I hear all is not happy: the large management structure, lack of technical autonomy (caused by lack of trust), and catastrophic disconnect between effort, achievement and reward, means that more and more (oftener and oftener?) I am hearing heart-rending bleats of discontent from my old friends. What they can do about it, I don't know, but I wish better for them.