Riding the Tiger

Amos left at the start of my second year at Tigerland and Meggs took over his role, as well as continuing with his own duties. The first thing that he needed to do was to replace Amos.

Meggs decided that he needed someone pretty serious to replace his compadre; somebody both skilled and experienced. And so he set about interviewing candidates for the role of a senior developer.

I was not present for any of the interviews, but I heard about what had happened from the technical director and from Meggs himself. Meggs reduced at least one candidate to tears during the interview process. For other candidates, if they were not doing well enough in the testing, he would get up and leave the room without saying a word.

Eventually, Meggs settled on a developer named Ivan. He didn't particularly like Ivan, but he was the only candidate who had actually passed Meggs' testing and soon enough Ivan joined us in the fishbowl. Ivan was very pleased to be in the role, and he expressed to me his delight about how much he was learning from Meggs, about how great a pleasure it was to be exposed to an intellect like that. No matter how badly Meggs treated him, Ivan seemed to be okay with it. I have never seen anything like it before or since.

Every day, for a solid half an hour just prior to lunchtime, Meggs would stand over Ivan and bellow at him;  insulting his work, his integrity, his experience and his intelligence. I remember Ivan's work being clean and tight and clear. He met all of Meggs' style guidelines and everything I ever saw of it looked good. Ivan was the only candidate who passed Meggs' screening; I'm pretty sure the problem was not actually in his court. Meggs liked to be feared, and without his friend Amos to snicker about it with he was probably lonely as well.

I started having my own problems with Meggs. His desk was abotu a meter and a half from my own and he started watching me over my shoulder. Meggs decided that I was compiling the source code a bit too often, which was costing me a valuable half minute or two each time I pressed F9. "Jared. You only need to recompile the module you're working in. Be more productive." Meggs' own productivity was tailing off, because, for one thing, he was spending so much time watching me when he wasn't yelling at Ivan.

Meggs assigned me a fairly complicated piece of logic to do involving persistent locking. My first attempt was a failure; I hadn't understood the idea of it properly and it didn't work when there was more than one client connected (which defeated the purpose of the exercise). Barely controlling his impatience, Meggs explained it in more detail, and even drew  me a picture on a scrap of paper and told me to build it. This was the first time I'd ever been shown a design drawing for anything at Tigerland... and, let me be honest, through the rest of my career, I've rarely seen a design drawing for anything that I didn't author myself. I any case, it made an impression on me.

This time I understood what was going on properly. I looked at the design carefully and I noticed what looked like a bad side effect. "No, that's the behaviour I want," said Meggs.
"Well, okay, then," I said.

I went and set up the locking the way he had asked. When the task was done he code reviewed it and I checked it in... and then I threw away the drawing.

About two weeks later, I found myself on the receiving end of a tongue-lashing on the same scale as the ones Ivan was subjected to. "Pikeman! This locking system you built is rubbish! Have you seen this side effect?"
"Meggs, I told you that was going to happen."
"I would never have agreed to that!"
"But... it was your design! I built it exactly the way you asked me to! You even drew me a picture!"
"Where's this picture, then?"
I've never thrown away a design drawing since.

By now Meggs had stopped contributing any code to the product. Ivan was building a special new feature, but it was Guthrie and I doing 80% of the work fixing bugs and migrating across to the new design. But it wasn't just the architecture changing. Meggs was making ad hoc changes to the feature set and functionality as  we went.

Meanwhile, I was becoming more and more interested in the design of the new infrastructure. I suggested to Meggs that it would be really nice if you could tell the framework what the cardinal relationship between different kinds of entities was and have it generate queries based on that. Meggs agreed that it was a good idea and he added some code that did indeed identify the relationships... but it was cosmetic only. None of the smarts I wanted to build were present. In hindsight, I think this actually half-assed attempt at an apology. I didn't criticize it but he could see I was disappointed.

The technical director could see that things were going badly. We'd barely enough money to cover costs during the year and the software was badly behind schedule. This latter problem was entirely due to the fact that Meggs kept moving the goalposts; making the design more and more elaborate and forcing us to constantly rework what we had done. He hadn't stopped coding himself, but he was coding a hobbyhorse project that had nothing to do with the work we were putting to market. Aware that something was amiss, the technical director instituted weekly development meetings.

These only made it worse. Meggs began to use the forums to taking credit for other people's ideas. Most usually, those ideas were mine. Sometimes Meggs would get the ideas wrong and miss the point... and if I would pipe up to correct him he would repeat what I said back to me, in front of the whole room, as if I was too dense to understand them.

Suddenly I was no longer afraid of Meggs anymore. I had learned a lot from him, but I knew that he was no longer this infallible font of technical wisdom. He had taught me that the design was the most important part of an engineer's job and I was going to stand by that principal, regardless of his sensitive ego.

Meggs' new design ideas were all to do with preventing programmers from making mistakes where his framework became unintuitive or failed to handle errors well, and I was assigned the task of making the little classes that would enforce this... classes that soon be scattered through the codebase like a bucket full or rabbit pellets. I didn't complain, but I became sly about it, and I started making these turd classes more and more elaborate; decorating them with superfluous template parameters and inheritance hierarchies. These exquisitely-polished turds passed through Meggs' formerly-rigorous code reviews without comment. I turned one of these projects (a particularly bad mixed-metaphor turd class) into a three-week-long project which to this day is one of the most conceptually-complicated and difficult pieces of technology I've ever worked on.

Meggs was even less attentive than ever, and Ivan was copping worse and worse abuse every day. In a bid to get me out of his hair, Meggs assigned me to port one of the biggest modules in the product across to his new framework, with a number or revisions to the functionality that he laid out in an email.

It was a disaster. If we'd ported the module as it was that would all have been fine, but the functionality changes were going to take a module that was compact and well featured and useful and turn it into something no customer would ever have a use for. Usually I wouldn't have known or cared about the customer's end of the experience, but we used that module internally to manage our development database, so, unlike some other areas of the application, I knew it inside out. The development team used that module on a daily basis in the same way that our customers did.

I told Meggs it was a bad idea. We argued loudly about it for a full day, but I refused to give ground. We left it over the weekend and took it to the development meeting the following monday morning.

Words got heated, but there was no yelling. Eventually the technical director threw up his hands and said "Pike, just do it the way Meggs wants it."

I went back to my desk and went to it like a man possessed. I was determined to get the work done as quickly as possible so that Meggs could see what a disaster it was before he forgot the arguments and blamed me for it. I didn't speak a word, I don't think I took a lunch break. I don't think I have ever produced as much code as I did in the following forty eight hours.

Once I had written most of the code and fallen once more into the sleepy cycles that followed which involved running the regression test suite and writing new tests... but mostly staring at the screen while the tests executed... once the anger started to fade... I realized that it was a futile exercise.

I have a vivid memory of going to a camera store with my mother when I was a boy. There had been some problem in developing a roll of holiday snaps and my mother was furious. The man in the camera store was an old Indian man. He stayed calm and my mother settled down. Once they had reached a peaceable agreement, he told her that he was glad. "You came in here riding the tiger," he said. "But don't you feel better now that you have climbed off?"

That was what I had to do. It was time to climb down from the tiger.

Ars Moriendi

A couple of days before the end of my vacation I received an email from Andy, the Project Manager at Fjord Systems.

"Pike, I hate to ruin your holiday, but everything's gone wrong. We're in deep shit."

Andy wouldn't tell me anymore. I arrived back in Sweden a couple of days later expecting to find the common area festooned with intestines and severed heads in the fridge. It sounded to me as if the project had died hard, but what I really found was... nothing.

The team had told me that they were good to go when I had left for my vacation, three weeks prior. They had all agreed to do the work. They knew who was supposed to be doing it, and how. That was what they told me.

Apparently, once I had left the office, instead of buckling in to do the work they had panicked. As best I could tell, the team had spent three weeks running in circles around the office, screaming and throwing their own faeces at each other. As far as the art of dying went, these kids were a bunch of amateurs.

Anders had finally asserted himself a little bit. There wasn't enough time to undertake the full scope of the project I had laid out, so they would be cleaning up the old core engine (the part that he'd told me made him feel sick to his stomach) and reusing it more-or-less as it was. The so-called 'core engine' was the piece of the product that actually did the majority of the work of the app, but it was not necessarily a complicated piece of technology. It sprawled a bit, but I did not believe it would have been difficult to rewrite it, or simply to break into chunks in preparation for a rewrite... as per the designs we'd agreed upon. Nonetheless, the team lead had finally made a decision and I decided to go with it. He agreed that the core engine would be integrated as per my designs so that it could later be unplugged, piece-by-piece, when we had finished this initial refit. Fixing foundations and the plumbing was the main thing.

I left the office with a bad taste in my mouth. Despite the fact that everybody had agreed to stick to the design, I'd heard them muttering. They didn't believe we could make it work. They didn't believe that we could make the deadlines. I was frustrated. I'd done this all before, with less people, and I knew it would work if they'd stop bitching and start coding. But I couldn't say that to them; that was Anders' job.

I went home and I tried to get to work on the driver integration with Chucky, but it wasn't that easy. My working hours in Australia would end right when the team's would begin, and I would finish a full day's coding with hours of emails, realtime chats and conference calls with the team. Every morning I would get up, sync down the source code, and find that the team had broken what I was working on the day before. Half of the productive part of my day would them be lost fixing it. I don't know what the team was actually doing, but it was clearly not working, since the project wouldn't even compile after their checkins.

I asked them nicely not to break the build. Weeks went by and I still found myself having to fix it on a daily basis. I tried a sterner approach, but this didn't help matters. I didn't yell down the phone, but once I abandoned politeness, the build breaks became less frequent... and meetings became commensurately more surly. I could hear them muttering at me in Swedish.

Joseph was granted a patent that he had filed. Nobody seamed to care. "Software patents don't mean anything," I was told.

Srinith and Sven complained to me that nobody else was working. Anders and the senior guys sat around in their office with the door closed. Tyko was wrapped up in his research. Nobody knew what the UI team was doing. I told that they were the A-team. They were the most productive guys and they were on board with my desire for progress. If we hasd to do this with a team of three, plus Chuck,y then so be it--I'd done it before, back at ATB Software, and this time I didn't have interference from management and marketing to contend with. I had to trust someone, so I trusted them. I asked them to look after the integration with the core engine and the UI team while I pulled my head in and finished the driver interface, and they agreed. This was a mistake.

My A-team introduced some new code libraries to instrument the source in order to catch memory leaks. This library was incompatible with my test apps, but it was cancerous: once it was in the application it proliferated everywhere and could not be removed. I complained: the new framework did not leak memory and the overhead of the instrumentation was huge. It threw lots of false positives and it made debugging a nightmare. Only the core engine needed to be instrumented, and there needed to be a way to turn the whole thing off. But they were adamant, and I had bigger fish to fry. I wrote new test apps and got on with it.

My A-team took the very simple structure that I had created to be a unit of currency between all of the subsystems and wrapped it in layers and layers of macros, so that they would act in an object-oriented way without being truly object-oriented... or debuggable.

My A-team changed all of the error handling routines so that, instead of passing error information up, they would throw assertions that would crash the application.

At this point I decided that I needed to rein them in, and I think they simply decided that the would instead replace me. They started to pull apart the infrastucture I had written, stripping out the inheritance hierarchy that would permit us to drop in new versions of the core engine. If they needed to make a small change to a class that I had written they would push the entire code module into an 'attic' repository and then create a new one... containing almost exactly the same code, but minus the history of changes and of course with his name on it as the original author.

I just took it. The design was all that mattered. If we could stick to the design for this release I would be able to steer the team back on course after the release date, I thought. I was up to my eyeballs in the driver interface and I didn't want to fight about it. But I knew that they were gradually stripping away all of my design and replacing it with Special Magic.

I start to get bug reports from the communications layer. I'd forgotten about the comms layer, with all of the other nonsense, but I didn't think it was a major issue. I couldn't duplicate the bugs, so I asked for log files. No matter how often I asked, nobody would furnish them to me... but the reports kept coming.

I did get a number of feature requests. "This comms layer is useless," Srinith would say. "It doesn't do X."
"It was designed so that could be built on top of it. It's easy enough."
"It should be handled inside the comms layer."

Like an idiot, I would then extend the comms layer to handle that behaviour, rather than insisting that the behaviour should be handled externally. Instead of thanking me, Srinith would then say "This comms layer is useless. It doesn't handle Y."

I did that three times, until Srinith ran out of new features. The comms layer was still buggy, and still a source of complaint, but by then Chucky had finished the driver and I had finished integrating it. Once I was able to focus my full attention on it, I was able to duplicate the underlying bug in the comms layer... and it was a serious one. Srinith jumped on it.

"Alright," I told the Andy, Anders and the A-Team. "I found the bug and I can fix it, but I need two weeks. Ten business days."

I showed them a design document, and they agreed to it. I got on it, and made good progress. Srinith threw some new feature requests at the comms layer for me, which he 'needed urgently', so I build those as well: twice each, once for the old comms layer (so that Srinith could continue programming with it), and once for the new one. On the ninth day I had finished all of the features and I was chasing the last remaining bug when Andy called me up. "We're pulling the plug on your comms layer. Srinith is going to write a new one."

I was on schedule. I would have the layer ready for them, bug free, on the day I had promised. It had the same interface as the old one; nobody would even notice when the new library went in, except that the bugs would disappear.

"Sorry. Anders told him to go ahead." In other words, Anders needed me to fail.

"What's he going to build? What's the design? Is there a document?"

"He's just going to do whatever."

I could see the death of my baby looming.

Joseph was laid off. Martin sent me a note to assure me that Chucky and this had nothing to do with Chucky and I, it was just that they didn't think Joseph's research had a place at the company.

A week later, Chucky quit. The drivers were done and he'd been offered something that paid better and gave him health coverage. Also, he said, he was tired of all the sniveling. I decided that I was, too. I was required to give three months notice, so I held out a few more days before I handed in my notice. My last day would coincide with the release date.

I wanted to make a graceful exit.

Brazil

After Vegas I returned to Australia with not much to work on. I proofread some of Jacob's patent apps, I went through his new matrix idea, I started building some of the stuff that we'd talked about back at ATB Software... but I didn't have anything formal to work on. I was still waiting for network access, but the company more or less shut down in the first week of December and most of my queries went unanswered. Occasionally I'd get a chirpy reply from someone in the IT department saying: "Done! There you go!" but no passwords or network details were forthcoming.

New Year passed, and so did the first couple of week of January. A full month since I had returned. Very slowly the logistics guy, Wils, organized me some flights out to Sweden to meet the team, to coincide with Jacob's next visit. The first itinerary he sent me was mental: the numbers on the flight times didn't add up and he had me routed through Atlanta. Once I got back in I had to look at it twice to discover that the flights he had booked originated in Melbourne, Florida.

Eventually, after I started to get a bit shirty about my network access, somebody in IT realized what had happened. They had indeed set me up with VPN access to the network and an internal email account, and then they sent all the information about it to that internal email. Several times. Once they put the information where I could find it I found that I could log into the network, no problem, but I had no access to anything beyond the intranet newsletter.

I flew to Sweden. Wils met me at the airport and took me back to the apartment that Jacob and I would be sharing  for my stay. I showered off my 35 hours of airports and aeroplanes and went straight to work. I was in the office by about lunchtime.

I was given the seat vacated by Åke, sharing an office with Tyko. Tyko was no problem; I knew him already, and I discovered that I had a greater tolerance for the techno music that pumped out of his computer all day long than I had expected. Åke, however, was another story, even though he was gone.

Åke's desk told a lot about the man and his work. It was filthy; covered with derelict CRT monitors, cables, mouldy coffee mugs, wastepaper, dust, and a layer of grease that I still cannot explain. I spent ninety minutes cleaning before I was satisfied that I wouldn't contract tetanus from sitting there. Åke's computer was on and unlocked, so I turned my attention to that next. He had left a thousand windows open, each one of them showing a gorgon's nest of ugly code. I carefully saved the contents of each window and closed them until I could again see the desktop of the PC. Then I shut the machine down, wrapped it in plastic, and buried it in a 12 foot hole. If we ever found a sudden need for whatever Åke had been working, on we were beyond hope.

(Alright, I lied about the hole. I put the machine in a corner of the room in case I might one day need something to accidentally-on-purpose put my foot through.)

I met the team. They seemed like a nice, if reserved bunch of guys. No overt social problems, although a couple of them were definitely capital-P Programmers. The guys Tyko said were good shared one office, the guys he said did not were in another. There was a third team, the UI guys, who lived in their own world and nobody was sure what they did... but it involved lots of code churn and without any visible results. Which is odd, you might think, for a team whose business is entirely wrapped in the visual element of the product. But they were not my problem.

I was finally granted access to the source code repository. I checked the code out of CVS, but I couldn't work out how to build it. When I asked where the main solution was... the build scripts, the make files... I was told that there was none: everybody just did whatever they had to in order to make it build on their local machine.

Which is to say, nobody knew what the fuck was happening, and if they did, they were keeping that information to themselves. It was like being in Terry Gilliam's Brazil.

I spent a day making the code I had build as best I could, but there were pieces missing. When I enquired after them I learned that they were in a different repository. I had to stand over the IT guys in order to get them to give me access. This happened three before I had all of the source code. I was shocked.

A source code repository is EVERYTHING to a team. If the code constitutes the raw ingredients of your product, the repository is the refrigerator. Even if the raw produce is low quality, you at least try to keep the fridge clean... because if the fridge goes you've got nothing. At Fjord systems, it was like they were keeping the ingredients in a series of buckets,w hich they'd stashed in the outhouse. I'd seen poor code and CM before, but this was rancid.

I spent most of the week getting it all to build in one place. Harder than it seemed: each of the four repositories had some overlap with the others, but they were not kept properly in sync. Once I was done I got IT to set up a new repository, I checked everything in and I said "Okay boys, this is the one place where you will get your code from, and the one place you will put it back." They seemed pleased with it, but it was hard to tell. In hindisght, I should have added "Try not to break it."

The code was horrible. Spaghetti and shitballs. It couldn't be saved. I said as much to Tyko, who gloomily agreed. I told Jacob, who took it pretty well. But our biggest immediate problem was the drivers. It was dangerous, letting our customers use those half-finished, barely functional drivers and we needed to fix them, stat.

I told Jacob that I thought Chucky was free... or at least he had been a couple of months earlier. Jacob remembered Chucky from ATB Software and so we got in touch with him. With Jacob on hand and in the office to hassle HR, we got Chucky sorted for a contract in days, not months, and we got his network access hooked up properly. We got him access to the new repository and turned him loose on the drivers. He worked sixteen straight hours from the minute his access was hooked up. The drivers, he said, were rubbish, but he could at least make them stable. Which he did. He did us proud.

Monday morning Jacob called a meeting of the development team. As soon as he had all of us in there (excluding those who came to work after 11am) he surprised me by saying "Aaaaand.... take it away Pike."
"Uh, what do you want me to say, Jacob?"
"Whatever needs to be said."

I took it by the horns. The product needed an overhaul and a redesign. I went through what was wrong with it, and how to fix it. I roughed out new designs on the whiteboard. I estimated about a year's work.

The team took it coolly. I was a stranger, the only non-Swedish speaker in the room, telling them that what they had been working for all of these years was wrong and needed to be done over. I worried that I was offending them. I worried that I was stepping on Anders, the new team leader... but something had to be done, and nobody else was prepared to step up. Nobody said much of anything when I asked for feedback beyond a noncommital "That looks okay." After each meeting I made an effort to talk to the other developers one-on-one, and the feedback they gave me was invariably good: "This seems like the sort of thing we need," said one of the better devs. Anders told me "We need to start over. I get a bad feeling when I look at the old code."

I started feeling good about it. I figured if this was Brazil, I was Harry Tuttle: renegade air conditioning specialist, suspected terrorist and all-round bad ass. When the going gets tough, Harry swings in through the window and fixes the problem, permit or not. He's played by Robert De Niro, so you know he is not-to-be-fucked-with.

So we made a plan. I would go back to Australia and start documenting up the new system and building  prototype infrastructure while the team finished off the new release. Chucky would begin work on a new set of drivers. Once the release was out and the infrastructure was ready we would reconvene in Sweden and bring the whole team onto the new project for build out. 

It was going to be a lot of work, and there was and obvious pessimism about whether we could accomplish any of this, but I was confident: I'd managed it before with a team a third of the size and a quarter as experienced, and this time I was going in knowing exactly what I was doing. In the eight or so years I had been a professional developer I'd yet to encounter a technical issue that I couldn't solve. It might take time or study or a shitload of pestering questions to somebody more experienced, but we'd solved every problem that came our way, every place I'd been. What I failed to realize was that it wasn't the technical problems that had proven to be insurmountable int he past; it was the people problems. And we already had people problems.

It was clear that Jacob wasn't going to be able to move to Sweden. Although I was sharing the apartment with him, I didn't know exactly what was going on until later in the week. I didn know that Jacob had been suffering from terrible migraines and that they were clearly work-related. This wasn't the easy-going, goofy Jacob I knew.

Anders and I were invited to a dinner with the CEO, the COO and another guest, Martin, who Jacob old me would be the new CTO. Since he couldn't make the move Jacob was stepping back into a similar role to the one he'd held at ATBSoft: that of a chief research consultant.

Martin was a charismatic, likeable guy from Sydney who impressed me as somebody who knew nothing at all about software development and who lacked even the barest interest in it. When they'd been recruiting a new CTO the executive staff had originally chosen Jacob over him. Now he was coming on board and Jacob would be working under him. That did not seem like a healthy position for Jacob to be in, and, with me as his chief henchman stirring up shit inside the development team, that didn't seem like a good place for me to be, either. It was clear that if shit went down... and it always does... Martin wasn't going to be much of an advocate for anything.

It was just like Brazil. The irrelevant and incompetent machinations of people way beyond my pay grade were going to be a problem for me in ways I didn't understand. It was just like Brazil, but I wasn't Harry Tuttle, the A/C fixit terrorist... I was Sam Lowry, the bureaucrat who, while trying to fix an administrative screwup, becomes an enemy of the state.