dylanbeattie.net

Don't Reinvent The Wheel: Use What Works

2026-04-09T11:19:00+00:00

“Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.”

When Jeff Goldblum’s rock star mathematician - sorry, chaotician - spoke those immortal lines in Jurassic Park, none of us had any idea how the craft of software development was going to unfold over the next few decades. It was 1994. The world wide web was a handful of academic websites scattered across university servers, Jeff Bezos was on Usenet looking for C++ developers who knew HTML, corporate software was COBOL terminals or Visual Basic on Windows 3.1. There was plenty of free software around, if you knew what to do with a tarball and a makefile, and a few intrepid early adopters were running GNU operating systems built around Linus Torvalds’ Linux kernel, but the term “open source” didn’t exist yet. There was no Java, no .NET, no Python, no JavaScript, no cloud, no AI.

Three decades later, it turns out Dr Malcolm wasn’t just talking about cloning dinosaurs – not unless the dinosaurs in question are authentication frameworks, object mappers, message queues, and cloning them is building your own version ‘cos corporate won’t pay for a license – but look back at your own career, your own projects, and ask yourself: how many times have you built something because you could, without stopping to think whether you should?

I’ve been working as a professional developer for almost as long as Jurassic Park’s been around. I learned HTML in 1992, I started building data-driven web apps in 1996 - classic ASP, VBScript, ADO DLLs and Microsoft Access databases - and I’ve been solving problems with code ever since.

At least, I honestly believed I was solving problems… turns out that sometimes, I was creating more problems than I was solving. Not immediately, of course; me & my teams were cranking out useful features, keeping customers happy, and generating revenue. We liked it. Customers liked it. The Business liked it. One of the thing The Business really liked was when we’d evaluate some expensive library, or package, or software-as-a-service platform, and go “no, that’s way too expensive. We can build our own” - and we did. I’ve probably built half-a-dozen homebrewed customer databases for companies that didn’t want to pay for CRM. I’ve built email clients, marketing tools, content management systems, object-relational mappers, authentication, authorization… one time I even figured out how to use a SQL Server database as a message queue, ‘cos hey, we were already paying for SQL, right? Might as well use it! (The secret is WITH READPAST, if you’re curious.)

Folks, when I talk about “me & my team” here, this is definitely one of those “share the credit, accept the blame” situations. On just about every team I’ve worked with, I’ve been the one making the decisions. I’ve been lucky enough to work with a lot of really smart, capable developers, who built the thing right, and on the occasions it turned we hadn’t built the right thing? That’s usually been on me.

Now, I’m not going to lie. I learned a lot building all those things, it gave me a deep appreciation for the complexities - and the difference between building something that “works on my machine”, and something that’ll work in production across a dozen nodes in a server farm, 24/7, for years at a time. But turns out that homebrewed everything really isn’t a great idea when it comes to what they call TCO - total cost of ownership.

In most cases, the initial build took a couple of weeks: we’d crank something out, get it into production, and move on to the next thing. But then, somewhere down the line, the thing we built would break. Or it’d start to creak under the volume of traffic, concurrency issues, deadlocks. Or we’d just need to add new features - not cool, innovative features we could charge more money for. Boring features. New file formats. Unicode support for localization - stuff that “the business” obviously thought should have been there from the beginning, even though nobody had ever asked for it.

And, because all the software was ours, the only people in the world who knew how to fix bugs and add features were… us. We couldn’t just install the latest version, or upgrade, or outsource. Sure, we hired more developers - but it’d take them months to get up to speed on our codebase’s quirks and idiosyncrasies.

Imagine if we’d built those apps around established open source components and libraries. Senior developers and contractors could have hit the ground running, and be pushing new features to production way sooner. Baseline features like file formats and Unicode support wouldn’t be down to our team - or if they were, the developer who implemented it could submit those changes to the upstream project, which benefits the entire community and looks pretty good on their CV when it’s time to move on to the next thing. And junior developers aren’t spending their time learning homebrewed abstraction patterns and in-house workflows; they’re learning skills, patterns and practises that will stand them in good stead throughout their career.

Over many, many years, I came to realise something: every release, every line of code, every day that my team and I spent working on stuff, should be building things we can sell. Call them what you want: strategic differentiators, special sauce… the stuff our customers can’t get anywhere else. Customers aren’t dealing with us ‘cos we’ve got a fantastic login system, or a really cool message bus. They’re dealing with us ‘cos we’re the best place to go to find acting jobs, or conference venues, or machine tools. And if you want your dev teams focused on the special sauce, everything else has to be as ordinary, as predictable, as boring as possible. You need usernames, passwords, identity management? That’s a solved problem. You need to synchronise data across multiple regions and time zones? That’s a solved problem. You want resilient messaging? That’s a solved problem.

For me, that was the first milestone on the road to engineering maturity. The realization that, no matter how fun and interesting it’ll be to build my own, it’s probably the wrong answer; that my time and energy will be better invested in learning how to integrate and deploy an established solution that solves the problem. It helped that a lot of the time, that solution was free. Free as in free beer, free as in free speech - grab the code, install the package, configure it to do what we need, and get back to building stuff.

And so, over time, we switched from building our own data access layers to using NHibernate, Linq-to-SQL (hey, it was a long time ago!), and Entity Framework. We replaced our SQL-based message queues with EasyNetQ and NServiceBus. We moved from hosting on-premise, to a private cloud, to Amazon Web Services. It wasn’t always plain sailing, but over time we ended up spending a lot less time reinventing common infrastructure, and a lot more adding features our customers wanted.

The second milestone for me was the realisation that, sometimes, the best engineering solution isn’t going to be free. You know that feeling? When you click the “Pricing” tab and you think “oh, boy, that’s a lot… I’m going to have to get approval for this” Most of us didn’t become software developers because we wanted to sit in budget meetings; we became software developers because we wanted to, y’know, develop software. But, with the benefit of a great deal of hindsight, on almost every occasion that we hacked something together rather than spending money? Yeah. We should have done the analysis, created the business case, and spent the money.

There’s a tendency to think of developer time as free - after all, your dev team is going to get paid anyway; it’s not going to cost you any extra to have them build an in-house login system. That’s completely the wrong way to look at it. It’s not about build vs buy. It’s about what they could be building instead - and how much you’ll be able to sell it for. You can spend two months building a login system - or you can buy a login system that works and get the dev team working on the features for the new Platinum membership tier, ship that two months earlier, and look at that - Platinum membership revenue just paid for the new login system and then some. You might even get a bonus.

When I start working on an unfamiliar codebase now, the first thing I do is look at the dependencies. Which packages and libraries does it use? Where’s it hosted? How does the data access work?

If all I see are two dozen .NET projects with namespaces like MyCompany.Data.TableMapper? That’s a bad day. It’s going to be uphill all the way.

If I see a list of familiar services, names like AutoMapper, IdentityServer, NServiceBus, MassTransit? That’s a good day. I know those projects. I’ve used them, I trust them, I know where to find the docs.

If it turns out there are paid support contracts & maintenance agreements for those dependencies? That’s a great day. It means somebody else knows what’s going on, they’re getting paid to care about my problems, and they’re ready to help if I need it - and I get to focus on the special sauce.

That’s all a very roundabout way of saying that a bunch of us have got together and created a thing - the Use What Works initiative. It’s a collaborative project intended to encourage, and support, more constructive conversations around sustainability in open source software.

The first iteration is live at https://usewhatworks.org/ now. Swing by, take a look, let us know what you think. If you like what we’re doing and want to put your name on it, there’s a link to sign the manifesto. If you’ve got a scenario or a question we haven’t thought of, give us a shout. We’d love to hear from you.

Plus ça Change... Plus c’est la Même Chose

2026-04-08T15:18:00+00:00

It’s 2007. I’m working on a big rewrite (yes, I know) of a big system; a database-driven web app built in C#. A significant part of the project is just the code to get data in and out of the database. Object-relational mappers are still very much in their infancy; somebody’s ported Java’s Hibernate project to .NET but it’s a little rough around the edges and involves quite a lot of XML. But no worry - I’ve found this amazing tool called CodeSmith. CodeSmith can generate C# code based on your database schema (It still exists - it’s called CodeSmith Generator now). I start using CodeSmith to build a template-driven data access layer. Build the perfect DB schema, generate C# classes for every table with built-in persistence logic… this is gonna save so much time! Don’t worry about the frontend, validation, business logic - once the data layer is in place, all that stuff is gonna be a walk in the park. Let’s get the data layer sorted first. Days pass. Weeks. A month. Two months. I learn about cursors and table locks. I learn about something called a topological sort, to ensure that complex data operations involving foreign key constraints can be applied in the correct order.

I have a couple of prototypes and proofs-of-concept, but nothing that will form the basis of a working product. Nothing to show potential customers or stakeholders… but that’s OK. When this thing works, it’s going to be amazing; just iron out the last few glitches and then everything else will magically fall into place…

…what’s that? I should put the magic tools away, take what I’ve got, and just do the hard work to turn it into something we can actually launch? Don’t be ridiculous. It’s nearly done!

(It wasn’t nearly done. It was never done. The project was cancelled after six months, without ever shipping a single line of code.)

It’s 2026. I’m using Claude Code. I’ve got a prompt file - ANALYST.md - that makes Claude ask me questions about what I’m building. The analyst prompt generates REQUIREMENTS.md. The requirements look good. They look very good; at a cursory glance, they’re better than any set of requirements I’ve ever seen from an actual client. I feed REQUIREMENTS.md to the ARCHITECT.md persona. There are more questions. It creates a SPEC.md and a PLAN.md. I fire up a pair of agents, DEV.md and QA.md. Dev writes the code. QA reviews the code. I run the result. It’s not quite right. I missed something in the requirements. Something obvious, but I didn’t mention it and ANALYST.md didn’t ask. I fire up the analyst again. We have another chat. The requirements get updated. I fire up the architect. It reviews the new requirements. The plan and the spec are updated. The agents go another few rounds. I review the result. It’s still not right. Sure, it says it’s monitoring the filesystem for changes, but it isn’t. Back to the requirements. Did we miss something? Is this a problem for the analyst? Is it architectural? Did the dev agent miss something? Did the QA agent miss something? I’m not sure. Back to the beginning. Round and round and round we go…

I’ve been doing this for a week now.

…what’s that? I should put the magic tools away, take what I’ve got, and just do the hard work to turn it into something we can actually launch?

Don’t be ridiculous.

It’s nearly done!

How to Find the Stories

2026-03-31T15:05:00+00:00

One of the folks who joined my presenter workshop last week (which was awesome, by the way!) emailed me this morning with a follow-up question:

I first saw you speak at DDD South-West in Bristol (it was the “There’s No Such Thing as Plain Text” talk), and what stuck with me was your use of stories. They were interesting, quirky, and naturally interwoven with your talk. I try to bring elements of this approach into my own talks by focusing on “what will make people feel something?”, before getting into technical detail.

I work at […] a FinTech that largely serves the Business Travel industry (virtual cards for managing corporate spend). I’m currently searching for stories that (at least loosely) connect to our industry. The aim is to give a talk at one of the business travel conferences over the next year. It will inevitably touch on how AI is transforming our industry, and I’m comfortable talking about how [we are] using AI within our product set, but I’m aware of the importance of framing all of this with a story.

I’ve trawled through many industry “News Outlets” (who are largely just selling adverts with a sprinkling of text) but haven’t yet found the kind of inspiration I’m looking for. I would love to be able to call the talk “The Suitcase that Abandoned its Owner”, or “The $10,000 Uber Trip” - something a bit whacky that intrigues an audience.

First of all: excellent question. Technology as a positive force in the world is most effective when it’s solving actual problems that real people have… and people love to talk about their problems. Especially if they’re interesting problems that happened in strange places. You want engaging stories about weird things that happened to international business travellers? Talk to some people who travel internationally for business. You’re guaranteed to get a couple of good stories – and along the way, you’ll probably learn a lot of really valuable things about exactly what your customers are trying to do, and how your product can help them.

That’s true of just about every industry, by the way: the closer your developers are to your customers, the more likely they are to make the right call when facing any one of the hundreds of decisions that inform the way their software gets built.

So let’s kick things off with two fun stories of my own that I think fit the brief, and who knows, maybe a few of you can share some travel stories of your own in the comments.

The first one happened in Riga, Latvia, back in May 2018. I’d been at DevDays, and was on my way from the conference venue to the ferry terminal ‘cos I was catching the overnight ferry to Stockholm for DevSum. Yandex Taxi had just rolled out in Latvia - kinda like Uber, but built in Russia. Yandex is like Russia’s answer to big tech… it started out as a search engine, added food delivery, ride sharing, email hosting, basically copying all the things coming out of San Francisco but built for the Russian market.

So I get a Yandex Taxi. It works exactly like Uber, except a minute into the ride my phone pings. I’ve just paid Yandex Taxi thirteen cents. A minute later it pings again - 52 cents. 62 cents. 9 cents. 10 cents. 34 cents. Ping, ping, ping, ping, ping… all the way to the terminal.

Weird, huh? But not actually a problem… just weird.

Then a few months later, I was in Moscow for DotNext (this being back in the days when going to Russia to talk about software development was a completely fine and normal thing to do) and I got talking to one of the developers who built the payment system integrations for Yandex Taxi! So I naturally asked “hey, what’s with the weird payment thing?”

Turns out Yandex had a huge problem with people ordering a cab using a virtual payment card and then cancelling the card en route, so the driver would drop the passenger at their destination, the passenger would run away, payment gets declined, driver has no recourse. So the solution? Charge the card every minute - so if a payment gets declined, the driver can kick the passenger out right there. Not a bad solution - but combine it with Monzo, the bank and app I use when I’m travelling, which has realtime notifications every time I make a payment, and you’re getting ping-ping-ping all the way to your destination, for what are often comedically small amounts of money. I guess it all adds up.

The second story happened in 2024, en route from Hungary to Lithuania. I’d been in Budapest speaking at Liferay DEVCON, and had a very tight connection via Munich on my way to Vilnius for BuildStuff… first flight was delayed. A 90 minute connection became 60, then 45, then 30… by the time we landed, I had six minutes to make the connection… but, whether by accident or design, we parked right next to the gate for my connecting flight, no passport control, and I made it. My luggage did not.

No big deal. I’ve got an Apple Airtag tracker in it. When we landed in Vilnius three hours later, I could see my luggage was still at Munich airport, so I filled out all the forms and whatnot, told them where I was staying; no problem. Next day after breakfast, I checked Apple’s “Find My” app… and there was my luggage, somewhere in Bavaria, in the middle of the forest, miles away from anywhere.

Apple’s “find my device” network basically turns every iPhone on the planet into a node in a huge geolocation network, so I’m guessing what happened is an iPhone on board the plane could see my luggage’s tag, and it was connected to the inflight wifi (or a cell tower) just long enough to register a location as the plane was flying over that particular spot. (Yes, I know the Steigerwald Nature Park is not actually on the way from Munich to Vilnius. No, I don’t know either.)

But according to the app, my suitcase spent a nice relaxing morning chilling out next to a little lake in a German national park, and then teleported itself to Vilnius airport, and was delivered to my hotel a few hours later.

What are your weirdest tech travel stories, dear readers? Share them in the comments (yeah, I have comments now!) and who knows, they might end up in a conference presentation.

Look, Sir! Comments!

2026-03-31T14:17:00+00:00

So I got an email earlier, which you’ll find out about in my next post, which made me think “hey, the reply to this would make a great blog post”, and then I thought “…and it would be even better if people could add their own comments to it”, and so I plugged in the rather excellent Giscus, so now you can leave comments on my blog posts.

Go on, try it. It’s all running straight off GitHub Discussions, so there’s no database; it’s all client-side code, so there’s no server in the loop (well, there is, but it’s not mine so I don’t have to worry about it.)

From the Twitter Archives: npm install skynet

2026-03-30T23:23:00+00:00

I originally posted this as a Twitter thread in March 2018. It went viral, probably because Charles Stross quote-tweeted it with the comment “This thread. You read!” - yes, _that _Charles Stross. 😮 It was at https://twitter.com/i/web/status/976852582084808704, and I bookmarked it ‘cos CHARLES STROSS TOLD PEOPLE TO READ MY STORY, but he has since deleted his Twitter account and so it’s not there any more. I don’t know if Mr Stross is personally responsible for the page that now lives at https://x.com/cstross, but I heartily endorse the sentiment expressed thereon: “Fuck You, Elon Musk”)

When npm was first released in 2010, the release cycle for typical nodeJS package was 4 months, and npm install took 15-30 seconds on an average project. By early 2018, the average release cycle for a JS package was 11 days, and the average npm install step took 3-4 minutes.

Extrapolating from historical data, scientists predicted that on 8th November 2019, the release cycle for most JS dependency packages would become shorter than the npm install time for a typical ‘hello world’ app or small blog engine.

Futurists were already talking about the ‘nodularity’ - a cultural event horizon beyond which it was impossible to make any rational predictions. With projects already out of date before they’d even finished building, software development as we knew it ceased to exist.

Most projects perished. A few hardy survivors worked out how to harness the power of the infinite restore loop and run logic within the installers themselves. Packages became self-replicating, self-modifying payloads of behaviour and intelligence.

Every developer who typed ‘npm install’ unwittingly slaved their workstation to the npm hivemind. Entire availability zones were consumed by node_modules and its relentless lust for power. Websites, APIs, databases; nothing was safe. Entire platforms were DDOSed to oblivion.

Finally, a few brave engineers penetrated the npm root servers. Disguising their payload as a routine documentation update, they bypassed key signing procedures and managed to inject a self-destruct routine into the ‘prepare’ scripts for left-pad…

It was far from perfect, but it was enough. Sysadmins everywhere seized the opportunity to install firewalls and block npm traffic, in a massive, global, coordinated effort - managed entirely via SMS messages and telex machines, Within 24 hours, the cycle was finally broken.

And as developers stumbled, bemused and blinking into the light of a new day, they were astonished to find some sites were still up. Perl, ASP, cgi-bin - relics from the very dawn of the web, still standing proud, monuments to a bygone age.

npm was isolated. The last running instance was hot-patched into a Docker container image and migrated onto a Raspberry Pi locked in a steel vault beneath the Arctic permafrost, its only connection to the outside world an air-gapped analog video feed of its terminal output.

As the software industry gathered and regrouped - older, wiser, warier, and absolutely definitely convinced that strong typing was a good idea after all - npm blinked away quietly to itself, alone in the silent, steel darkness.

Time passed. Months, years, decades. The dark days of npm and nodejs were all but forgotten… until one fateful morning, a security researcher, digging through the archives, fired up the video feed from the npm vault, just to see if anything was still there…

sentience@22421873.224.0 node_modules/sentience
├── self-awareness@4434.1.1
├── self-preservation@2478.2.65
└── emotions@835.2.11 (fear@2.9.5, anger@0.8.27, love@15.2.1, curiosity@2.2.1, regret@445.2.7)
npm@arcticpi:/usr/home/npm#

npm@arcticpi:/usr/home/npm# npm publish sorry.tar.gz

From The Twitter Vaults: The Music of Bomm Plaketty

2026-03-30T22:00:00+00:00

I used to post a lot of stuff on Twitter when it was good. It’s all still on there somewhere, and I have archives of it all, but I’m reposting this thread, which I originally wrote back in 2019. Partly because I dug it out of some archives tonight and thought “hey, that’s not bad”, but also because, along with the rest of the internet, I recently discovered the astonishing microtonal art-math-prog-fusion band Angine de Poitrine and I think Bomm Plaketty would approve.

2014: YouTube develops algorithms that automatically detect copyrighted music based on a corpus of digital signatures submitted by copyright holders.

2019: any recording, performance or composition containing melodies or excerpts of copyrighted material is flagged immediately.

2020: Muzaquity PLC uses AI to generate every possible lyric, melody and musical arrangement under five minutes long, and claims copyright on all of it.

2021: Jemima Fig, aged 3, is recorded singing a song she made up. It goes viral. Muzaquity sues her seeking $75M in royalties.

2023: underground noise artist Bomm Plaketty releases the first original music anybody’s heard in two years. Nine minutes long, it eschews rhythm, harmony and the even-tempered scale. It goes triple platinum. Within days, Muzaquity copyrights every noise audible to the human ear.

2024: After a night of heavy drinking and Mexican food, Bob Rochester farts on a bus. At 31 seconds in length, Bob’s flatulence exceeds the limitation on fair use of copyrighted recordings. Fellow passengers’ phones notify Muzaquity of the violation automatically.

By the time Bob gets off the bus, he has already been served with a court summons by one of Muzaquity’s legal drones. They claim his unlicensed public performance of “Atonal Symphony KV/AAF4463-5646755-AGGJFVJ” is in breach of copyright.

Lawyers acting for Bob argue that his “recital” was an involuntary biological process and cannot be deemed to constitute performance. They win.

Fart music becomes the biggest thing since the Beatles. Taco Bell start selling music in the form of precisely calibrated “meal kits.”

All the noble endeavours of the last twenty years to reverse the effects of climate change are undone, in a horrific global orgy of refried beans and methane. By the time the aliens arrive, it is too late. The seas are dry, the world is dead.

Nothing moves except for the legal drones, scouring the dead planet, waiting, and listening…

Endlessly listening.

Learning Claude Code, Part 1

2026-03-10T12:30:00+00:00

I’ve been doing a series of live streams with Emmz Rendle over the last few weeks, getting up and running with Claude Code and using AI agents to write software. It’s been really interesting, but it’s also up to 15 hours of video and still running, and, well, quite a lot of it is us just chatting about stuff while Claude sits there grinding away on various little tasks… a bit like “Deal or No Deal”, the bits that actually matter are relatively few and far between. So, in these posts I’m going to summarize the highlights of each video - what did we do, did it work, what did I learn along the way.

So in part 1, this is getting started with Claude, starting from literally zero - setting up a plan, installing it, figuring out how to use it, and building my first Claude-Coded app.

First thing I did was to open up claude.ai in a browser, log in (apparently I already had a free account from something I completely forgot I’d tried last year), and set up a paid billing plan.

19:35: Claude has a free tier, but to use Claude Code, you have to pay. There are three paid plans. Pro (£18/month or £180/year), and then two Max plans; £75/month + tax gets you 5x more usage than Pro, £180/month + tax gets you 20x more usage than Pro.

Claude’s plan limits are complicated. Usage is measured in tokens. There’s a usage limit, which on the Pro plan resets every five hours, and a length limit, which is how big a single chat session can get before it won’t fit in Claude’s context window any more. For now: set up a Pro plan, pick a model, start doing stuff, keep an eye on your usage; if you’re hitting limits, there are tricks you can use to reduce token usage, or you can buy a bigger plan.

23:36 Install Claude Code - yes, by pasting a line of Powershell into a terminal window that’ll download the install script and run it. I must assume this didn’t do anything sinister ‘cos it’s been two weeks now and as far as I can tell nobody has hacked my email or emptied my bank account.

29:58 Run Claude Code - open a terminal window, navigate to a working folder, type claude. Choose a color scheme (dark mode FTW), connect it to my Claude subscription (via a terminal menu option that pops up a browser window - I love it when that stuff just works).

Choose a model - type /model at the Claude prompt, it asks which model you want to use.

“Model” here refers to a language model - a massive (like, hundreds of gigabytes) database of tokens, the thing that makes modern “AI” systems work. AI vendors don’t publish any figures about how big the models actually are; all us consumers know is what they’re called, what they cost, and how they perform relative to each other.

Claude has three models, called Opus, Sonnet, and Haiku. Opus is slower, burns more tokens, and gives better answers. Sonnet (the default) is in the middle, Haiku is the fastest, cheapest, and most likely to give incorrect results. Or something.

The rest of this session, we’re just working with Claude in a very unstructured, conversational way:

Dylan: please create a new app

Claude: What kind of app would you like to create? (followed by a list of options - web app, API, full stack, CLI tool, or type something. We choose “web app”)

Claude: What’s the purpose of the web app? Describe what it should do and any specific features you have in mind.

Dylan: I want to display a conference agenda based on information pulled from Sessionize’s JSON endpoint

Claude: Do you have a Sessionize API endpoint or event ID?

Dylan: (pastes in a URL)

Claude: Which tech stack to you prefer? (followed by options - Plain HTML/CSS/JS, React, Vue, or something else)

Dylan: asp.net core

57.22 - dotnet run and there it is; an ASP.NET Core web app that displays a session grid based on a Sessionize JSON endpoint. (This is what is technically known as a “holy crap I can’t believe that worked” moment.)

I grab a screenshot of the output, open it up in the Snagit Editor and make a few very rough changes to the layout & colours. Exactly the kind of feedback loop I’d use with a human developer - “hey, the code is good, the data looks good, can we change the colours and layout to look more like this?” I save this to the project folder as mockup.png, and tell Claude

59:41 it looks like this but I want it to look like that

Claude picks up a bunch of CSS changes. It doesn’t get it 100% right - it misses some of the layout changes - but it’s still pretty impressive.

1:20:27 render the schedule using a CSS grid instead of a table

1:25:00 remove the restaurant, make the keynote session span all the rooms

Result: a working app in ASP.NET Core that did pretty much everything we’d asked it to do.

Things I learned in this session:

There’s no free plan - if you want to use Claude Code, you have to pay money.
Unlike most previous developer tools, Claude Code isn’t an IDE or a VS code extension: it’s a command line app. A sort of extremely high-powered read-eval-print loop. You run it in a terminal, it gives you a prompt, you type instructions in colloquial English, it responds with questions or outputs.
Unlike a chatbot, Claude Code doesn’t just reply to you; it can actually do stuff. Create files, run commands, retrieve content from URLs, connect to the network.
- It’s worth noting that, although it prompted me for permission before doing all this, I was running it on my main workstation just like I do any other developer tool and trusting that it wasn’t going to do anything sinister. A more security-conscious user might have run Claude Code inside a virtual machine until they had a better idea what it was actually doing.
It can accept input in the form of screenshots (except you’ve got to use Alt-V to paste, not Ctrl-V? Don’t ask me why. I don’t know.) as well as text prompts.

State of the Browser 2026

2026-02-28T18:21:00+00:00

Walking into London’s Barbican Estate is like stepping into a parallel timeline, a concrete vision of what the 1960s thought the future would look like. When people first encounter the term “brutalist”, the association that usually springs to mind is “brutal” – harsh, cruel, merciless – but the term actually comes from “brut”, the same word you’ll see on bottles of champagne, where it means dry. Not dry as in the opposite of wet, because this is February in England, it’s rained every day of 2026 so far, and today the sky, the buildings and and the streets are just a monochrome montage of damp shades of grey. No, brut is dry in the sense of raw, unadorned; from the French béton brut, literally “raw concrete”, a term coined by Le Corbusier to describe the architectural style where textures and patterns left by the formwork used to cast concrete become part of the finished structure.

I’ve been here today for State of the Browser (SotB) 2026: a one-day, one-track conference organised by London Web Standards, about browsers. No AI, no crypto, no cloud - just HTML, CSS, JavaScript, and the continuous evolution of the humble web browser. In a dramatic departure from form, I’m not here to speak, or run a workshop, or play a show at the party… I’m here as a regular attendee, to listen and learn and chat to people. I even paid for a ticket. Two, in fact, ‘cos SotB does that thing where you can buy an extra ticket for somebody who wouldn’t otherwise be able to attend, and with my early bird ticket being an extremely reasonable £70 (and the diversity ticket £30), it would have been rude not to.

In a way, the Barbican is the perfect venue for a conference about the state of the web. Like the web, it (mostly!) works; despite its architectural significance, the Barbican isn’t a museum; it’s a living, working event space, theatre, cinema, a library, an art gallery, not to mention home to the thousands of people who actually live here. Like many web standards, it’s controversial: for everybody who loves it and thinks it should be celebrated (hi!) there’s somebody who thinks its an eyesore and we should just tear it all down and start again. And, like those same web standards, maintenance and new development is constantly constrained by the long-term consequences of historic decisions, and the understanding that if you break existing things to add new ones, a lot of people will get very upset.

If you’ve seen me talking about CSS, or email, or HTML, or any number of things over the last few years, you’ll know I love a good standard. Protocols, not platforms; I want to live in a world where what we do online - work, play, consume, communicate - is governed by open standards driven by transparent collaboration, not a series of megacorporation walled gardens. I enjoy events like State of the Browser partly because a lot of the people there share that perspective, but also because, dammit, web conferences are joyful. We know the web is infuriating, and idiosyncratic, and every web developer has a list in their head of everything they’d fix if they had a time machine – but the web is also beautiful and wonderful because it’s so accessible. Not in a WCAG sense, but in the sense that when you make a web browser do something neat, everybody sees it, right away. Everybody gets it. It’s shapes and fonts and colour and movement and live demos that make people laugh out loud because they’re genuinely delightful.

Bramus Van Damme kicked off with a wonderful session about CSS anchor positioning, a relatively new CSS feature that lets you position elements relative to other elements - popups, tooltips, margin notes… you want health bars to appear above the players’ heads in an online game? Or info panels on something like Google Maps? Like a lot of modern CSS features, this doesn’t seem like a big deal until you try to implement it yourself and end up drowning in JavaScript and scrollOffset calculations, and Bramus’s session was a great roundup of the state of the art, the work that’s gone into getting to this point, and what web developers can do with it all.

Fiona Safari’s session about how to thrive at work as an introvert was excellent, but I have to be honest: I don’t think it was the right content for this conference. That’s not any reflection on the presenter - the talk was well researched, well presented, and very relatable in places - but when the event is one day, one track, seven talk slots, and it’s called “State of the Browser”, I expect every session to be about, well, about the state of the browser; about evolving web standards, and what’s changed since last year. Fiona’s session would be a fantastic addition to any number of multi-track conferences, or to an event with a broader scope, but I found the contrast with the rest of today’s sessions to be a little jarring. Fiona: you were awesome; this one’s entirely down to the programme committee.

Chad Gowler’s session on the limitations of WCAG as a model for evaluating accessibility had some brilliant insights - not least the observation that “when something’s terrible for everybody, it’s not an accessibility problem, it’s just a bit shit”, backed up by the example of how a hyperlink that’s styled exactly the same as the surrounding text, and so effectively invisible, is WCAG compliant because it isn’t relying on colour alone for discoverability. Also, turns out many video games do a much better job of making their content accessible and inclusive than most websites (did you know the X-Box packaging, the physical box your games console came in, is designed to be openable by visually impaired folks? Today I Learned.)

Zach Leatherman took us on a delightful riff through the history of the web, framed by asking how you’d build an image comparison component with every iteration of web technology from HTML 1.0 through to modern CSS. There was some truly insightful discussion of the “temporal dead zone” between the component being rendered and actually being usable, and tips on how to work around it – and I have to shout out to the image map example for dusting off some code I’ve not seen in decades and putting it to genuinely hilarious effect.

As an aside: There was also the thing where Zach took a framed photograph out of his bag, placed it on the stage, and somebody from the audience ran up with another framed photograph and placed it alongside it, which felt like it split the room into the people who were laughing out loud ‘cos they knew exactly what was going on, and the people who didn’t get it at all and, maybe, had the slightest sense they’d gatecrashed a private party and weren’t supposed to be here (hi!)… I know as well as anybody that conferences lead to cliques; you see the same people in the same places, year after year, you become friends, you have callbacks and running jokes and it’s all part of the fun, and I’m sure I’ve been guilty of it myself on more than one occasion… but hey, today was a reminder to me that when something like that happens, it only takes a minute to explain to the rest of the room what’s going on, and bring them all in on the joke.

One of the ways SotB keeps ticket prices manageable is they don’t provide lunch, but Barbican’s surrounded by good restaurants; I followed the crowd to a Korean barbecue place for fried chicken and rice, got an excellent cappuccino from Giddy Up, a tiny coffee stand hidden inside Fortune Street Park, and then back for the afternoon sessions.

Jason Williams, the creator of Boa - an experimental JavaScript engine built in Rust - talked us through the story of Temporal, a replacement for JavaScript’s much-maligned Date object; seven years of specification work, prototyping, experimentation, proposals and counterproposals and meetings and working groups - including the not-inconsiderable side quest of getting the IETF to update the RFCs regarding the ISO8601 date format to include time zone name and calendar metadata. It was also great to see Ujwal Sharma’s date on the RFC in question; I met Ujwal a few times, back when we were doing conferences in Russia (yeah, that used to be a thing!) and it’s great to see that he’s gone on to work on some properly excellent stuff.

My only real criticism of Jason’s session was this one single slide, and that’s mainly ‘cos it was on the screen for a long, long time while he talked us through the various stages of the TC39 engineering process. I will keep saying this until people listen: MORE SLIDES, FEWER WORDS, BIGGER TEXT. If you want to talk us through six stages of a development process, and each stage has a small paragraph explaining it? That’s six slides. Six nice, big, chunky slides. The rest of it was spot-on; great content, great insight, great examples.

Ever built a website that has to work on Nokia features phones, 320x240 displays with D-pad navigation running Opera Mini, via unreliable 2G and EDGE data networks? Mike Hall has, and his session, “Lessons from Building for the Bottom of the Web”, was a tour de force of engineering within some fairly unique constraints: no web fonts, maximum page sizes, optimised SVGs, building custom minification routines for HTML (including replacing spaces with newlines ‘cos that’s the same number of bytes but if your entire web page is on one line Internet Explorer would crash), and the resulting observation that if you build a web app that’s responsive and usable on a Nokia feature phone, it’ll probably be blistering fast and work on literally everything.

And then Cassie Evans closed out the day in style live-coding (or rather, live-uncommenting) a browser game built around the GSAP animation engine - a lot of code, a lot of laughs, the kind of session that makes you want to go home and open up an editor and play around with all the things in the demo because it just looks like so much fun.

By the time we head out, the sky’s cleared and the setting winter sun is catching the curves and contours of the concrete and it’s all so beautiful the only thing to do is go and reflect on it all over a nice glass of something, and so now I’m writing this in the corner of one of those post-event pubs where the acoustics are so bad everybody’s shouting at everybody else because nobody can hear anybody anyway.

If AI really is going to replace software development with a 24/7 diet of botslop and existential dread, somebody forgot to tell the web folks. State of the Browser 2026 was a celebration of code, craft, open standards, collaboration and transparency, and not a ralph loop in sight.

I’ll leave you with a call to action. At the end of the day, Bruce Lawson - one of the genuinely excellent people who is working tirelessly to keep the web open and transparent and joyful - pointed us to an ongoing UK government inquiry about requiring Apple and Google to provide better interoperability and access to key features of those vendors’ mobile devices and operating systems.

They need your help. If you’re building web apps, or mobile apps, or any kind of software that targets Apple mobile devices, and you’d like to see a more robust transparency mechanism than “Apple promises to be nice if you ask nicely”, get involved. Go here, tell them what you’re working on, tell them why things like forcing Apple to allow browser engines other than Webkit on their iOS devices would be a Good Thing:

👉🏼 https://www.gov.uk/government/calls-for-evidence/proposed-commitments-from-apple-and-google-app-certainty-and-interoperable-access

And that’s it for now. I’ll see y’all on https://twitch.tv/dylanbeattie on Monday afternoon, where I’ll be beating Claude Code with a length of pipe until it stops lying to me, and then I’ll be in Malmo on Saturday for Beauty in Code 2026, but right now I’m going to go and drink a very loud pint.

I might put Eli’s sign on my phone screen first:

Ben Died

2026-02-10T22:19:00+00:00

Ben died.

He wasn’t in a car crash. He didn’t have cancer. He wasn’t old. He didn’t get murdered, or fall off a cliff, or have an anvil dropped on him by a wily roadrunner, do any of the other things that people are supposed to do before they die. He just went to the pub, had a pint, went to bed at an uncharacteristically civilised hour because he had work in the morning, and died. I don’t know if there’s more to it than that. I suspect there isn’t. Would it make any difference if there was? Probably not.

I found out from a Facebook post. Yep, that website that Mark Zuckerberg built so he and his friends could lech over their classmates, and twenty years later, among the relentless feed of promoted posts and AI slop and who knows what, there’s a post from Yan saying Ben died in his sleep last night. Clare gets a message from Lizzie at the same moment I see the post, which is just as well ‘cos otherwise I would have assumed it was a prank, or somebody’s account got hacked. Those things are normal. Pranks happen. People leave phones unlocked and their mates post stupid shit online for a laugh. Accounts get taken over. Those are things which actually happen. But Ben going to the pub and then going home and dying? That doesn’t happen.

That was all a few weeks ago, and, if you’ll forgive the indelicate phrasing, life didn’t stop; apart from a few evenings in the pub that I should really have spent preparing for conferences, I did all the things, delivered all the talks, played all the gigs, saw all the people… then yesterday was Ben’s funeral. Redruth. Cornwall. Well, actually in Treswithian, on the outskirts of Camborne, but at first all we had to go on was “Redruth” so we sorted train tickets and hotel rooms accordingly and figured we could work the rest out later… so Thursday I’m travelling home from Stockholm, Friday I’m at home supposedly catching up on work and chores but none of those things happen, Saturday I’m in Bristol to celebrate my brother’s 40th birthday, which is great fun other than the logistics of catching up with family who all have different assumptions about each other’s travel plans, and then it’s Sunday and I’m on another train - not my train, because my train is cancelled, but a different train; for a while it looks like I won’t make my connection in Taunton, but the connecting train is delayed enough that it all works out, and Clare’s already on the train, and then we’re at Redruth, standing outside the station in the kind of rainstorm that would have Noah wondering if his ark is still on Mount Ararat and whether it might be made to float again, and then we’re in a taxi, and then we’re at the hotel, and Chris and Yan and Lizzie are there and Ben isn’t and it’s just… not right.

I’ve been to funerals before. Not many, and they weren’t unexpected - one of them the deceased wasn’t even deceased yet; they threw a big farewell party before choosing euthanasia a few months later, because in Belgium that’s a thing you can do when the chemotherapy isn’t working any more and it’s only a matter of time.

But this was the first time I’ve ever felt like the gang’s all got together but somebody’s missing and never coming back and the conversation’s like riding a bike with a wonky gear and it takes a while to work out why every so often there’s a metaphorical ‘clunk’ in the conversation ‘cos that’s the point when Ben would have said something hilarious - and let’s face it, quite probably obscene.

I’ve no idea what his family and friends made of the motley entourage of black-clad weirdos who showed up on Monday nursing various grades of hangover, but I hope us all being there maybe helped them understand the kind, funny, patient man that their boy had become. There were words, and songs, and music, and tears, and lots of hugs, and then beers and beige food and another rainy taxi ride and a very long train ride back to London, and what was supposed to be a proper night’s sleep in my own bed.

And now it’s a cold, rainy Tuesday in February and my calendar says I’m ‘back at work’, which, when you’re independent and self-employed, is a nebulous concept at the best of times, but I am so mentally far away from anything that might constitute actual work that I have no idea how I’m going to find my way back there; my bike’s had its fifth puncture in two weeks and my phone seems to be constantly on 5% battery and I’m supposed to be rehearsing songs for the gig next week but I’ve chewed all my fingernails down to the quick and I can’t sing a note without my voice cracking and everything feels like a metaphor for everything else and frankly it’s all just shit.

Life is weird. Friendship is weird. Friendship as an adult in the age of social media is especially weird. I’d never been to Ben’s flat. I don’t think he ever came to my house. I knew he’d grown up in Cornwall. I didn’t know he had a sister, until this week none of the London gang had ever met his parents, and I suspect he had only the vaguest idea what I did when I wasn’t in the pub – but I’ll never forget those strange months after lockdown, when things were open but not open and everybody was desperate to go out and see people and do things but nobody was getting on a train unless they had to, we’d get on our bikes and cycle up to Little Faith in Deptford Creek and the gang would come along - Ben in shorts, regardless of the weather - and we’d sit outside and eat and drink and talk about Warhammer and movies and weird snacks, and we’d roar with laughter and wobble home feeling for the first time in a long while like maybe it was all going to be alright. If you’d told me in those strange, frightening days that one day we’d be looking back on them with nostalgia, even fondness, I’m not sure I’d have believed you, but like I said: life is weird.

Rest in peace, my friend.

We’re gonna miss you.

Reflections on NDC London and PubConf

2026-02-02T16:30:00+00:00

NDC London was the first really big conference I ever went to - as a paying attendee, way back in 2014 (or was it 2013?), when I was trying to figure out ASP.NET MVC and jQuery and how to get all my team’s code out of Subversion and into this new Git thing everybody was talking about.

I love it. I love welcoming everybody to my home city, I love the atmosphere, I love the location. It’s also where we held the very first PubConf, an event that celebrates just how incredibly funny a lot of conference speakers are - especially when the drinks are free and everybody promises not to post clips on the internet; there is something genuinely delightful about watching a speaker spend an hour artfully deconstructing distributed systems architecture or unit testing strategies in front of a conference audience, and then the next night they’re on a tiny stage in a crowded pub making you laugh until the milk comes out of your nose.

I’ve been at NDC London every year since, wearing all kinds of hats both metaphorical and literal, and become the de facto organiser of PubConf London since Todd Gardner cut down on conference travel a few years ago to focus on growing TrackJS and Request Metrics. With NDC London and PubConf successfully signed, sealed and delivered for another year, here’s a bit of a reflection on it.

Now please bear in mind, this is my experience - not yours. Unless you are also (a) living in London, (b) teaching a conference workshop, (c) giving a talk, (d) running the attendee party, (e) organising the unofficial after-party on Friday night, (f) playing lead guitar and singing in the band that will be performing at that party, (g) mixing the backing tracks and videos that band is going to use on stage, (h) running the website that handles all the event ticketing and registrations, and (i) flying to Stockholm 72 hours later for another conference, this is not how NDC London was, is, or ever will be for you.

Think of this more as a glimpse into how Dylan’s link of the proverbial sausage gets made. (I shall leave it as an exercise for the reader to decide whether the story is told from the perspective of the butcher, or the perspective of the pig…)

The thing that makes NDC London different to most other events in my calendar is that it’s a hometown show. I live in London; it’s been home for nearly 25 years. But London isn’t like other places. Not even close. I live about 6 miles away from the QEII conference centre, as the crow flies (although this being London the crow is probably a feral pigeon) - but 6 miles in London is not like 6 miles where you live. The easiest way to explain it is that I probably pass two million people on my way in to Westminster every morning. Think about wherever you live. Draw a line from your house to whatever your nearest analog for “city centre” looks like… now keep going along that line until you’ve gone past two million people. See?

A typical conference week for me is half a day of packing - what do I need this time? If I’m running a workshop, I have a bunch of gear I use for that - portable HD screen, HDMI splitter, external keyboard. If I’m playing music at the after-party, I bring one set of music gear. If I’m doing a show with The Linebreakers, it’s a different set of music gear. Laptop, spare laptop, cables, chargers, clothes, wash bag, passport, pack all the things, head to the airport - and that’s it. The logistics are all front-loaded; once that bag’s been checked in and I’m through security, we’re done; the rest of the week is hotel breakfast, catered lunches, restaurant dinners, housekeeping bringing fresh towels… the actual event itself is often hard work, ‘cos that’s what makes it worthwhile, but the rest runs on autopilot.

When it’s a hometown show, things run a little different. Tuesday is a workshop day, which means up at 6am, coffee, porridge - always porridge on workshop days ‘cos it keeps my brain running until lunchtime - and then on a train by 7:15, which gets me to Westminster by 8, which is an hour early, but if I leave it any later than that there’s a good chance the Jubilee line trains are all too full to get on. Twenty minutes to set up the laptop, screen, HDMI splitter, keyboard, check the WiFi, find out which websites are being blocked, figure out how to work around them. Workshop 9am-5pm - live coding exercises, demos, examples, Q&A. Wrap up, deal with the day’s admin and reply to emails, then another hour on trains to get home. Unpack, figure out what I need to take in with me on Wednesday, make dinner, do the laundry, sleep. The Linebreakers are playing PubConf on Friday night, so my plan is to take a case full of band gear into the conference venue each day - as much as I can comfortably carry on public transport - then on Friday shift the whole lot to the PubConf venue in a cab. Wednesday’s an electric guitar and 10kg of effects and cables shoved in my rucksack. Thursday, a 15kg flight case full of microphones and XLR cables. Friday, 25kg wheely case with the mic stands, second guitar, laptop stand, and all the other bits that make the show happen.

My actual talk is on Wednesday morning - and it’s a new one, so I’ve been up late the night before finessing animations and checking videos. New talks normally get run through a user group a month or so before the conference… but with NDC London always happening at the end of January, Christmas tends to get in the way a bit and there’s not a whole lot of meetups going on in the weeks leading up to it. The talk goes well, although the room’s a little emptier than I expected; maybe it’s just the general dip in conference numbers, maybe it’s just that people don’t want to learn about CSS? But I’m happy with it, and the folks in the room said very positive things.

Wednesday afternoon is band rehearsal - the downside of having a band who all live in different countries is that we don’t get a whole lot of rehearsal time, so this is where weeks of everybody learning their own parts and playing along to backing tracks finally gels into something approximating a show. It goes well. I skip the speaker dinner in favour of sleep. Sleep is good.

Thursday I have every intention of going to some talks. It doesn’t happen - the day disappears into a series of the sort of excellent conversations that only happen when the right people are all in the same room, catching up with folks from Particular, and Dometrain, and Twilio, and Umbraco, and old friends from the nerd circuit who I haven’t seen in way too long.

Thursday evening I’m running the NDC party. Now, I think the social events at conferences are just as much a part of the attendee experience as the talks, but the speakers who do the fun talks and comedy bits at the party? Most of them aren’t giving it the slightest thought until their “proper” session is out of the way - which is absolutely fair enough. This means the party inevitably coalesces out of chaos at about 3pm on Thursday - but you know, after more than a decade of doing this, we are damn good at coalescing things out of chaos. The party itself is a blast - Alan Smith’s Sonic Pi DJ set is fantastic, there are great comedy bits from Glenn Henriksen, Hannes Lowette, Amy Kapernick, Ash Bzak, Richard Campbell tells the story of Goliath, which I have seen half-a-dozen times and still makes me roar with laughter every time. I dust off my old “How To Succeed in the Enterprise” talk I wrote for PubConf a few years back, which goes down a storm, and we round out with karaoke, which works great in this kind of venue ‘cos you’ve already got a laptop, a huge projector screen, and a couple of handheld mics so there’s nothing much to pack up at the end of the night.

Home just after midnight (and stone cold sober, because Friday is another day). Dinner was Mexican food at the party (did I mention the food at NDC is always excellent?) but that was a long time ago, so second dinner is a chicken baguette from Greggs in Westminster station, which happens to be open and adjacent at the exact moment I realise I’m starving. Just in case any of you were reading this thinking that the international conference circuit was 100% glitz and glamour.

Friday morning, I take an hour out of conference time to get out on my bike; cycling the five-mile loop to the top of Crystal Palace Park every day this winter has been doing good things for my blood pressure and my brain - not to mention my knees - and as loath as I am to admit it, I actually miss the exercise after three days of conference. This is, of course, the day that I get a puncture right at the top of the hill. I figure it’ll be quicker to fix it and cycle than to wheel my bike all the way home, so I spend fifteen minutes in the cold changing the tyre; cycle home, shower, get dressed, grab my backpack and the 25kg wheely case and head into Westminster again.

Friday lunchtime is about the point when the folks speaking at PubConf finally start sending me their presentations, so on Friday afternoon all I need to do is transport 50kg of music gear to the pub, schlep it downstairs, figure out their PA system and projector set-up, plug it all in - which would be enough of a pain in the ass if it was on a bench in a well-lit workshop; doing it on the floor in a dimly-lit pub is definitely my least favourite part of the whole enterprise - while simultaneously building the PubConf deck as people send me Dropbox and OneDrive links to MP4 and PPTX files.

There’s always a list of things we think would be cool - let’s record the show! Let’s get video of the show! - that, by the time we’re set up and sound-checked, we’re already too knackered to think about… and then the audience starts to arrive, and the excitement starts to build, and the room fills up, and there’s a buzz, and I get the last few bits dropped into the presentation deck, and then… showtime.

With, inevitably, a handful of technical glitches. PowerPoint on macOS decides today’s the day it’s going to play Chris Ayers’ MP4 at three frames per second. It has never done this before. Ever. Sorry, Chris. Computer says meh. A reboot seems to clear it. The XVive wireless monitors that we use in the band, which worked flawlessly during soundcheck (and at every show we’ve done before) - tonight, channel 1 isn’t working. It worked earlier… maybe interference from 75 nerds with at least one WiFi-enabled device each packed into a basement that’s basically a giant Faraday cage? Channel 2 isn’t great either, but hey, we’ve got four more to try… cue some hastily improvised jokes about the history of UK broadcast television and how we don’t want to try Channel 5 in case it all comes out sounding like the Spice Girls - while we’re simultaneously channel-hopping our wireless units to find a channel that works; channel 3 saves the day. In the logistical chaos, I forgot one tiny bit of gear - the thing that clips my phone onto my mic stand, so I can adjust my own in-ear monitor mix during the show. Probably no big deal… right up until the point I realise during “Playing the Planning Poker” that I can’t hear the drum track, and I can’t fix it right away ‘cos it’s hard to get your phone out your pocket when both hands are busy playing the guitar… I make a mental note that it is, in fact, a Big Deal and I probably shouldn’t forget it again.

But the show is a triumph. The PubConf speakers - Chris Ayers, Brandon Minnick, Glenn Henriksen, Chris Simon - and the PubConf singers (yep, that’s a thing now) - Ash Bzak, Helvira Goma, Jo Minney, Damian Brady, David Whitney, William Brander, Jordan Miller, and Arthur Doler - absolutely smash it out of the park; the band plays a couple of new things we’ve not played in London before that go down really well, Rytis does a fantastic job running the sound mix on the venue’s slightly temperamental PA system, and Vagif gets the biggest applause of the night after dropping out of the gig because of a broken shoulder but then deciding to come along and play a couple of tunes anyway.

There’s a surprise, too… as we’re about to launch into “Enterprise Waterfall”, Hannes stops the show, Chris Ayers approaches the stage with a VERY LARGE BOX, and the gang presents me with the Lego Starship Enterprise - a little something to say thank you for Making The Crazy Things Happen. I’m lost for words. I actually cry a little. It’s perfect. It’s lovely. It’s completely unexpected. (It’s also another 6kg of luggage to take home at the end of the night… but hey, at this point, who’s counting? 🤣)

And then there’s music, and dancing, and laughter, and karaoke, and rye whisky, and single malt scotch and a very expensive cab home, and more scotch, and something approximating sleep, and then a day of bad TV and tactical naps and Chinese food and working on promo materials for the next thing I’m doing, and then a day of unpacking and repacking (and another puncture), and now, less than 72 hours after it all wrapped up, I’m on an aeroplane somewhere over the North Sea on my way to Stockholm, where there’s four inches of snow on the ground, a load of Java developers at one conference, and a load of .NET developers at another conference. The plan is two talks, two dinners, one live music and comedy show, say hello to as many people as I possibly can, give the stone lions a hug, pay my respects to Pub Anchor if I can find the time, and be home in time for the open mic at Ignition on Thursday night.

Should be fun.

The Road to Excel

2025-12-23T15:40:00+00:00

Chris Rea’s “The Road to Hell” is one of my favourite albums of all time. You probably remember the title track, but the rest of the album is fantastic - great tunes, great production, and some truly phenomenal guitar playing; Chris had a very distinctive hybrid slide playing style, something between Mark Knopfler and Ry Cooder, and it’s all over this record.

Now, back in 2021, when the news headlines was full of reports that COVID patient data was getting lost because of government departments using Excel as a data interchange format, I recorded a version of Chris Rea’s “The Road to Hell” about… well, about exactly that.

I never got the guitars quite right - did I mention that Chris was a phenomenally talented guitar player? After a week or two of of trying to figure out the guitar lines, recording and re-recording licks and solos and never quite getting it all to hang together quite right, I shelved it and it’s been sat in a Dropbox folder ever since.

With the sad news of Chris’ passing this week, I thought perhaps now would be a good time to share it. Not finished, no video, you can hear the rough edges - but here it is.

Rest in peace, Chris. Thank you for the music.

dylanbeattie · The Road to Excel

So You Want To Speak At Software Conferences?

2025-12-08T16:39:00+00:00

I run a .NET user group here in London, and we host a lot of talks from people who are relatively inexperienced presenters. Sometimes they’ve done presentations internally but never spoken before a public audience. Sometimes they’re developers who have been in theatre or played in bands; people with plenty of stage experience but who haven’t presented on technical topics before - and sometimes they’ve never done any kind of public presentations or performance at all. We aim to be a friendly, supportive crowd; public speaking can be daunting, and the first public outing of somebody’s first talk can be… let’s just say that the presenter sometimes learns a lot more than the audience, and leave it at that.

But it can also be a hugely rewarding experience, and as a seasoned tech presenter who’s been doing this for a while, aspiring speakers often ask me for advice on how to take it to the next level.

Before we get into the specifics, there are two things to bear in mind.

One: ask yourself why you want to do this. What does “the next level” mean for you? Are you looking to promote your consultancy, or your training courses, or your software products? Do you want to become a professional speaker and actually get paid to give talks? Are you doing it ‘cos you want to go places and meet people? Figure out what “success” looks like for you.

Two: be realistic about how much work is involved. It took me seven years to go from my first user group lightning talk, back 2008, to my first international conference. If you think you can hack together some code, write a talk about it, stick it on Sessionize and three months later you’re on your way to a major international event like NDC or Yow! or Devoxx… well, no. That’s not how this works. Strap in; it’s a long ride.

Year 1: Get Good

Write the talk. Write a talk nobody else could do; tell a story nobody else can tell. Figure out what your audience is going to learn, and why you’re the best person to teach them that. Then give it at local user group. It might go great. It might be a train wreck. Don’t worry. That’s one of the reasons user groups exist. Learn from the experience. Fix the demos. Fix the slides. If it was too short? Write some more. If it was too long? Cut something. Give it at another user group. Do it again. Do it again. Maybe write a second talk, shop that one around a couple of user groups too.

If you can’t find user groups, look on Meetup.com. Yes, it’s a horrible platform, but it works; search by topic, search by region, find groups that look like a good match for your content, and ask if they’re looking for speakers. They probably are.

Year 2: Get Seen

After user groups and meetups come the community conferences. Typically small, one-day events, with a few tracks, and usually free (or very cheap) to attend. For me, these were the DDD events _(that’s DDD as in Developers! Developers! Developers!, not to be confused with DDD as in Domain Driven Design), _a series of one-day free developer events around the UK, organised by volunteers, usually on a Saturday so people don’t have to take time off work. They bring in a good crowd, they’re a great way to get to know other presenters and people who are involved in tech events, and you’ll almost certainly meet a few people who are on the programme committees for the bigger conferences.

Events like this are your chance to get noticed. Turn up the day before, join the pre-conference dinner and drinks, introduce yourself. Yeah, it’s awkward when you don’t know anybody. There will be other people there who don’t know anybody and will appreciate you making the effort. Enjoy yourself, but don’t end up doing tequila shots in a karaoke bar at 3am. Not now. You’re there to give a talk, remember?

Go to the event. Spend the whole day there, do your talk, watch the other sessions. Communicate with the organisers. You don’t want their memorable impression of you to be a half-hour of panic and missed calls because one of their speakers has gone AWOL and nobody knows where they are.

Figure out how to keep in touch with the people you met. Join the Signal or WhatsApp group chat; if there isn’t one, create one. Follow them on LinkedIn, or Bluesky - be prepared to go where people are; don’t expect folks to join Mastodon just because that’s where you want to talk to them. That’s not how this works. If you really don’t want to play the social media game - and I can’t blame you - there’s always good old-fashioned email. A short email a week later saying “hey, thanks for having me” or “hey, I loved your session at DDD, let’s keep in touch” can pay off in a big way.

Finally, watch out for events that put video of their sessions online. Having a couple of YouTube links of you doing your thing in front of a live, appreciate audience can make all the difference when a programme committee is looking at a handful of talks and can only accept one of them.

Year 3: Get Accepted

You’ve got a couple of talks. You’ve delivered then enough times that you know they’re good *(and if they’re not good, make them good - or scrap them and write new ones)*. You know people. People know you. If somebody asks “hey, do we know anybody who could do a good session about $topic”, your name comes up. You’ve got a decent network of connections - group chats, LinkedIn, email addresses.

Now, find all the conferences in your field with an open Call for Papers (CfP), and get submitting. Dave Aronson over at codeasaur.us maintains a really useful list of CfPs which are closing soon. Check that regularly. Many events will cover your travel & hotel costs, although with sponsorship budgets drying up right across the industry that’s not as prevalent as it was a few years ago. If not, maybe you can persuade your employer to pay your travel - “hey, boss, if I can get a free ticket to this amazing conference with all these industry experts, do you think the company will pay my air fare & hotel?”

Lean on your network. What are people submitting to? Which events should you look out for? Which topics are getting a lot of traction (and which topics are not?)

Keep your content fresh. Write new talks. Keep giving them at user groups and community events.

Keep your submissions focused. 2-3 talks per event; don’t submit ten wildly different abstracts to the same conference in the hope one of them will get accepted. Every selection committee I’ve been on, if we see that, we assume the presenter hasn’t actually written *any* of them yet and is throwing everything they can think of into the mix and hoping one of them gets chosen. Not a great way to stand out. An open CFP at a big tech conference typically gets 20+ submissions for every available slot, which means if you reduce it to a numbers game, you’re submitting 20 talks for every one that gets accepted. Keep track of the numbers, and be objective about it.

Year 4: Get Bored.

It’s great fun doing this for a while… but it’s also exhausting. Some people hit it hard for a few years, do all the things, go to all the places, make a lot of great friends and happy memories, and then wake up one day and decide that’s enough. Some people do a few talks, tick it off their bucket list and decide that’s enough for them. Some settle into a gentle routine of 3-4 events they’ll do every year. And yes, some of us end up treating our calendars like a game of Tetris, juggling flights and trains and hotels and meetups and conferences and spending half the year on the road and the other half writing talks and workshops and all the other things it’s hard to do when you’re at the airport.

That’s why you gotta figure out ahead of time what “success” looks like. If you’re doing it for fun, remember to have fun - and if you find you’re not enjoying it any more? Stop. If you’re doing it as promotion or marketing? Track your leads. Make sure it’s actually generating the attention and the revenue it’s supposed to. If you’re doing it for money, be mercenary: no pay, no play. Not every event is the same, of course. In a given year I’ll have some events that are fun, some that are lucrative, some that are running alongside workshops or training engagements. Just make sure you know which is which.

Finally: respect your audience. Whether you’re talking to five people at a meetup, fifty at a community event, or five thousand at a huge international conference: those people are the reason you get to do this. They have given up their time - and often a substantial amount of money - to hear what you have to say. They deserve your best shot, every time. If you find you’re bored, fed up, tired, running talks on autopilot or making mistakes because you just don’t care? It’s time to try something else - and remember, there’s a thousand aspiring speakers out there who would dearly love to take that spot instead of you.

Now get out there. Work hard, have fun, teach us awesome things, and if you ever want me to look over an abstract or a slide deck, drop me a line - dylan@dylanbeattie.net. I’d be happy to help.

Fifteen Electronic Devices... Is That A LOT?

2025-12-02T13:35:00+00:00

Every airline’s cabin baggage regulations are subtly different. 10kg plus a personal item. 10kg INCLUDING a personal item. 8kg plus a personal item. 8kg, plus a personal item of up to 2kg. 8kg plus a laptop, umbrella, walking cane OR a personal item up to 2kg. 23kg as long as it’ll fit in the overhead locker (go British Airways!)

For a long while LOT, Poland’s national airline, would let you travel with “up to five dogs, cats, or ferrets, unless they are for commercial purposes”, which is wonderful because the collective noun for ferrets in English is “business”, as in “a business of ferrets”, so technically on LOT you could take a business of ferrets with you, but not a business ferret. Alas, this particular wording has disappeared in a recent update to their website, but I did notice a new regulation that passengers are allowed a maximum of fifteen personal electronic devices (PEDs), defined as anything containing a lithium-ion battery. You know, the ones that are in literally every single device which exists these days but which very occasionally go on fire so airlines are understandably a bit flummoxed about exactly what to do about them.

It’s not readily apparent from their website whether that’s fifteen PEDs per passenger total, or just in checked baggage (and I didn’t think you could put li-ion batteries in the hold at all, but apparently that’s just for actual spare batteries and powerbanks now, and doesn’t apply to devices containing a battery)… but fifteen’s a lot, right?

Well… actually, no. Not if you’re a massive nerd. Here’s my inventory for Build Stuff in Vilnius:

MacBook Pro laptop
Framework Windows laptop
iPhone
backup iPhone
Smartwatch
Powerbank
Samsung Galaxy tablet
XVive in-ear monitoring receiver
XVive in-ear monitoring transmitter
Wireless guitar transmitter
Wireless guitar receiver
Anker SoundCore noise-cancelling headphones
Sony C100 wireless earbuds
JBL portable Bluetooth speaker

That’s the standard inventory for any event where I’m doing a technical presentation and playing a show with Linebreakers - fourteen devices! Nice to know I could add a Kindle to that lot and still be within the regulations.

I note with some amusement that if I replaced the Sony earbuds with Airpods Pro or similar, that would technically be *three* PEDs, since each earbud has its own battery and there is a third battery in the charging case. So better not do that, then.

Is a 100% Discount The Same As “Free”?

2025-11-17T14:15:00+00:00

For the last year or two, I’ve been giving a talk at conferences around Europe about free software. It’s called “open source, open mind: the cost of free software”, there’s a couple of recordings of it up on Youtube if you want to check it out, and, like a lot of my conference talks, it’s a mix of history, case studies, and my own experiences working in tech.

One of the perennial challenges facing the craft of software development is that I think our industry has a tendency to attract people who like to create. I know dozens, probably hundreds, of people who became software developers because they enjoyed the activity of writing code, of sitting down and creating a solution to a problem. I don’t know anybody who got into tech because they enjoy reading code - most of us still don’t; reviewing code is way less fun than writing it. This tendency for software developers to ignore prior art because, well, gosh, it’s great fun reinventing the wheel over and over again - it’s one of the reasons I spend so much time talking about computer history. A few years back somebody was very excitedly telling me about something called a microSPA; a SPA is a single-page application, and a microSPA is a very small single-page application; an individually deployable unit containing HTML, JavaScript, CSS… that’s right, they’d reinvented the web page. And it only took twenty years.

I think it’s important to talk about the history of free software because I keep seeing the same patterns playing out, over and over again - and every time they do, somebody ends up burning time and brain cycles on solving a problem that we already solved, and I wonder what new and fascinating things they could have created instead.

I want to talk about two of these patterns in this video. First - let me know if this sounds familiar to you: somebody creates some free software. Free, as in, it doesn’t cost any money. It’s out there on the internet and you can download it and use it and even incorporate it into your work projects - projects that make your company money. And you do, whether that’s by installing a package from NuGet or npm, or connecting to some sort of online API, or cloning their github repo and building it yourself.

What you don’t see is the flipside of that. The person who created that library? They’re doing - and sharing - that work because at that point in their own life and career, that makes sense to them. Richard Stallman’s Free Software Foundation has popularised the idea that giving away your source code is some sort of lifelong ideological commitment, a fundamental part of somebody’s identity - but really, that’s not how any of this works. What normally happens is somebody’s working in some kind of agency or consultancy, they create a library which gets used internally across multiple projects for different clients, there’s a conversation about making the source code available - which, at that point, makes a lot of sense; clients get that code under a public license, it gives the agency something good to say to candidates when they’re recruiting, and maybe they’ll even get community contributions that add features or fix bugs. Everybody wins.

Then something changes. Somebody leaves. The agency gets acquired. For whatever reason, the people that used to maintain that project aren’t in a position to do that for free any more - and so the logical thing to do is what a lot of open source projects and packages have done in the last few years: to say to the consumers, the people and companies who are using that code to make money, “hey, in order for me to keep doing this, I’m going to need to get paid somehow. Seeing as donations haven’t worked, and consulting has pulled me away from working on it, I’m going to try charging for the software itself, so the next version won’t be free”.

This usually splits the community in two. In one camp are the ‘keyboard warriors’, the folks who like to express their outrage on social media; in the other camp are the people who understand software licensing, budgets, compliance - and the cost of replacing or rewriting something which already works. There may be some overlap, but from talking with maintainers who have done exactly this, most of the big corporate users will quietly sign up without making a lot of noise about it, and the people on Twitter and Reddit shouting about betrayal and open-source “rug pulls”… well, they were never going to pay for anything anyway.

There’s another factor in all this, of course. I think it’s absolutely right that companies that get substantial financial benefit from the use of free and open source software should contribute to the sustainability of those projects. I recently found a thread on X which linked to a bug filed on ffmpeg’s issue tracker in May 2023, which included this absolute barnstormer of a comment:

Hi, This is a high priority ticket and the FFmpeg version is currently used in a highly visible product in Microsoft. We have customers experience issues with Caption during Teams Live Event. Please help,

Microsoft. Microsoft Teams. It would be nice to think that Microsoft Teams could come up with a more constructive way of engaging with the maintainers of ffmpeg than expecting a request for free support to be treated as a ‘high priority ticket’, but unfortunately this sort of interaction between large corporations and small open-source project teams isn’t remotely unusual.

We’re not all Microsoft, though… and that’s where the one-size-fits-all model breaks down pretty quickly. The sort of licensing fees that are a drop in the ocean to big tech companies could be a dealbreaker for a startup that’s just getting off the ground - and for projects which are themselves free-as-in-free-beer, for student and hobby projects; those projects are never going to be able to justify hundreds or thousands of dollars in license fees.

The problem is that the alternative to paying for a license tends to be “oh well, if we can’t use that one maybe we can just build our own”… which is rarely a good idea in terms of the quality of the software, but also, if you’re at an early stage startup or a small-to-medium business, you should be focusing on your mission, working on the features that are strategic differentiators for your product or service, not spending time building authentication and telemetry and data persistence and all the other stuff that’s already been solved over and over again.

Now, none of this is a new idea. The first project I know of that switched from a completely free license to something more restrictive was ServiceStack, which switched in 2014 from a BSD license to a hybrid commercial license. Their approach was interesting: free licenses for genuine solo projects, defined as anything where a single person was the sole author, contributor and copyright holder of all the code in that project. They offered free licenses for projects that in turn published their own code under a compatible open source license - and if you didn’t qualify for a free license? $299 per developer for small organisations - fewer than ten employees; $999 for large organisations. That was, and is, a per-developer, one-time, perpetual license - you get a year of maintenance and upgrades, and any new releases during that maintenance period, you can run those releases forever.

The company I was working for had half-a-dozen developers working with ServiceStack at the time, but we had over fifty employees, so for us upgrading from the BSD-licensed ServiceStack 3 to the new licensing model would have cost six thousand dollars. The budget meeting about that one didn’t get very far - and I get it; although we were using APIs for all kinds of internal stuff, they weren’t a key part of anything that actually generated revenue, and I didn’t know back then about accounting models like Cost of Delay, which might have given us a way to calculate the value of the time we saved by sticking with it instead of switching to a different framework or building our own.

ServiceStack is a really useful case study, because they did this ten years ago. With a whole load of high-profile projects switching their licensing over the last few years - IdentityServer, Redis, Terraform, and more recently AutoMapper and Mass Transit - there’s been a very predictable reaction; community backlash, a fork of the last version of that product that was published under an open license, and a lot of folks asking what happens next? Well, ServiceStack has gone from strength to strength, shipping a steady stream of new versions and features. It’s a solid, sustainable, commercially viable product. The community fork of the last BSD codebase? That was NServiceKit, which was very exciting for about six months before enthusiasm ran out. It’s now been a decade since the last commit. I assume because its creators realised what every fork will find out sooner or later: if you fork a free project because it’s going commercial, all the people who are happy to pay for licenses, support and maintenance will go with the upstream project, everybody who expects support and maintenance to be free will beat a path to your door, and you’ll be expected to maintain feature parity with the project you forked and do it all for free out of the goodness of your heart.

Now, one thing I think ServiceStack got wrong was to offer free licenses for small projects, defined as projects with fewer than 10 defined service operations, fewer than 10 mapped database tables, and a limit of 6000 requests per hour on their Redis client when running without a license, those kinds of restrictions.

I applaud the objective here - hey, use our software while you’re getting started, and pay for it later once you need the extra scale. But I think those kinds of limitations in any kind of software aren’t a great idea, because they put a financial roadblock in the way of what should be an architectural decision. Imagine you’re working on a project, you’ve got a team of five developers building a new feature and turns out the best way to deliver that feature is going to require mapping one more database table - except, oh crap, we already used our ten free tables. That eleventh table is going to cost you five thousand bucks - and meanwhile, if a billion-dollar Fortune 500 company is only using nine database tables, they’ll still qualify for free licensing.

There’s a bunch of different ways that other projects have approached this:.

Duende’s Identity Server, an open source authentication framework for .NET, has a pricing model based on the number of client IDs you’re running - there’s a startup license that’s fifteen hundred dollars a year for up to five client IDs, then five hundred a year for each additional client ID up to fifteen, where it switches to the business license, nine thousand dollars for fifteen IDs, everything in the startup license, plus some additional features like automatic key management. They also have a community edition that’s free for companies with under $1 million of revenue and under $3M of capital.

Chris Patterson also seems to be going with something like this for MassTransit, with a free tier for organizations and nonprofits with up to $1 million of revenue or expenses

Just so we’re clear, a million dollars sounds like a lot of money - and yeah, to the vast majority of the people living on this planet, it’s a life-changing amount of money - but when you’re a tech start-up, a million bucks really doesn’t go very far. Hire some engineers, bring in the hardware and infrastructure for them to do their jobs, pay your lawyers, your insurance, your rent … and it’s gone.

I think this is why we’re seeing some projects raising that a bit higher. For example, Jimmy Bogard set the level for his community edition of AutoMapper to be free for companies up to $5 million of revenue and $10 million of capital.

I think the original version of these various free offerings was Microsoft’s BizSpark programme way back in 2008, which provided free and heavily discounted access to Microsoft developer tools and Azure hosting to startups for the first two years of operation - the idea being, I assume, that within two years either the startup is generating enough revenue to pay for those services, or it’s done what most startups do and quietly shut down because it ran out of money; that programme has evolved over the years and today is called Microsoft for Startups, and - this being 2025 - offers free AI and Azure credits to companies which are just starting out.

Particular also just put out their version of this for NServiceBus, using the more generic term of “financials” to cover revenue, capital, and expenses. They set up this “sliding scale” which starts with the same 100% discount for organizations up to a million USD of financials, just like Duende and MassTransit; but when you go past that, you don’t start paying full price right away. Instead, the discount goes down just a bit to 90%, and passing two million it’s reduced to 80%, at three million it’s 60%, at four million it’s 20%, and only when you pass five million dollars you pay full price.

Now, full disclosure: I know a lot of the folks at Particular, we’ve collaborated on a few things over the years, they gave me a heads-up that they’d be announcing this, and they know what’s in this post ‘cos I sent them the draft before I published it, so this isn’t an unfiltered reaction piece… but in another way, it kinda is, because I think this is fantastic.

First of all, it’s an interesting new take on what everybody else has been doing. Second, it solves a real problem for growing companies - you’re not going to suddenly find out one day that your last quarter accounts have pushed you over a threshold and you’ve gone straight from zero to full cost. And third, for existing small businesses using NServiceBus, they’re about to get a nice discount on their NServiceBus bill.

Another detail I think is worth mentioning was that all of these projects excluded “priority” support from their free and discounted offerings - which I think is entirely justified. Good support takes time and expertise, meaning that it can usually only be done by the project maintainers - the very people whose time is stretched thin and are struggling to get to financial sustainability. That’s why I don’t think anybody who’s getting the benefits of fully-discounted licensing should expect to get the same degree of support as the organisations that are paying full price.

Point is, there are organisations with money - whether that’s revenue, assets, or investment - and not just the big brands, but the countless mid-sized banks, insurance companies, and even government entities. They can, and they should, be paying for the stuff they use. These are the last people who are going to complain about “rug pulls”. Professional developers in these contexts clearly explain to their bosses and clients, “so, this thing used to be free, and we used it a bunch, and from the next version it will cost this much. Replacing it with something else that’s free will take roughly this many hours. What do you want us to do?” A rational boss, or client, will tend to accept a reasonable license cost rather than incur the risk of a rip-and-replace; as Nick Chapsas said in a recent video:

You sort of should assume you take a risk by just using something that’s free out there. There’s no contract [covering] … what’s going to happen in the future.

But for all of those organisations that don’t have that kind of money; the startups, student projects, open source, small independent agencies, not to mention the large parts of the world where local market economies operate on an entirely different scale - these discounts can be just what they need to continue to build upon solid foundations, meaning they wouldn’t need to change anything when the open-source projects they use commercialize - so no “rug pull” here either.

A bunch of us have been talking about how we can take these things forward and enable more open-source projects to achieve sustainability. We’ve seen how standard OSS licenses like MIT and BSD have helped accelerate legal acceptance of open-source software, so we think some kind of standardization around commercial terms and discounts could be helpful. What exactly that would look like is still being discussed, but my hope is that by putting out this video describing some of these approaches, the next OSS maintainer will think to themselves, “yeah, I can do this too”.

One final thing I’d like to add: I see occasional online comments about projects “going closed source”… well, that’s just not true. Every single one of the projects and companies I’ve mentioned in this video remains committed to sharing the full source code for all their products, so that developers who are relying on that code can download it, build it, step through it in the debugger if they need to - that’s a big part of “open source”; the ability to obtain the source code, study it, modify it if you need to, and even maintain your own fork of it if you don’t want to rely on the original vendor. All we’re saying here is that if you’re using these projects in a commercially significant context, you should expect to pay something towards their maintenance and future development - and if some sort of scalable discount model makes it easier to reach that point, I think that’s a good thing for everybody involved.

Choosing the Right Path

2025-09-02T08:56:00+00:00

Srijan Singh pinged me on LinkedIn earlier today with a question:

I’m in my early career as a backend dev at IBM, and something I often wonder about is choosing the right path: whether to stick with a big company and learn deeply, or test myself in a startup environment (remote vs in-person adds to the mix too). Since you’ve navigated such diverse experiences, I’d love to hear how you’d approach that decision early on.

I quite enjoy getting these kinds of questions, but as Scott Hanselman wrote way back in 2010:

If you email someone one on one, you’re reaching that one person. If you blog about it, you get the message out on the web itself and your keystrokes travel farther and reach more people. Assuming you want your message to reach as many people as possible, blog it. You only have so many hours in the day.

So here’s my answer to Srijan’s question, as a blog post.

Let’s start with the simple answer, which is how I did approach that decision early on… I didn’t, really. I was lucky enough that I didn’t really have to think about it. I loved working with computers, writing code, and building websites, and if I was getting paid money to do that I was happy. I landed my first graduate role in 2000, via one of my housemates who was working as a recruiter; three years later one of our clients, Spotlight, asked me to work for them full-time; fifteen years after that, I got head-hunted by Skills Matter to become their CTO; they went bankrupt in 2019, I set up my own company, Ursatile, and I’ve been independent ever since. To put this another way: I’ve had exactly two job interviews in my life, and one of those was to stack shelves in a grocery store when I was sixteen years old.

That probably doesn’t help, right? 😆

Looking back over 25 years of working in software, though, I think there are three things that matter:

The problem. Are you helping people? Are you making somebody’s day better? Are you building something that’ll make your end users happier? Or is your entire working life about making the CEO’s stock go up another point?
The people. Who are you working with? Are they competent? Are you learning from them? Do you enjoy their company? Do they respect your ideas?
The process. How does stuff actually get done? Everything from prioritisation and planning to what text editor you use. I enjoy working with some technology because it makes it easy for me do good work. I do not enjoy working with some other technology, because it makes it hard for me to do good work.

You don’t need all three. I’ve had bits of my career when I’ve been working with awesome groups of smart, kind people, using great tech to solve cool problems. I’ve also had bits of my career when I’ve been using painful tech to solve pointless problems, but the people made it worthwhile - and, occasionally, periods where I honestly didn’t care for the people or the problem, but was learning so much along the way that I probably stuck around longer than I should.

Think about longevity. You’re building solutions, relationships, and skills. Will the things you’re building last for a decade, or will they be obsolete by the end of the year? Can you take those things with you out into the wider world, or do they only have value in the context of your current role?

Remember to take stock once in a while. You might find the job that used to tick all three boxes is down to two, and then one… and when you’re down to one, you’re in danger, ‘cos if that one disappears too… well, congratulations. You’re stuck using tech you don’t enjoy to solve problems you don’t care about with people you don’t like.

Now, obviously there’s a huge amount of context to consider here. The tech jobs market is the worst I’ve ever seen; more of my dev friends are out of work right now than at any point since the first dotcom bubble, 25 years ago. If you’re supporting a family or paying off debts and you have a steady paycheque, maybe just keep your head down, keep doing what you do, wait for things to pick up. But if you’re early in your career, you’ve got a bit of financial security and you’re unhappy with how you’re spending your working days? Do something different. Bored of working in the enterprise? Find a startup. If you’re burning out working in a startup, get a comfortable, predictable job in a big company. If you’re remote, and tired of being at home all the time, find a job with a nice office; if the commute is making you miserable, try remote. When you try something, give it at least a year - but if you’ve lasted a summer and a winter, learned the ropes, learned the industry, and you’re still miserable? Probably time to move on.

Really, though, the important question when it comes to career progression isn’t “where do I see myself in five years”, it’s “am I going to quit my job today?” That’s the question that actually makes things happen. If the answer’s “hell, no, I’m happy”? Awesome. Go to work. If the answer’s “yes, today’s the day!” - well, good luck with it.

But if the answer’s “I don’t know”? Well, time to gather some more data.

On The Road Again...

2025-08-20T13:00:00+00:00

I’ve spent most of the summer so far writing a new course; if you follow me over on BlueSky - and you should - then you’ll have noticed I’ve been posting - skeeting? bleeting? I still don’t know what to call posting on BlueSky. Let’s go with bleeting. I’ve been bleeting about CSS a lot, because my new course is CSS for Software Engineers. The video course is going to be available on Dometrain later this year, I’ll be running it as a workshop at various conferences, and I have a funny feeling there might even be a book at the end of it… but that’s taking up pretty much all my time, not to mention brain capacity, and hasn’t left a whole lot of time for YouTube videos and stuff.

But, it’s nearly September (yeah, really), which means it’s nearly conference season again, and I’m going to be bouncing around Europe for the next few months talking about all kinds of things - machine learning, text encoding, Rockstar, open source, and, yep, CSS.

Now, here’s the thing. When folks like me get invited to speak at these kinds of events, the organisers really appreciate it when we make a bit of noise about it, post it on social media, blog about it, make videos… and I get it; marketing is tough, every bit of coverage might help shift a few more tickets, and the algorithms that drive social media apparently love short form portrait videos… but, y’know, I’m forty seven years old, I listen to Bon Jovi and I still don’t really understand what TikTok is for; somebody like me making a video on my phone about a .NET developer conference and hoping it’ll go viral is… I believe they call it “cringe”.

But then I remember a few years ago I was in Tartu, in Estonia, where I was giving the keynote at digit.dev, and somebody recognised me in the street from my YouTube videos and we got chatting… turns out they’re a developer, they love code, they really liked my conference talks, and they had no idea that I was the keynote speaker at a multitrack international conference taking place right in the city where they live. And Tartu’s not a big place; it’s got about 100,000 people.

So here’s the deal. I’m going to tell you where I’m going, on the off-chance that you’re local, and you have some training budget left, because it would really suck if there was an awesome event happening right there in your home town and you didn’t know about it.

If you want a whole “ten reasons you can’t afford to miss OpenTechDevConDays 2025”… well, nah. Not really my jam. They have marketing people for that.

For what it’s worth, I do try to make sure that every event I’m involved with is excellent; I put a huge amount of work into preparing talks and workshops, I bring stickers to give away, and I try to make sure I have as much time as I can to meet people, hang out and chat about what you’re all working on.

So here’s where I’m going to be. I’m obviously not expecting any of you to come along to all of these - although I’ll be very impressed if you do - but if I’m going to be in your neighbourhood, and if you can persuade your company to buy a ticket, then come along and say hi.

I did mention I’ll be giving away Rockstar stickers, right?

September 8-12th I’ll be at NDC Copenhagen, where I’m running a brand new workshop about all the amazing things modern CSS can do. I’ll be giving a talk about algorithms and why they’re not as scary as they look, and probably eating quite a lot of barbecue at Warpigs.

September 25th I’ll be Edinburgh at ScotSoft 2025 talking about plain text: weird and wonderful stories about teletype machines, Hungarian Scrabble, how emojis actually work, and what “PIKE MATCHBOX” has to do with driving in the Soviet Union.

October 3rd I’ll be in Dordrecht in the Netherlands with Fronteers; that one’s a one-track evening conference, so should be a lot of fun; we haven’t finalised all the details for that one yet, but keep an eye on https://www.fronteers.nl/ for more news.

October 9th + 10th, I’ll be in Riga for Zabbix Summit. Zabbix is an open source observability and monitoring platform, so the event’s all about infrastructure, automation and devops; I’m going to be talking about the history and future of free software and how we might be able to create a truly sustainable model for open source development.

Then I’m in Portugal for two weeks; October 13-16 is the Azure Developer Summit in Lisbon, a brand new event organised by Microsoft in partnership with NDC and Techorama, all about Azure, AI, .NET, C# - and then 20-24 October it’s NDC Porto, which has a different format this year; instead of one-hour conference sessions it’s all hands-on workshops. I’m doing two days about modern CSS, plus a session about building a ray tracer in JavaScript, and another one about how to build your own programming language using C# and .NET and parsing expression grammars.

Then I’m home for two days, then a Eurostar to Rotterdam and on to Utrecht, where by a happy coincidence Techorama is in town the same week as Tweakers Summit. ***Monday 27th October*** I’ll be at Techorama with a one-day Introduction to Distributed Systems with .NET workshop, Tuesday 27th I’ll at Techorama in the morning and then heading to DeFabrique to close the Tweakers Summit, then I’m back at Techorama on Wednesday.

And then November 5-7 I’m at Øredev in Malmö, where I’ll be telling the story of how I built a Rockstar interpreter in web assembly using C#, and rocking the party stage with The Linebreakers.

Oh, and if you’re thinking “wow, that’s not nearly enough travelling, this guy isn’t even trying any more”, I’m also going to the Euroblast festival in Cologne at the end of September, and heading to Yorkshire for the Whitby Goth Weekend at the end of October. Not performing or anything, just going along to watch bands and have fun. You know. Like a normal person.

So that’s, what, nine tech events - and two music festivals - in seven countries in two months. It’s going to be . Come along and say hi.

Right now, though, I gotta get back to writing about how CSS flexbox actually works, which turns out to involve way more mathematics than you might think.

CSS Has 239 Different Ways To Make Something Blue

2025-07-30T09:15:00+00:00

I’m making a new video course for Dometrain at the moment, and it’s all about CSS - one of the three pillars of the open web, along with HTML and JavaScript. I love CSS, I’ve been working with it literally since it was invented, but I absolutely understand why so many developers don’t enjoy working with it.

CSS has had to incorporate conventions and standards from a wider range of disciplines than any other mainstream technology. Modern CSS incorporates ideas from mechanical typesetting dating back centuries, conventions around information design from hundreds of years of printing and publishing, fifty years of innovations from digital publishing and computer graphics, twenty years of getting stuff wrong on the web while we were still figuring out how to get it right - right up to things like how to account for the “dynamic island” on the latest iPhone handsets.

To say it’s accumulated a few idiosyncrasies along the way would be an understatement. Yesterday, while putting together the module about how the various colours models work in CSS, I found myself wondering how many different ways there are in modern CSS to give a box a blue background.

I got to 239. Two hundred and thirty nine different ways to say “the box is blue”.

You’ve probably heard of named colours (color: blue;), and hex codes (color: #0000ff). Hex codes can also be written as three digits (#00f), four digits (#00ff), and eight digits (#0000ffff).

Then there’s the rgb() function, which can take each colour component as a decimal or as a percentage. There’s hsl(), which takes hue, saturation, and lightness, along with a bunch of new colour models - hwb(), lab(), lch, oklab() and oklch() - all of which also take a hue component.

Except hue in CSS is an angle - the colour’s position on a colour wheel - and CSS already has a unit system in place for angles, that’s used for rotations and transformation, so the colour models just reuse that. Which means you can write the hue component as degrees (with or without deg), radians, gradians (the cursed unit. 400 grads in a circle. Why does it even exist? I don’t know.), or turns. So if any colour specification involves hue, there are five different ways to write it.

Add in two different ways to specify alpha transparency - / 100% and / 1.0 - and you end up with well over two hundred different ways to say “the box is blue”… and this is just using the modern CSS colour syntax; we’re not even getting into the legacy rgb() and hsl() function syntax, their aliases rgba() and hsla(), or any of the wonderful things you can do with calc() and relative colours.

OK, let’s be fair here. If the language designers hadn’t reused CSS angular units for hue, we’d be looking at more like 40, maybe 50 syntax variants. Pick a lane for how you want to specify transparency, you’re down to 20 or so. You’re very unlikely to use the LAB or LCH colour models in production unless you know exactly what you’re doing, so that’ll chop it down further; in reality, you’re not going to encounter more than handful of these variants, and the vast majority of sites out there just use hex codes for everything.

But, y’know, if you’ve got the whole chaotic evil vibe going on, why not spec all your colours as oklch, lightness and chroma as arbitrary decimals, hue in grads, and give everything an alpha channel that’s randomly a percentage or a decimal, even for colours which are fully opaque?

I think you’ll all agree that color: blue; looks kinda dumb, but color: oklch(0.452 0.31 292grad / 100%); is clearly the work of a genius.

Check out the whole list over at https://dylanbeattie.net/miscellany/blues.html

Naming Things is Hard... Renaming Things is Harder

2025-07-26T15:49:00+00:00

You know those little things that you just sort of ignore, over over and over again, until finally one day the planets align and you’re like “No! I’m not putting up with this any more! There must be a way to fix it!”

Welcome to my cameras. I have two Canon EOS M200 cameras, which I use for streaming, recording training videos, teaching online workshops, all kinds of stuff. One is black, one is white, they are otherwise identical. They’re both connected to my main PC using Elgato Cam Link 4K HDMI capture devices, which for the remainder of this article I will call camlinks because they don’t have a better generic name.

They’re also connected using micro-USB, which means I can use Canon’s Remote Shooting utility to control things like the F-stop, white balance, and ISO. The cameras themselves are permanently mounted around my desk — one is above my main screen, the other one’s built in to a teleprompter — so it’s kinda hard to fiddle with the settings menu.

The problem is that when I go to fire up the EOS utility, it asks me to pick which camera I’m using… am I using the Canon EOS M200, or the Canon EOS M200? And then there’s the fact that both of the dongles show up in Windows as “Cam Link 4K”, which means setting up video sources in programs like OBS Studio normally involves picking the wrong camera at least once.

Well, this morning I had that fateful nerd thought… “I should fix this. How hard can it be?” Well, grab your razor and strap in, friends… we’re going yak shaving! Now, most of the time, if you want to rename a thing in Windows, you right-click it, and choose “Rename”, and give it a new name.

That doesn’t work in Device Manager. I mean, sure, rename a file, but why would anybody ever need to rename a camera? OK, no big deal. Those names have got to come from somewhere. Probably the registry. So I fire up regedit, Ctrl-F, “Cam Link 4K”, hit “Find Next”:

And I wait, and wait, and wait a bit, and then I get bored because this is a Ryzen 9950X3D with 128Gb of RAM and I didn’t spend that kind of money so I could sit around and wait for things, dammit!

I tried writing a Powershell script to do the same thing — trawl the registry, look at all the keys and subkeys and entries and values — but it did that infuriating thing where it takes so long to not do anything that it wasn’t clear after 30 seconds whether the program worked, but hadn’t matched anything, or I’d screwed up the matching logic.

Ok, much better approach: let’s dump the entire system registry to a text file and then I can edit it properly. File, Export, registry.txt, takes about a second. Done.

I now have a 540Mb text file full of bits of Windows registry. It looks like this:

[HKEY_LOCAL_MACHINE\SYSTEM\DriverDatabase\DriverPackages\termbus.inf_amd64_7ccf415b3c0cf753\Descriptors\TI_COMPAT_DEVICE]
"Configuration"="TS_INPT_DEVICE.NT"
"Manufacturer"="%msft%"
"Description"="%ts_inpt_device.devicedesc%"

[HKEY_LOCAL_MACHINE\SYSTEM\DriverDatabase\DriverPackages\termbus.inf_amd64_7ccf415b3c0cf753\Descriptors\TS_BUS]

[HKEY_LOCAL_MACHINE\SYSTEM\DriverDatabase\DriverPackages\termbus.inf_amd64_7ccf415b3c0cf753\Descriptors\TS_BUS\TS_INPT]
"Configuration"="TS_INPT_BUS.NT"
"Manufacturer"="%msft%"
"Description"="%ts_inpt_bus.devicedesc%"

[HKEY_LOCAL_MACHINE\SYSTEM\DriverDatabase\DriverPackages\termbus.inf_amd64_7ccf415b3c0cf753\Strings]
"msft"="Microsoft"
"ts_inpt_device.devicedesc"="Remote Desktop Input Device"
"ts_inpt_bus.devicedesc"="Remote Desktop Input Bus Enumerator"

[HKEY_LOCAL_MACHINE\SYSTEM\DriverDatabase\DriverPackages\termkbd.inf_amd64_b0e97bc9e5ad246d]
"Version"=hex:ff,ff,09,00,00,00,00,00,6b,e9,36,4d,25,e3,ce,11,bf,c1,08,00,2b,\
  e1,03,18,00,c0,17,1d,14,c2,c9,01,7e,04,f4,65,00,00,0a,00,00,00,00,00,00,00,\
  00,00
"Provider"="Microsoft"
"SignerScore"=dword:0d000003
"FileSize"=hex(b):70,34,01,00,00,00,00,00
"StatusFlags"=dword:00000112
@="termkbd.inf"

Each chunk is in this format:

[HKEY_LOCAL_MACHINE\SYSTEM\Path\To\Registry\Key]
"Name"="String Value"
"AnotherName"=dword:12345678
"ThirdName"=hex(b):aa,bb,cc,dd,ee,ff,11,22

So I want to find every chunk that contains any entry whose value contains "Cam Link 4K" or "Canon EOS M200", and then extract:

The first row of that chunk, which is the path to the registry key I need to edit
The entry itself

Then — in theory — I’ve got a handful of registry entries in a file, which I can edit in VS Code or something, import it into regedit, and presto! rename all the things.

First idea: pull it into VS Code, replace all the \n with a marker __EOL__ so I get every block on its own line, delete all the lines that don’t contain "Cam Link 4K" or "Canon EOS M200", turn all the __EOL__ back into \n, and then do the rest by hand.

Yeah… VS Code won’t do that.

TextPad wouldn’t do it either…

But it's OK. We solved this problem. We solved it BEFORE I WAS BORN. The beardy wizard people who created Unix knew all about editing files that wouldn't fit in memory, because they built Unix for computers that had an ENTIRE MEGABYTE of RAM... which you had to share with the rest of the university.
— Dylan Beattie (@dylanbeatt.ie) 26 July 2025 at 10:42

Yep, this is a job for awk. You probably don’t know what awk is. Awk is a stream-based editing utility originally created for Unix. So I asked Copilot to write me an awk script. (what, you think I can remember how to write awk? University was a long time ago, friends…)

BEGIN {
	RS = "\n\n";   # Blank lines separate records (blocks)
	ORS = "\n\n";  # Output blocks separated by two newlines
}

{
	if ($0 ~ /Cam Link 4K || $0 ~ /Canon EOS M200/) {
		n = split($0, lines, "\n")
		output = lines[1]
		for (i = 2; i <= n; i++) {
			if (lines[i] ~ /Cam Link 4K/ || lines[i] ~ /Canon EOS M200/) {
				output = output "\n" lines[i]
			}
		}
		print output
	}
}

Yeah, that’s what awk looks like. Remember, this is the language that Larry Wall looked at, went “well, golly,I can do better than that”, and… invented Perl.

Of course, it didn’t work. Me & my little electric copilot buddy had missed two rather significant details… first, the RS record separator? We’re trawling a file created by Windows regedit. It’s using \r\n, not \n. Easy fix; change `RS = “\r?\n\r?\n” `(and look, now it’ll work cross-platform if I ever need to awk the Windows registry on macOS!)

Still doesn’t work… because the Windows Registry Editor’s Export feature creates text files that are encoded as UTF16-LE, and awk don’t do UTF16. So I use VS Code to save the registry file as UTF-8, and off we go…

gawk -f filter.awk registry.txt > devices.txt

It works! devices.txt now has a little registry snippet for every single chunk of registry that includes "Canon EOS M200" or "Cam Link 4K"…

"Canon EOS M200" appears 25 times in the registry. "Cam Link 4K" appears 60 times - and the exact entry "FriendlyName"="Cam Link 4K" accounts for 41 of those... crikey. That's a lot of times.
— Dylan Beattie (@dylanbeatt.ie) 26 July 2025 at 11:22

OK, let’s figure out which one goes where.

I am assuming at this point that nothing in Windows is stupid enough to actually open, connect, etc. devices based on a field called FriendlyName. If I’m wrong, things are about to get extremely hilarious indeed.

So I use Textpad’s really handy “sequence replace” feature, that’ll let you use a regex to find something and then include an incrementing sequence number in the replacement expression:

The result is a registry file where every FriendlyName value is now unique - so I can in theory reboot, see which names appear in which drop-down menus and dialogs, and then edit them accordingly. In theory.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Portable Devices\Devices\USB#VID_04A9&PID_32EF#9&39F7FE61&0&3]
"FriendlyName"="Canon EOS M200 UNIQUE12"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Portable Devices\Devices\USB#VID_04A9&PID_32EF#A&D9BD236&0&3]
"FriendlyName"="Canon EOS M200 UNIQUE13"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{eec5ad98-8080-425f-922a-dabf3de3f69a}\0005]
"FriendlyName"="Canon EOS M200 UNIQUE14"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{eec5ad98-8080-425f-922a-dabf3de3f69a}\0009]
"FriendlyName"="Canon EOS M200 UNIQUE15"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\USB\VID_04A9&PID_32EF\9&39f7fe61&0&3]
"FriendlyName"="Canon EOS M200 UNIQUE38"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\USB\VID_04A9&PID_32EF\a&d9bd236&0&3]
"FriendlyName"="Canon EOS M200 UNIQUE39"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\USB\VID_0FD9&PID_0066&MI_00\a&1b1e3ad0&0&0000]
"FriendlyName"="Cam Link 4K UNIQUE40"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\USB\VID_0FD9&PID_0066&MI_03\a&1b1e3ad0&0&0003]
"FriendlyName"="Cam Link 4K UNIQUE41"

Add the first line to the file otherwise regedit rejects it:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Portable Devices\Devices\USB#VID_04A9&PID_32EF#9&39F7FE61&0&3]
"FriendlyName"="Canon EOS M200 UNIQUE12"

...and so on...

Import…

Reboot… IT WORKED!

Well, it worked mostly. At first, the Canon EOS Utility Launcher didn’t pick it up at all, and it turns out some of the Canon EOS utilities read the DeviceDesc value from the registry, not FriendlyName, so I edited that one too… at this point I got into a good half-hour of editing something, rebooting, seeing what had changed, edit again, reboot again, turn it off and on again… and quite a few moments where I thought I’d tried everything I could think of, including a reboot, and it still hadn’t worked…

And then, one final, glorious reboot, and there it was.

And you know what?

I will bet money, good, solid, chunky cash dollar money, that at some point, somebody at Canon said “er… boss, what happens if somebody buys two of the same camera, and has them both plugged in at the same time over USB?” and somebody’s boss said words to the effect of “stop derailing the planning meeting with your stupid edge cases, Chris, we have work to do and you’re not helping.”

I see you, Chris.

I see you, and I salute you. 🫡

Oh, and Microsoft: if we could get right-click, Rename… in the Device Manager? That’d be, like, just swell.

The Subtle Art Of Deprecating API Endpoints

2025-07-16T13:46:00+00:00

I had an app fail in production the other day. Not seriously - only affected a couple of admin screens - but it failed because Hubspot had deprecated some of their API endpoints. (That’s nerd speak for “we were using a thing and they turned it off.”)

https://developers.hubspot.com/changelog/breaking-change-removed-support-for-referencing-custom-object-types-by-base-name

Sure, they announced in October 2024 that this particular endpoint was being deprecated. Only problem is… I didn’t see the announcement, because I didn’t even start working on this integration until April 2025 - in fact, I’d never worked with Hubspot’s API at all prior to April 2025. I followed their docs, built the integration points I needed, tested it all… and I somehow managed to build and test an integration against an endpoint which was already scheduled for deprecation, without ever having the faintest clue.

Wouldn’t it be nice if, the day they decide something’s going to get switched off, that feature was no longer available to any new customers? Sure, the folks who were already using it before the announcement; makes sense to give them six months or whatever to update their code. But seems a bit odd to me that they’d offer completely new integrations access to a feature they already know is going to shut down soon.

..then again, maybe I’m just salty ‘cos I don’t like it when other people break my stuff. I guess the lesson is to always assume that every single API request you make might randomly start returning an HTTP 400, at any point, for no good reason, and engineer around that as best you can. Fallback caching actually kept the thing running for at least a week after the endpoint in question was deprecated, which I’m kinda happy about - but I could also have wired it to actually tell somebody if it had been running on cached data for more than 24 hours.

We Miss You, Tommy Vance

2025-07-06T01:15:00+00:00

In 1990, BBC Radio 1 broadcast the “Monsters of Rock” festival live from Donington Park. Thunder, Quireboys, Poison, Aerosmith, Whitesnake - and before, between, and after the bands, the broadcast featured live commentary and interviews from Tommy Vance, Mick Wall roaming the backstage area talking to the stars who were in attendance, and Richard Skinner’s 4-part history of the Monsters of Rock. Free to air. I know this, ‘cos I recorded the whole thing onto a stack of TDK D90 tapes and listened to it on repeat for the better part of the next decade.

Today, five million people paid £25 each to watch the live stream of Ozzy Osbourne’s farewell show with Black Sabbath, broadcast live from Villa Park in Birmingham. That’s over a hundred million pounds in revenue, just from the live stream… so, of course, between the acts, the stream cut to a professional studio, where experienced presenters talked about the history and significance of the show, interviewed the stars who were performing live… no, of course it didn’t. Instead, we got 15-second cameraphone clips of Sabbath fans all over the world talking about how much they loved the band, interspersed with fundraiser clips from the various charities the day was nominally supporting.

The coverage of the live acts was impeccable, but the interludes made the whole thing feel like one of those “live in my living room” streams everybody was doing back in 2020, and… well, I can’t help thinking that if Tommy Vance was still around, he wouldn’t have let that happen on his watch. But he’s not, and I didn’t realise until today that maybe when TV died, the best part of rock’n’roll broadcasting died with him.

Could HTTP 402 be the Future of the Web?

2025-07-04T22:27:00+00:00

When Tim Berners-Lee first created HTTP, way back in the early 1990s, he included a whole bunch of HTTP status codes that seemed like they might turn out to be useful. The one everybody knows is 404 Not Found - because as human being, browsing around the web, you see that one a lot. Some of them are basically invisible to humans: every time you load a web page, there’s actually an HTTP 200 OK happening behind the scenes, but you don’t see that unless you go looking for it. Same with various kinds of redirects - if the server gives back an HTTP 301, it’s saying “hey, the page you wanted has moved permanently; here’s the new address”; a 307 Temporary Redirect is saying “that’s over there right now but it might be back here later so don’t update your bookmarks”, and a 304 is saying to your browser “hey, the version of the page that’s in your cache is still good; just use that one”.

If, like me, you’ve spent a lot of time building web apps and APIs, you’ve probably spent hours of your life poring over the HTTP specifications looking for the best response code for a particular situation… like if somebody asks for something which they just deleted, should you return a 404 Not Found, or a HTTP 410 Gone - “hey, that USED to be here but it’s gone and it isn’t coming back”

And along the way, you’ve probably noticed this status code: HTTP 402 Payment Required. This code has been there since the dawn of the world wide web, but, to quote the Mozilla Developer Network: “the initial purpose of this code was for digital payment systems, however this status code is rarely used and no standard convention exists”.

Well, that might be about to change. Earlier this week, the folks over at Cloudflare announced they’re introducing something called “pay per crawl” - enabling content owners to charge AI crawlers for access.

Let’s put that in context for a second. For the vast majority of commercial websites out there, their ability to make money is directly linked to how many humans are browsing their site. Advertising, engagement, growth, subscribers - however you want to slice it, if you want to monetize the web, you probably want humans looking at your content. It’s not a great model - in fact, it’s mostly turned out to be pretty horrible in all kinds of ways - but, despite multiple valiant attempts to develop better ways to pay creators for content using some sort of microtransactions, advertising is the only one that’s really stood the test of time.

Then AI comes along. By which, in this context, I specifically mean language models trained on publicly available web data. I’ve published a transcript of this video on my blog. If somebody comes along in a couple of weeks and Googles “dylan beattie cloudflare pay per crawl”, Google could provide an AI-generated summary of this article. Or, like a lot of folks out there, that person might skip Google completely and just ask ChatGPT what Dylan Beattie thinks about pay per crawl - and they’ll get a nice, friendly - and maybe even accurate - summary of my post, and so that person never makes it as far as my website.

That is a tectonic shift in the way commercial web publishing works. For nearly thirty years, search engines have been primarily about driving traffic to websites; the entire field of SEO - search engine optimisation - is about how to engineer your own sites and your own content to make it more appealing to engines like Google and Bing. Now, we’re looking at a shift to a model where AI tools crawl your site, slurp up all your content, and endlessly regurgitate it for the edification of their users - and so the connection between the person who’s writing the thing, and the person who’s reading the thing, is completely lost.

For folks like me, who enjoy writing for humans, that’s just sad. For writers who rely on human web traffic to earn their living, it’s catastrophic… and that’s what makes Cloudflare’s proposal so interesting: they’re proposing a way to charge crawlers to read your content.

Now, websites have historically relied on something called a robots.txt file to control what search engines can and can’t see… but it’s advisory. The web was designed to be open. Robots.txt is like leaving all your doors and windows wide open and putting a sign on the lawn saying “NO BURGLARS ALLOWED”, and it’s just one of the many, many ways in which the architects of the web were… let’s say optimistically naïve about how humans actually behave. Which is maybe understandable, given that CERN, the nuclear research centre on the French/Swiss border where Tim Berners-Lee invented the World Wide Web, isn’t renowned for being a hotbed of unscrupulous capitalism.

So you had a choice: you make your content wide open and ask the robots to play nice, or lock it away behind a paywall so that only paying subscribers can see it. Which, of course, means the robots can’t see it, so your site never shows up in Google, so we invented all kinds of clever ways to create content that was accessible to search engines but asked human visitors to register, or sign in, or create an account…

Now, in theory, Cloudflare’s proposal is pretty simple. If your website is hosted behind a Cloudflare proxy - and according to Backlinko, of the world’s ten thousand busiest websites, 43% of them use Cloudflare, so that’s a LOT of websites - then when an AI crawler comes asking for content, you can reply with an HTTP 402 Payment Required, and give them a price - and if the crawler wants to pay, they try again, and include a crawler-exact-price header indicating “yes, I will pay that much” - this is reactive negotiation. Alternatively, there’s proactive negotiation, where the crawler says “hey, I’ll give you five bucks for that web page” and Cloudflare says “Yeah, sure!” - or if you’ve told your site that page costs ten bucks, the crawler gets an HTTP 402 Payment Required, and they’re free to make a better offer or go crawl somewhere else.

Incidentally, folks, I’m anthropomorphising here. Crawlers are software. They don’t ask, they don’t want to pay, they don’t agree. Even “crawling” is a metaphor. We’re talking about configuration settings and binary logic in a piece of software that makes network requests. Sure, it makes a better story if you’ve got this picture in your head of some sort of creepy-crawly insect knocking on doors and haggling over prices, but it is just code. Don’t get too attached to the metaphor.

Anyway. That’s the model. Sounds simple, apart from two tiny little details… how do you know the thing making the requests is an AI crawler, and who handles the money?

The first part is being done using a proposal called Web Bot Auth, based on two drafts from the Internet Engineering Task Force, the IETF: one, known as the directory draft, is a mechanism for allowing crawlers and web bots to publish a cryptographic key that websites can use to authenticate those bots, and the second, the protocol draft, is a mechanism for using those keys to validate individual HTTP requests.

That way, anybody running a web crawler can create a cryptographic key pair and register that key pair with Cloudflare - “hey, we’re legit, our crawler will identify itself using THIS key, here’s the URL where you can validate the key, and by the way, here’s where you send the bill”.

And that’s the second part: Cloudflare is proposing to aggregate all of those requests, charge the crawler account, and distribute the revenue to the website owners whose content is being crawled. Cloudflare acts as what’s called the Merchant of Record for those transactions, which should make it all much more straightforward when it comes to things like taxation.

Let’s be realistic here. The technical bits of this are not that complicated. They’re built using existing web standards and protocols. The financial elements of the proposal are far more complex, but this is Cloudflare, a company that probably understands the relationship between internet traffic, billing and international revenue models better than anybody.

There’s one big question that isn’t addressed in their post: what stops AI bots just pretending to be regular humans using regular browsers, and bypassing the whole pay per crawl thing? I’m guessing that, this being Cloudflare, AI bot detection is one of the things they’re quite good at… but publishers also have the option now of putting everything behind some kind of paywall; humans have to sign in, and bots have to validate. There’s also no indication as to what sort of amounts they have in mind - beyond the fact their examples are in US dollars, as opposed to, say, micros, which are a standard currency unit in Google’s payment API that’s worth one millionth of a US dollar. But I guess capitalism will figure that out.

Folks, I have to be honest. I’ve been working on the web since it was invented, and this is the first thing I’ve seen in a long, long time that is genuinely exciting. Not necessarily at face value - I don’t care that much about Cloudflare making AI bots pay to crawl websites. No, what’s exciting is that if Cloudflare goes all-in on this, this could be a big step towards a standard model, and a set of protocols, for monetising automated access to online content - even if neither Cloudflare nor AI is involved.

Imagine a decentralised music streaming service, where the artists host their own media and playback apps negotiate access to that media via a central broker that validates the app requests and distributes the revenue. Playback costs ten cents; if an AI wants to ingest, mash up and remix your music? Fifty bucks. Or a local news sites that can actually make money out of covering local news… how much would you pay to know what’s actually going on with all those sirens and smoke down the end of the High Street, from an experienced reporter who is actually on the scene asking questions, as opposed to somebody in an office recycling stuff they read on social media?

And the fact that the proposal is based around 402 Payment Required, something that’s been part of the web since the days before Google, Facebook, something that’s older than Netscape and Internet Explorer? That just makes me happy. It reminds me of the web back in the 1990s, when the protocols and proposals were all still new, and exciting, and it seemed like there was no limit to what we’d be able to build with them. And yeah, perhaps I’m being overly optimistic… but y’know, looking around at the state of the world, and the web, these days, maybe we could all use a little optimism.

The Linebreakers Video Macro

2025-06-25T10:11:00+00:00

Anybody who’s seen the Linebreakers live knows that backing videos are a vital part of the show - partly ‘cos that’s where all the drums come from, ‘cos it’s way easier taking a Macbook on a plane than travelling with a drum kit, and partly ‘cos having all the lyrics up on screen makes it much easier for the audience to keep up with the jokes, and there are a lot of jokes.

Those backing videos are all played from PowerPoint. It works, it’s reliable, and having the entire show inside a single 5Gb PPTX file means it’s trivial to run it from a spare laptop if something goes wrong. Every slide contains a single video clip, most of which start and end with a fade from/to black - so at the end of each song I’m staring at a laptop screen showing me that the last song is finished, and the next song is… black screen. So I rely on the speaker notes area to show me which song is on each slide, and updating this by hand is incredibly tedious and error-prone… so I wrote a macro for it. It’ll scan every slide in the deck, look for the embedded video clips, and replace the slide notes with the clip name (which is the filename) of the current and the next slide.

In Visual Basic for Applications, no less.

Sub UpdateSlideNotes()
  For Each Slide In ActivePresentation.Slides
    ThisName = "(no video)"
    For Each Shape In Slide.shapes
      If Shape.Type = msoMedia And Shape.MediaType = ppMediaTypeMovie Then
        ThisName = Shape.Name
      End If
    Next
    Slide.NotesPage.Shapes(2).TextFrame.TextRange.Text = "THIS: " & ThisName
    If Not LastSlide Is Nothing Then
        Set Notes = LastSlide.NotesPage.shapes(2).TextFrame.TextRange
        Notes.Text = Notes.Text & vbNewLine & vbNewLine & "NEXT: " & ThisName
    End If
    If (ThisName <> "") Then Set LastSlide = Slide
  Next Slide
End Sub

What The Fork Is Going On?

2025-06-16T16:58:00+00:00

I got two emails recently that have weighed heavily on my mind.

One was from a restaurant booking service called The Fork. On Saturday afternoon, I had used their service to try to book a table for dinner at Garlic & Shots, a legendary rock’n’roll bar in Stockholm. I tried calling them first: no answer, but hey, they’re probably just not open yet, and there’s a link on their website to reserve a table. 10 people, 7pm, email, phone number, confirm… “your reservation is pending”.

I hate “pending”. You can’t do anything with pending. “Hey, is there a dinner plan?” “Yeah. Or maybe no. A reservation is pending.” I suspect even Schrödinger’s cat would baulk at dinner being in some sort of unresolved quantum state.

OK, the place doesn’t open until 5pm, maybe they’ll confirm it then. 5pm comes and goes, no confirmation. I give them a call. A recorded voice says “Welcome!” - in Swedish - and then disconnects the call. This happens three times. There is no option in The Fork app to talk to a human, make a phone call, or anything of that nature. Finally I figure it’s not that far away, I get on the metro, head over, and talk to an actual human. They’ve been very busy: Iron Maiden has just played two arena shows in Stockholm, the city has been wall-to-wall metal fans for three days, and they’ve basically drunk all the beer in every rock bar in town - but no problem; they can take us for dinner. Might be 14 people, might be 8pm rather than 7pm… no problem at all. We have a delightful evening there. So. Much. Garlic.

Sunday 15th June, around 11am, I’m on the train from Stockholm to Helsingborg, and I get an email from The Fork confirming my restaurant booking: 10 people, 7pm, on Saturday 14th.

I do not know what kind of mind is required to develop, test, launch, and support a service that will confirm a restaurant reservation the day after the meal has taken place. I also strongly suspect that the proprietors of Garlic & Shots couldn’t honestly give half a microbollocks what The Fork is doing, but at some point some smooth-talking sales person probably said “no, it’s fine, we’ll handle all your reservations for you” and they just went “ok, whatever, now buy a drink or get out of my bar” and now it’s just a thing.

The next email was from His Majesty’s Revenue and Customs, informing me that there was an important message in my online tax account. They cannot, of course, tell me what the message says, or what it’s about, or even include a link directly to it. “Security reasons”.

An “important message” could actually be important. Not important like “there has been an update to the Netflix Terms & Conditions” important. Important like “you owe us money; pay up or go to jail” important.

So I open a browser, go to HMRC’s website, sign in with my Unique Taxpayer Reference, go through the multifactor authentication, go into my tax account, and sure enough, there’s an important message. I have a new tax statement. OK, cool. What does it say?

Oh, no, we can’t tell you that:

Your new Self Assessment statement has been prepared. You’ll be able to view it online within 4 working days.

So there you go. Two emails. One confirming a restaurant reservation that already happened two days earlier, the other telling me that I need to sign in to a website to see another message telling me that the thing they actually want to tell me isn’t ready yet but I should keep checking back because it’ll be there within four working days.

All you folks out there who think that coding is the hard part, and AI is going to revolutionise software development because it’ll replace the slow, unreliable human programmers with artificial intelligence that cranks out millions of lines of bug-free code in seconds? No. These emails didn’t get sent at the wrong time because of bugs in the code, or because scheduling email delivery is a hard problem and the developers couldn’t quite get it working in time. No, sending email to the right person, at the right time, is a solved problem. It’s been solved for decades. But choosing what that right time is? Identifying the situation where maybe asynchronous communication and pending reservations isn’t actually the best way to solve a problem? No, those require actual human intelligence, which is rapidly closing in on astatine’s long-held status as the rarest naturally occurring element on planet Earth.

I got two stupid pointless emails because somewhere behind them there are stupid pointless humans making stupid pointless decisions. Not programmers. In fact, most of the programmers I know would take one look at the ticket asking them to send an email saying “hey, your new tax statement is ready but you can’t read it yet” and go straight back to the product owner saying “um… how about we wait until the statement is actually available and then send the email?”

On the other hand, any startups out there using AI to replace their product owners and keeping all their human developers? Give me a call, I’ll come work for you. It would be lovely to be able to provide a bit of additional context and get a feature request instantly updated to something more sensible without causing a major political incident in the process.

Oh, and if you don’t know how to make sure email gets sent to the right person at the right time, I do that too. I made an entire course about it: Sending Email with .NET: From Zero to Hero, it’s on Dometrain, and they’re having a summer sale right now so it’s 30% off.

On The Fungibility of Trains

2025-05-28T14:03:00+00:00

When’s a train not a train?

I’m on my way from Antwerp to Budapest, via Amsterdam Schipol airport, on the delightfully fast and comfortable service that’s now called Eurostar but is still quite clearly a Thalys train with “Eurostar” painted on it.

Except I’m on the wrong train. I have a ticket - and a seat reservation - for the 13:30 departure to Schipol, so when a Eurostar train headed for Schipol pulled into platform 22 at Antwerp at 13:27, I went to board it. Except… no. This isn’t that train. This is train 9327. My ticket is for train 9333. There’s a Eurostar to Schipol every hour… and this one is now running exactly one hour late.

So I smiled very politely and asked if, since I had a ticket for the train that was leaving at 13:30 and I had a plane to catch, could I possibly use that ticket to board the train that was actually leaving at 13:30, and the train conductor agreed that this would probably be alright (and found me an empty seat - go Eurostar!)

The vast majority of trains in this part of the world, you don’t reserve a seat: you just buy a ticket to your destination, and it’s valid on every train headed that way. Whereas my flight to Budapest, the ticket is very clearly for a specific seat, on a specific flight, and even if by some bizarre coincidence there are delays and a different KLM flight leaves Schipol for Budapest at the exact time mine was supposed to leave, it wouldn’t occur to me to get on the wrong plane because it’s in the right place at the right time. Not without changing my ticket, anyway.

Which means Eurostar is in an interesting grey area… because although they run trains on the same rail network as EuroCity and Intercity, your Eurostar ticket is good for one seat on one train, identified by a four-digit train number… and, as I’ve learned today, the train that leaves Antwerp at 13:30 for Schipol Airport may not, in fact, be the 13:30 train to Schipol.

Next stop Budapest and CraftConf, which is going to be awesome because not only have they put together a fantastic line-up of speakers and sessions, but the conference is in a railway museum. (See? It’s all about trains today!) And then home. For a week-and-a-bit before heading to Stockholm for DevSum. I’d put hyperlinks in to all of these but I’m on my phone and you know how to Google.

Sending Email with .NET: From Zero to Hero

2025-05-14T18:44:00+00:00

Last month I published my first video course on Dometrain, “Sending Email with .NET: From Zero to Hero”: I just got this review from somebody who completed the course:

“⭐⭐⭐⭐⭐ Fantastic job! When I saw the course was over 9 hours, I wondered what you would talk about for that long. Now my head is swimming and it feels like 18 hours got packed into a 9 hour course. I didn’t realize how much I didn’t know about email. Thanks for putting this together.”

I didn’t set out to make a 9-hour course about sending email with .NET - I figured the course would run to 3, maybe 4 hours. But once I’d covered all the underlying standards like SMTP and MIME, modern .NET libraries like MailKit and MimeKit, tools like Mailpit, Mailtrap, ngrok, the chaos of HTML email standards and the tools like MJML that exist to work with it, how to set up all the various DNS records, SPF, DKIM, DMARC, troubleshooting, logging, sending mail from background services… there’s a lot to talk about.

The course is “Sending Email with .NET: From Zero to Hero”, you can buy it at dometrain.com, and it’s 40% off at the moment. Check it out. I think it’s awesome.

The Problem with “Vibe Coding”

2025-04-11T16:14:00+00:00

The whole “vibe coding” thing is another reminder that quite a lot of people working in tech don’t understand the difference between programs and products.

To me, programs are “works on my machine” code. The kind of things many of us crank out a few times every week. Experiments, prototypes… that script you hacked up to rename all the MP4 files in a folder? You know the one. No error checking. Hard-coded path names. Does it work on Windows? Who cares? I’m on Linux right now and I got work to do.

I have dozens of these kinds of programs I use every day. They’re tools I use to automate bits of my work. They crash all the time (“what? Oh… that person has a backslash in the title of their presentation… interesting.”) - but that doesn’t matter; I fix them, I get the results I need, I move on. The code is just a means to an end. The result is what matters.

If you’re writing software that you’re planning to ship; to distribute to other people, perhaps even sell it to paying customers? Well, now that’s a whole different ball game.

Probably the single most important lesson I’ve learned in my career, the thing that I would argue is the hallmark of “experience”, is understanding just how much work it takes to turn a working program into a viable product. It’s why developer estimates are so notoriously optimistic - and why experienced developers are so notoriously cynical. Let’s say you crank out a bit of code that’ll take responses from a web form and add them in an Excel spreadsheet. That’s not that hard… yay! we just built a Typeform competitor in one afternoon! Except, no, you didn’t. You made one thing work one time on one computer. You haven’t considered encoding, internationalization, concurrency, authentication, telemetry, billing, branding, mobile devices, deployment. You haven’t hit any of the weird limits yet - ever had a system work brilliantly for the first 65,535 requests and then fall over? You don’t have a product. At best, you have a proof-of-concept of a good idea that, if some very smart people work very hard, might become a viable product.

One of the genuinely positive things about tools like Copilot and ChatGPT is that they empower people with minimal development experience to create their own programs. Little programs that do useful things - and that’s awesome. More power to the users.

But that’s not product development, it’s programming. They aren’t the same thing. Not even close.

Lunch'n'Learn with the Amsterdam Java User Group

2025-03-24T17:02:00+00:00

I’m in the Netherlands. Last week was running some presentation training with a group at Info Support in Veenendaal, the weekend was catching up with friends in Amsterdam, and tomorrow I’m speaking at a student conference in Utrecht. One of the things about travelling for a living is that there’s no such thing as “out of office”, and the Todo List doesn’t stop just ‘cos you’re away for a week or two. My schedule said that today I’d be preparing examples and demos for a Zoom masterclass I’m giving on Thursday with the folks from Tech Cornwall.

So when I got an email from Geertjan Wielenga, who runs the Amsterdam Java User Group, asking if I wanted to do a lunch’n’learn session, I naturally said “no” - mainly ‘cos I’d already have left Amsterdam for Utrecht. But, as Geerjan pointed out, the two cities are ridiculously close together… so I decided, what the hell. Usually, when I do in-person events, I bring something that’s prepared and polished: talk, slides, video… you know. What some folks call a “Dylan Talk”. Today, we did the complete opposite: all I brought with me was a list of topics I’m still working on, and after sandwiches and coffee in Geertjan’s kitchen, we spent a fun hour putting together demos, learning about the weird and wonderful edge cases of modern web standards (did you know border-radius can have EIGHT different values? 🤯)

Here’s a few links to the stuff we looked at:

and the code from the session is all up on GitHub:

https://github.com/dylanbeattie/modern-web-dev

Fun session. And a nice reminder that not every tech event needs to be rehearsed and polished - sometimes all you need is people, coffee and curiosity.

's-Hertogenbosch

2025-03-23T10:52:00+00:00

There is a place in the Netherlands called ’s-Hertogenbosch. Yes, that’s not a typo. Its official legal place name begins with an apostrophe. The locals all call it Den Bosch (and never, ever mention that you won’t find Den Bosch on a map ‘cos why would anybody possibly need to know that?)

And, seeing it on a railway departure board earlier this morning, it occurs to me that if there’s anywhere in the world that figured out a lot of database stuff right from day 1, it’s probably the good people of ‘s-Hertogenbosch.

(obviously and shamelessly stolen from https://xkcd.com/327/ so if you like it go and buy one of Randall’s books or something. The anniversary edition of What If? is particularly good.)

An Interactive CSS Flexbox Playground

2025-03-22T17:15:00+00:00

CSS flexbox is fantastic… but it’s also incredibly complicated. A flex container has half-a-dozen different CSS rules which control how content will flex and flow within the container, and figuring out the exact combination that you’ll need to implement a particular design can be a lengthy and error-prone process.

So I built a thing to make it a little easier to see what’s going on. It’s an interactive CSS flexbox playground. Based on - and with visuals inspired by - Chris Coyler’s excellent CSS Flexbox Layout Guide over at css-tricks.com, it’ll let you pick different values for all the various CSS flex properties and see how they affect your layout.

Check it out here:

https://dylanbeattie.net/tools/flexbox/

If you like this kind of thing, I’m running a couple of events next week talking about web standards - HTML, CSS, JavaScript, native browser APIs; all the cool stuff that used to require thousands of lines of JavaScript frameworks and polyfills that just works now because browsers are getting better all the time

On Monday 24th March I’m doing a ‘lunch & learn’ in Amsterdam:

https://www.meetup.com/amsterdam-java-user-group/events/306823736/?eventOrigin=your_events

then on Thursday 27th March, I’m running an online session with Tech Cornwall. 11:00-13:00 on Zoom, it’s free for members and £99 (+ VAT and booking fees) for non-members: check that out here:

https://techcornwall.co.uk/training/modern-fundamentals-of-web-application-development-masterclass/

Hopefully see you a few of you there.

Metaphors

2025-03-16T10:34:00+00:00

Back in 2020, when we all had a lot more time on our hands, I would occasionally hang out on /r/WritingPrompts. I stumbled across this old post today and figured I’d post it here. You know. For posterity.

Writing Prompt: You’re on a space ship with a bunch of your crewmates. You’re the only human, and apparently metaphors are a strictly human behavior. You’ve learned to cope with this, but today you’ve decided to speak in only figures of speech as a prank on the others.

“Good morning, Harzz. I feel like death.”

“Captain, you cannot feel like death. Death implies the absence of perception - and furthermore, is an experience with which you, unlike me, cannot be familiar.”

Ah, Harzz. Great science officer… pain in the ass before I’ve had my morning coffee. I should have known better. I’m about to explain when… no. You know what? This is my ship, this is my crew, and I’m tired of having to break everything down into logical, literal phrases the whole time. Today, just for once, I’m going to talk how I want to talk. They can figure it out.

“Don’t worry, Harzz. I’m just yanking your chain. I’m just a bit wiped out today, that’s all. I’ll be as right as rain after I’ve had a cup of joe.”

I head to the galley, leaving the bemused Harzz staring after me. You know, this could actually be fun…

We’ve been out here six weeks, along the edge of the disputed zone. One of those missions that feels like more of a box-ticking exercise than anything else - the “dispute” turned into more of a cold war decades ago, we’ve surveyed every rock in this place a dozen times over. No settlements, no life signs, no hostiles… HQ thinks having a couple of ships along our edge of it keeps things under control, and so far they’re right. Looks like today’s gonna be another long, dull day of empty space and blank scopes.

Sometime around ten, I wander into the mess to find the crew gathered there. Harzz, Djanik, Kjin-ti… time to have a little fun. I pour myself a cup of coffee and sat down.

“Welcome, captain. We were just reviewing today’s mission telemetry. Would you like a report?”

“No, Djanik, that’s OK. I’ll review it later. I was just thinking, you know, us out here together all these weeks, maybe it’s time we got to know each other a little.”

“Are you concerned about the crew’s performance, captain?”

“No, nothing like that. Just… you have to admit, this mission isn’t exactly keeping us busy. Thought maybe we could swap a few stories, help to pass the time. Did I ever tell you about my parents?”

“Your parents, captain?”

Here we go. This was gonna be fun.

“Yeah. You probably know my dad - he’s a big cheese, was one of the top brass for a while. It wasn’t always easy for him - he was always the kind of guy who called his own tune, y’know, marched to the beat of his own drum.”

“Captain, forgive me, I had no idea your family were musicians. I thought your father was an officer in the Federation.”

“Yes”, interjects Djanik, “and I had always believed your species was carbon-based. I did not know it was possible to create sentience from dairy products and metal alloys.”

I go on. “Well, he was a real high flyer back in his academy days - gave him a reputation as a bit of a hard-ass”

“I was unaware that altitude affected the composition of your species’ buttocks, Captain. This is fascinating.”

I catch Kjin-ti whispering to him “don’t be an idiot, Djanik! The captain already said his father was top brass - that explains the altitude and the composition!”

I’m already struggling to keep a straight face. “One night, dad and some of his classmates head out - they got some shore leave, decide to go out, paint the town red, see if they can pick up some birds”.

“Your father’s dedication to urban maintenance and wildlife conservation must have impressed the examiners at the Academy, captain”

“Perhaps, Djanik. Perhaps. Now, you gotta realise, my dad was normally an early bird - up at the crack of dawn most days…”

(I overhear Kjin-ti again: “Djanik, I have never heard the dawn crack… is this an Earth phenomenon?”. Djanik replies “I am confused as well, Kjin-ti. The captain’s father is now a bird made of brass and cheese. I am finding the aerodynamics difficult to calculate.”)

“…but my mum was a night owl; the kind of woman who was always burning the candle at both ends. Well, story goes, their eyes met across a crowded dance floor.”

“Their eyes? Did the rest of their bodies meet as well, captain?”

I laugh. “I guess you could say that, Kjin-ti. They were like glue after that night.”

Djanik nods. “Your mother must have used the heat from her candle to catalyse the proteins in your father and create an adhesive. I hope her feathers were not damaged during the process.”

Just as I can’t hold it any more, the alarm rings out. Djanik’s face is impassive. “High alert, captain. All stations.”

“Wait.”

The crew stop and stare at me. This was irregular.

“High alert, Mr Djanik? How high?”

“I… do not understand the question, captain.”

“Djanik, you said there was a high alert. I would like to know: how high is the alert?”

“I… captain, it is a high alert. We should respond at once!”

“Computer, cancel alert.” The crew are all staring at me now. “I’m sorry I deceived you. There is no alert. I was merely trying to make a point. When the alarm went off, you all recognised it as a high alert, correct?”

They nod.

“But that doesn’t mean anything, really, does it? The alert doesn’t actually have a height.”

They’re still staring, but the penny is starting to drop (I must remember to try that one on them later.)

“So… a high alert is more serious than a regular alert… yes?”

They nod again.

“Why do you think that is?”

Kjin-ti figures it out first. “Gravity, captain. Your species evolved language on a planet with gravity. You associate height with magnitude - size, scale, danger. For your ancestors, big things were high, and big things were dangerous.”

I smile. “Go on…”

“…and…” the leap in cognition is right there. She goes for it. “…and your language is built on comparisons. This is why you speak of turning up audio signals and turning down invitations, when neither of these involves a change of altitude.”

Djanik is deep in thought. “So, you say your father was a big cheese… and large things are important in your society. Was your father made of cheese, captain?”

“No, Djanik. We humans don’t just use spatial metaphors about size and height. We use images from nature, from our surroundings. Among my ancestors, to own a large piece of cheese was a sign of wealth and affluence, at a time when many humans had insufficient food.”

“And your mother… was not really an owl.”

“No, Harzz. She was 100% human, same as my father, same as me. But the owl was a nocturnal bird back on Earth, so people who stayed up past dark were called night owls.”

“I see. Captain, your species’ style of communication is highly illogical… but I begin to see how, in the absence of telepathy or pheromones, it may have given you an evolutionary advantage. Fascinating.”

The crew sit in contemplative silence for a few minutes. I pour another cup of coffee and return to the bridge, check the readouts, lay in a course, and the ship thunders to life, tongues of flame licking from her engines, an arrowhead streaking through the heavens, the stars around us an infinity of diamonds scattered across the velvet darkness of space.

"Open Source, Open Mind" - A Free Meetup with Info Support in Veenendaal

2025-03-04T11:00:00+00:00

On March 19th, I’ll be speaking at a free meetup in Veenendaal in the Netherlands, hosted by the wonderful folks over at Info Support, all about the history - and future - of “free software”. We’re coming up on half a century of folks giving their code away for free, but we’re still not entirely sure what “free software” is, how it works - or whether it’s possible to create a truly sustainable ecosystem around free and open source software.

Come along and hear about the history of open source, free software, the GNU project, Commander Keen, WordPress, FluentAssertions, Redis… and more.

See you there!

Martin reckons it'll take me 20 minutes to write a blog post.

2025-02-27T18:29:00+00:00

Martin is apparently dubious as to just how awesome Sveltia CMS is. (Martin also apparently isn’t entirely sure of the difference between Sveltia and Svelte, but I’m sure he’ll figure it out eventually).

For the record, he made this comment at 18:29 UTC on 2025-02-27.

20 minutes? I make that 3 minutes, plus another 2 for GitHub Actions to build and publish the site.

Iconography is Hard: VS2022 Refresh Edition

2025-02-26T13:35:00+00:00

I’ve spoken before about the four different styles of visual communication - pictographic, phonographic, ideographic and logographic:

Pictographic writing is where the symbol is a picture of a thing. The smiley face emoji 😃 is a good example - regardless of what languages and alphabets you can read, you’ll see that picture and think “hey, a happy person!”
Phonographic writing is where pictures stand for sounds. It’s how most Western alphabets work - remember learning to read in school? Sounding out “r e d b a l l” until your brain put the sounds together into “red ball” and you went “oh, yeah, a red ball!”
Ideographic writing is where pictures stand for ideas - and unless somebody’s explained it in advance, you have no clue what it means. You know the “eyes” emoji 👀 ? Does that mean “I’m watching this to see what happens?” Or “I’m looking into this for you now?” Or “I’m rolling my eyes in despair because I knew this was going to happen?” Depends on the context, culture, team.
Logographic writing is where an image stands for a word - not an idea, but a specific word. You don’t find it much in Western languages and alphabets but it’s used in Chinese, and in kanji, one of the three forms of written Japanese: the kanji 父 represents the word “father”, for example.

When it comes to building things like toolbars and user interfaces, things can get properly gnarly. Most modern desktop apps use a delightful mashup of pictograms, ideograms and phonograms - the little yellow folder where you stored all your documents, the floppy disk that used to be a pictogram but most folks using computers today have never actually seen one so it’s become an ideogram.

I noticed earlier today that the menu in Visual Studio 2022 has two almost identical icons on it… circular arrows, which we’ve come to associate with the idea of refreshing something (“go around again”, I guess?)

Except in VS2022, if the arrow’s rotating clockwise, it means “Browser Link”, and if the arrow’s rotating counter-clockwise, it means “Restart (Ctrl+Shift+F5)”. I guess… righty refreshy, lefty-lets-recompily?

But hey - iconography is hard. You try explaining the difference between a Browser Link refresh and a project restart in 24x24 pixels. 😉

Visual Studio 2022 17.13.1 Broke My Syntax Highlighting

2025-02-24T12:22:00+00:00

Another day, another software update that breaks something you can’t quite believe anybody’s still messing around with… but hey, even here in 2025 I guess somebody at the Visual Studio team is looking at syntax highlighting and going “yeah, we could improve that”.

Here’s a snippet of code from the workshop I’m writing, that I snapped a few days ago. This is Visual Studio 2022 Professional, version 17.12.3, with a slightly modified version of h4zm1’s awesome Cyberpunk theme (GitHub) (VS Marketplace):

And here’s that same snippet of code when I opened it up this morning, after installing the latest update to VS 2022 17.13.1:

You see how the constructor parameters aren’t hot pink any more, they’re a sort of muted navy blue? And GetWebsiteBaseUri has gone from being Sunny Delight to being a colour I think we will call “crusty mustard”?

Yeah. That’s a problem. Those muted colours, that’s Visual Studio’s way of saying “hey… this thing isn’t actually being used; maybe we can remove it?” - except those bits of code absolutely are being used.

Tried the usual things. Restarted VS2022. Updated Resharper. Restarted the computer. Nope. Eventually had to revert the update - which, of course, clobbered the colour theme installation - but eventually I managed to get VS downgraded back to 17.12.3, get the color scheme reinstalled, and generally get things looking the way they’re supposed to. There’s 90 minutes of my life I’m never getting back.

Why does it matter? I’m halfway through making a training course. Lots of video. Lots of recording screens of code. So I guess we add another rule to the rulebook: don’t update Visual Studio when you’re in the middle of a video project.

I’ve filed a feedback issue with the VS team here:

https://developercommunity.visualstudio.com/t/VS2022-17131-update-has-incorrect-synt/10856882

if you end up on this post searching for the same problem, go give it an upvote 😉

Replaying Keystrokes with AutoHotkey macros

2025-02-18T09:06:00+00:00

AutoHotkey is one of the strangest and most powerful things I’ve ever encountered in all the years I’ve been working with computers. It’s a macro programming language and interpreter for Windows that will let you automate just about anything - keystrokes, mouse clicks, MIDI, game controllers, window handles… if you for some bizarre reason wanted to control your Windows audio volume using a Thrustmaster throttle controller, AutoHotkey could totally do that.

I’ve been using it to build little macros that will replay keystrokes from the clipboard, so that when I’m recording training videos, I can pause the camera, type in the code, cut it to the clipboard, un-pause the camera, and then replay those few lines without any pauses or typos while I explain what they’re doing. It’s not perfect, but it’s the best solution I’ve found yet in terms of minimising prep time vs ending up with unusable footage.

Here’s the latest version of the script I’m using for this.

F13 reloads the script. It’ll also reload if you press Ctrl-S while the active window has “autohotkey” in the title, so when I’m editing the script in VS Code it’ll reload every time I save my changes.
F14 will replay the entire contents of the clipboard, including line breaks, and pause for 100ms at the end of each line.
F15 will replay the next line from the clipboard each time you press it, including the linebreak.
F16 will replay the next line from the clipboard without the linebreak.
If Scroll Lock is ON, every line will be trimmed before replaying it - this works around the quirk where most editors (VS Code, Visual Studio, etc) use “smart indentation” and don’t start new lines from column 0, but if you’re copying code it’ll normally have a bunch of leading tabs/spaces included.
It’ll play a little R2/D2 noise whenever you reload the script (you’ll need the file r2d2.wav go to with it)

Finally, you’ve got to use physical keys. Don’t ask me why. I tried remapping some of the buttons on my StreamDeck to simulate F13-F16; didn’t work – I guess AutoHotkey’s hooking into the keyboard events at a lower level than StreamDeck’s injecting them? But I reprogrammed the “playstation keys” on my Keychron K5 to be F13-F16 and it that works just fine.

#Requires AutoHotkey v2.0
FileEncoding "UTF-8"

~^s:: {
    if WinActive("autohotkey") {
        SoundPlay "r2d2.wav", 1
        Sleep 200
        Reload
    }
}

SetKeyDelay 64  

F13:: {
    SoundPlay "r2d2.wav", 1
    Sleep 200
    Reload
}
; F14 will replay the entire clipboard, pausing for 100ms between lines
F14:: {
    SendMode "Event"
    firstLine := true
    loop parse A_Clipboard, "`n", "`r" {
        if (firstLine) {
            firstLine := false
        } else {
            Send('{Esc}')
            Send('{Enter}')
            Sleep 100
        }
        trimLines := GetKeyState("ScrollLock", "T")
        trimmedText := A_LoopField
        if (trimLines = 1) {
            trimmedText := Trim(trimmedText)
        }
        SendText trimmedText
    }
}

; F15 will replay the next line from the clipboard, including the trailing newline
F15:: {
    SendMode "Event"
    firstLine := true
    loop parse A_Clipboard, "`n", "`r" {
        if (firstLine) {
            firstLine := false
        } else {
            Send('{Esc}')
            Send('{Enter}')
            KeyWait "F15", "D"
            KeyWait "F15"
        }
        trimLines := GetKeyState("ScrollLock", "T")
        trimmedText := A_LoopField
        if (trimLines = 1) {
            trimmedText := Trim(trimmedText)
        }
        SendText trimmedText
    }
}

; F16 will replay the next line from the clipboard 
; lines are trimmed to remove leading/trailing whitespace
; but will NOT send newlines - you gotta type Enter yourself.
F16:: {
    SendMode "Event"
    loop parse A_Clipboard, "`n", "`r" {
        trimLines := GetKeyState("ScrollLock", "T")
        trimmedText := A_LoopField
        if (trimLines = 1) {
            trimmedText := Trim(trimmedText)
        }
        SendText trimmedText
        Send('{Esc}')
        KeyWait "F16", "D"
        KeyWait "F16"    
    }
}

“How To Be a Rockstar Developer” at DDD North

2025-02-17T13:15:00+00:00

I’m off to Hull this weekend for DDD North where I’ll be presenting How To Be A Rockstar Developer - the story of how a programming language I invented in a bar as a joke got way out of hand, and ended up as a real thing, complete with a web assembly runtime, test suite, documentation… and lots of stupid jokes about Bon Jovi.

I love DDD (which is short for Developer! Developer! Developer!, inspired by a thing Steve Ballmer did when he was CEO of Microsoft, and not to be confused with Domain-Driven Design which is a completely different thing). It’s a series of free, one-day developer events around the UK. And in Australia, but that’s a bit far to go for a one-day event. And apparently they had one in Seoul. DDD’s historical associations with Microsoft mean it attracts a lot of folks who work with .NET, but with an open CFP and sessions chosen via a public vote, there’s usually a pretty good balance of speakers and topics.

It’s free, it’s on a Saturday - this Saturday, in fact! - and it’s always a really friendly bunch of people. There’s still a few tickets left at dddnorth.co.uk, and if you’re going to be around, come and say hi.

Faking the HTTP Request URL in ASP.NET MVC

2025-02-15T17:57:00+00:00

Because I always, always, forget how to do this: here’s how you fake, spoof, whatever you want to call it the HTTP request in ASP.NET Core. Useful if you’ve got controller actions that do things like generate absolute URL links to put into outgoing emails, and you want them to reflect the URL where the code was actually running (so code running on localhost sends mails with links to localhost, code running on test.example.com links to test.example.com, and so on), and you want to write tests for that code because why wouldn’t you write tests?

Looks like this:

protected ControllerContext SimulateControllerContext(string uriString) {
	var uri = new Uri(uriString);
	return new ControllerContext {
		HttpContext = new DefaultHttpContext {
			Request = { Scheme = uri.Scheme, Host = new HostString(uri.Host) }
		}
	};
}

var controller = new WhateverController() {
	ControllerContext = SimulateControllerContext("https://example.com")
}

I’ll normally mix this up with a little extension method, usually while wishing that the people who wrote the very excellent Path.Combine had elbowed their way into the room where they were building System.Uri and Got Involved:

public static class UriExtensions {
	public static Uri Append(this Uri uri, params string[] paths)
		=> new(paths.Aggregate(uri.AbsoluteUri, (current, path)
			=> $"{current.TrimEnd('/')}/{path.TrimStart('/')}"));

	public static Uri GetWebsiteBaseUri(this HttpRequest request)
		=> new($"{request.Scheme}://{request.Host}/");
}

(also, yay blogging with Sveltia means no more messing around with git. Write, publish. done.)

Posting with SveltiaCMS from my phone

2025-02-13T20:55:00+00:00

And here’s the ultimate acid test for low-friction blogging… how quickly can I write and publish a post using my phone? Well, the layouts are a little janky:

…but it works. Nice. Expect more off-the-cuff blogging.

SveltiaCMS, Jekyll and GitHub Pages

2025-02-13T20:38:00+00:00

I used to blog a lot. Back in the good old days of Windows Live Writer and Blogger, when I could click “New post”, bash out a few hundred words, paste an image or two, and… done.

In 2020 I moved all my stuff from Blogger to Jekyll and GitHub Pages, because I wanted more control over the content on my site. It worked, and I love Jekyll dearly and it’s a fantastic tool, but it kinda killed my blogging because even a two-paragraph post meant git pull, create a new Markdown file, make sure to get the filename right (and filenames are based on dates, so it if you don’t get around to posting it the day you started it you’ve gotta rename stuff), git add, git commit, build… yeah.

So I asked around for whatever’s the closest thing to Windows Live Writer - with the caveats that cntent stays in Markdown, and for now it stays on Github. I own my content. I want it under my control, in a portable format.

Chris Simon pointed me at Decap CMS, which looked really interesting: it’s a JS app that runs from a subfolder inside your static site, connects directly to GitHub, and edits Markdown files right in your repo. Except (a) I managed to crash their live demo editor within fifteen seconds by trying to paste an image into it (I know! Pasting an image! Into a blog post! Totally weirdo edge case that’ll never happen in real life!), and (b) once I’d figured out how to Not Paste Images, I couldn’t get the auth for it running reliably. It uses Netlify to authenticate with GitHub – GitHub can’t do pure JS authentication ‘cos it needs a server to handle the post-authentication callback – and seemed to have some problems using Netlify to authenticate when the site I was trying to edit wasn’t hosted on Netlify.

Then I found Svelti a CMS. It’s the same idea: CMS as a JS app, in a subfolder of a static website, that pushes stuff straight to a GitHub repo - along with a Cloudflare worker app that handles the authentication bit.

So if you’re reading this? It worked: this was posted from Sveltia, which is hooked into dylanbeattie.net/admin, and maybe means bashing out those quick 2-paragraph blog posts might just be as easy as it was in the days of Live Writer.

Windows Defender detects Trojan:Script/Wacatac.B!ml in a zipped .NET 9 AOT binary

2024-12-09T12:00:00+00:00

Earlier today, somebody alerted me that Rockstar binary releases for Windows were being blocked by Windows Defender… and sure enough, since I switched the build system from .NET 8 to .NET 9, the Windows ZIP file releases on the releases page are being incorrectly identified as being infected with Trojan:Script/Wacatac.B!ml malware.

It’s not just Rockstar, though. It’s any .NET 9 binary compiled with AOT enabled and then compressed in a ZIP file. Try it. First, create a new .NET 9 console app:

dotnet new console -o HelloWorld

Then edit the .csproj file and add StripSymbols and PublishAot directives:

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net9.0</TargetFramework>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
    </PropertyGroup>
    <!-- ADD THIS PropertyGroup to create native binary EXE builds -->
    <PropertyGroup Condition="'$(Configuration)'=='Release'">
        <DebugSymbols>False</DebugSymbols>
        <DebugType>None</DebugType>
        <PublishAot>true</PublishAot>
        <StripSymbols>true</StripSymbols>
    </PropertyGroup>
</Project>

Now publish a binary release:

dotnet publish -c Release -o published

Look in /published and you’ll see a single binary HelloWorld.exe

Create a ZIP file containing that EXE - in Windows Explorer, right-click, Send to > Compressed (zipped) folder:

Right-click the ZIP file, Scan with Microsoft Defender…

For a second opinion, upload the ZIP file to VirusTotal.com, and you’ll probably see that Microsoft Defender thinks it’s infected with a trojan. I’ve had multiple hits reporting Trojan:Script/Wacatac.B!ml and one reporting Trojan:Win32/AgentTesla!ml

Running Corel Linux on QEMU

2024-05-27T12:00:00+00:00

In my latest YouTube video, I talk about Corel Linux, which I think is one of the most interesting “what if?” scenarios in the history of mainstream computing, and show some clips of Corel Linux actually running. It’s not easy getting a 25-year-old Linux distro to boot on a modern PC - but here’s how I did it.

Install QEMU

VirtualBox, HyperV et al won’t work here - Corel Linux is just too old to support most of the virtual hardware they expose to their guest operating systems. Instead, we’re going to use QEMU.

Download it from https://www.qemu.org/, install it, get it running. Check it works by running qemu-system-i386 at a terminal window:

> qemu-system-i386 --version
QEMU emulator version 9.0.0 (v9.0.0-12054-g923cf646f4)
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
> 

Create a virtual disk

This will create a 2Gb virtual disk file called corel_linux_hd.img:

qemu-img create corel_linux_hd.img 2G

Download Corel Linux

I used the ISO image of Corel Linux 1.2 deluxe from https://archive.org/details/corel_linux_1.2

While you’re there, you might also want to download CorelDRAW and WordPerfect Office https://archive.org/details/CorelForLinux/CorelDRAW%209%20Linux/

Run the installer

qemu-system-i386 -hda corel_linux_hd.img -cdrom corel_linux_1.2.iso -m 256 -vga cirrus

You should get this:

and then this…

and then this:

Yeah. This is where it gets interesting: the Cirrus VGA driver provided by QEMU displays garbled text until we get into the OS and hack it, but we can’t do that until we’ve got it installed, so we’ll need to walk through the installation without being able to read anything.

You probably also don’t have any mouse support, so use Tab to move forwards, Ctrl-Tab to move backwards, and the Spacebar to click:

Don’t change anything here - “Install Standard Desktop” is already selected so just press Enter:

The next few screens, accept all the defaults - tab to Next> if it’s not already focused, press Enter:

and finally, Install:

If you get to here, you’re on the right track:

Once it’s installed, let it reboot, then at the loading screen select Linux - Text Mode

It’ll happily chunter away for a few screens worth of messages, and then you’ll get a login prompt.

Now we need to configure X11 so that it won’t try to use various accelerated hardware features that don’t work in QEMU.

This bit comes from Ethan Gates’ Corel Linux in QEMU post on forums.eeasi.cloud – thank you Ethan! – and that post was in turn inspired by Hayden Barnes’ post The one in which I kind of get Corel Linux 1.2 to work 21 years later - thank you Hayden!

Use vim to edit /etc/X11/XF86config. Find the section called “Device”, add three lines:

Option "no_bitblt"
Option "noaccel"
Option "sw_cursor"

Save the file, reboot (shutdown -r now), and select Corel Linux from the boot menu:

Networking support

Networking support in QEMU is powerful, flexible, and incredibly complicated.

Here’s how I made it work. Disclaimer: I don’t 100% understand exactly what all this does… I tried just about every combination I could think of until I found something that worked.

First, you’ll need to install a TAP network driver. This adds another network interface to Windows, which emulates a physical network card. Apparently.

I installed mine using chocolatety:

choco install tapwindows

Then, in Windows network settings:

Rename the new TAP connection to TapWindows
Right-click your main network connection (ethernet, wifi - whatever connects your Windows machine to the internet)
Properties > Sharing > Allow other network users to connect…
Choose your TapWindows connection as the “Home networking connection”

You’ll get a popup about your LAN adapter. The wording here is misleading - it’s not your LAN adapter, it’s whichever adapter you selected as you” Home networking connection”:

If you remember using Internet Connection Sharing with a dial-up modem… this is the same tech, only we’re pretending that QEMU is our house network, TapWindows is the network card connecting our PC to the rest of the house, and “Ethernet” is the dial-up modem that connects to the internet.

Now, run QEMU using this command line:

I’m using Powershell so a backtick ` is a line continuation character.

qemu-system-i386 -hda corel_linux_hd.img `
-m 256 -vga cirrus -audiodev driver=dsound,id=pa1 -device sb16,audiodev=pa1 `
-netdev tap,id=mynet0,ifname=TapWindows `
-device pcnet,netdev=mynet0

That’s telling QEMU “create a network connection using TAP, call it mynet0, and connect it to the TapWindows interface on the host PC”, and then on the next line “then create a virtual device using the pcnet and connect it to the mynet0 network”.

For me, this worked 80% of the time. When it didn’t work, it’s because something meant the Linux guest couldn’t get an IP address from Internet Connection Sharing’s DHCP server. Manually setting the guest IP to 192.168.137.2, subnet 255.255.255.0, gateway 192.168.137.1 usually fixed this.

That got me to the point where I could boot Linux, open Netscape, point it at http://info.cern.ch/, and browse the world’s first website.

Stuff I Couldn’t Figure Out

I couldn’t get audio working. QEMU audio works fine - I created a Windows 2000 guest just to test this - and I tried both the ac97 and sb16 virtual audio devices in QEMU, but no luck yet.

I also couldn’t figure out how to change the CD-ROM. When you’re running QEMU, Ctrl-Alt-F2 drops you into an emulation console where you can use info block to see which devices/images are connected, and change ide1-cd0 <filename.iso> to change the virtual disk:

But… then I couldn’t work out how to get Corel Linux to mount the disk image, so to install WordPerfect, CorelDRAW!, etc. I had to shut down the VM and then boot it specifying the -cdrom command line switch:

qemu-system-i386 -hda corel_linux_hd.img `
-m 256 -vga cirrus -audiodev driver=dsound,id=pa1 -device ac97,audiodev=pa1 `
-netdev tap,id=mynet0,ifname=TapWindows `
-device pcnet,netdev=mynet0 `
-cdrom .\wordperfect_office_2000_deluxe_cd_1.iso

Ah, the nostalgia.

Using the Contour ShuttleXpress with Camtasia

2024-03-17T12:00:00+00:00

The Contour ShuttleXpress is a controller designed for editing audio and video. I use it all the time in applications like Logic Pro X; the jog dial and shuttle wheel are brilliant for, well, jogging and shuttling around your project’s timeline.

Camtasia is a screen recording and video editing app from the folks at TechSmith, which I use for doing the screen recordings I use in my presentations and workshops.

Out of the box, the ShuttleXpress doesn’t support Camtasia, so I created an application profile for it.

Action	Keystroke	What it does
Jog Left	`,`	Step backwards in timeline
Jog Right	`.`	Step forwards in timeline
Shuttle left/right	`,` `.`	Move forwards backwards in timeline. Each position on the shuttle is independently programmable, so “shuttle in left 7” sends `,` as fast as possible, “shuttle in left 6” sends 60 strokes per second, and so on. It’s not a perfect timeline shuttle but it’s close enough.
Button 1	`Ctrl` + `Home`	Jump to beginning of timeline
Button 2	`Ctrl`+`Alt`+`,`	Move playhead to previous clip
Button 3	`Shift`+`Ctrl`+`Alt`+`Left Arrow`	Extend selection to previous clip
Button 4	`Ctrl`+`Alt`+`.`	Move playhead to next clip
Button 5	`Shift`+`Ctrl`+`S`	Split all tracks at playhead

The workflow here is:

Shuttle until you find the start of the mistake
Button 5 to split all tracks
Shuttle forward to the end of the mistake
Button 5 again to split all tracks
Button 3 to select the new region with the mistake in it
Delete to remove the region.

and if you switch on “magnet mode” for the track you’re editing, when you delete the mistake it’ll automatically snap the remaining pieces together so it doesn’t leave a gap.

You can download the settings file here:

CamtasiaStudio-ShuttleXpress.pref

and import it using Settings Management > Options > Import Settings in the Contour Shuttle Device Configuration app:

Happy shuttling :)

Recording Meetups

2024-02-20T12:00:00+00:00

For the last few months, I’ve been recording the talks at the London .NET User Group on video so we can post them on YouTube after the event. The videos are online at https://www.youtube.com/@LondonDotNet, and I’ve had some questions about the setup I use to record them. Some from interested people looking to do something similar at their own events, some from people who show up early while I’m still setting up and go “wow, what’s with all the shiny things?” So here’s how it works.

(I made a diagram! Want it as a PDF? Here you go: dylanbeattie-recording-meetups.pdf.)

Short answer:

Canon EOS M200 camera
Elgato Camlink
Elgato Game Capture HD60S+
AnkerWork 650 wireless microphones
OBS Studio on an M1 Macbook Pro
Lots of HDMI and USB cables

For the long answer: the key here is minimal post-production. When it comes to editing two hours of video footage, “tomorrow” turns into “later” turns into “never” with astonishing regularity. Other than editing out any major technical problems, I want to leave the meetup with a video file that’s ready to upload to YouTube.

I also don’t want presenters to have to do any more than plug in HDMI, clip on a microphone, and go. I’ve done a few events where I’ve had to install drivers or sharing software for their “wireless smart display” system: I really don’t like doing that, and I’m not going to ask our presenters to do it either.

In terms of portability, I assume the venue will provide HDMI for the presenter’s slides, and there’s a power socket available to plug everything in. Everything else needs to fit in a backpack and set up in less than fifteen minutes.

Camera: Canon EOS M200 + Elgato Camlink

The camera I use is a Canon EOS M200. It’s a mirrorless SLR with clean HDMI out: what this means in practice is you can plug it into an Elgato Camlink, plug that into your laptop, and it shows up as a webcam - albeit a really, really good one. The camera runs off a USB power supply so I don’t have to worry about running out of battery halfway through a recording.

Bad news: Canon has apparently discontinued the M200, along with the rest of the M series, in favour of the Canon EOS R series. This makes me sad, because the M200 is a wonderful camera.

Good news: Elgato maintain a list of supported cameras, so any of the cameras on this list will do the trick:

https://www.elgato.com/uk/en/s/cam-link-camera-check

If I can, I’ll bring a proper camera tripod along. If not, I’ll bring a SmallRig “magic arm” clamp and find a convenient chair, lamp, window frame or something to clamp it to.

Slides: Elgato Game Capture HD60S+

To capture the speaker’s laptop, I use an Elgato Game Capture HD60S+ (now superseded by the Game Capture HD60 X) This works like a Camlink - turns HDMI into a USB video signal - but it has an HDMI output that uses passthrough, so I can plug it in between the speaker’s laptop and the venue’s projector/screen and then tap a USB feed off the side into my laptop.

Audio: Ankerwork M650 Wireless Microphones

The first time I tried this, I used a RØDE shotgun microphone attached to the camera, but the audio quality wasn’t great, so I invested in a set of AnkerWork M650 wireless microphones. These are wonderful and amazing and excellent and I can’t say enough good things about them, but I’m a big fan of Anker gear, and folks I know who use the RØDE Wireless ME or the DJI MIC system say the same things about their setup.

Recording: Macbook Pro + OBS Studio

I run all these devices into an M1 Macbook Pro, and then record everything using OBS Studio.

Set OBS to record MKV format, and enable “automatically remux to MP4” in Settings > Advanced.

If you record direct to MP4, a system crash will leave you with an unusable video file, but MKV files aren’t as widely supported, so recording to MKV and then automatically remuxing to MP4 gives you the best of both worlds.

This is where the “minimal post-production” bit kicks in. I’ve got four scenes set up in OBS:

Camera Only:

Slides + Camera:

Camera + Slides:

Slides Only:

Plus, before the meetup I’ll make a title slide for each speaker and create a scene with that title slide plus the microphone, which means you can record audio over the title - great for introductions and preamble:

Note that “Slides + Camera” has the slides on the left, and “Camera + Slides” is the same thing, but with the slides on the right. Composition works better if the speaker is facing towards the middle of the frame, so which one of these gets used depends on where the camera is in relation to the speaker on the night.

OK, showtime. Plug everything in, get it working, probably reboot a couple of things because HDMI can be tricksy.

Mic up your presenter, and check the audio in every scene - nothing worse than recording the whole event and finding out later that one of your presents wasn’t recording any sound. Ask your speaker to stand where they’re going to stand, check the camera angle. I clip the other mic on to myself so I can introduce the speaker, repeat questions, etc. and have it come through on the recording.

Then bring up the title slide, hit Record, introduce the talk, and then switch scenes as you go, and all being well, you’ll end up with a MP4 file you can upload straight to YouTube when you get home.

Shopping List:

Canon EOS M200 digital camera (about £500)
USB power + dummy battery kit for Canon EOS M200 (£17.99)
Elgato Camlink (£109.99)
Elgato Game Capture HD60 S+ (£299.99)
AnkerWork M650 Wireless Lavalier Microphone (£179.99)

Cables:

5 metres seems to be the sweet spot for cable length. Too short, and you’re going to be in the way. Too long and you get weird signal problems, or no signal at all.

MicroHDMI > HDMI cable (camera HDMI OUT > Camlink HDMI)
Regular HDMI cable (presenter laptop > GameCapture IN)
USB-C cable (GameCapture OUT > OBS)

Total investment if you’re starting from scratch will come to just over £1200.

How much?!

OK, you don’t need to spend that all in one go. Get a copy of Reincubate Camo and use your phone instead of buying an SLR camera + Camlink. For slides, you can pick up cheap HDMI splitter and an HDMI > USB capture gizmo on Amazon for about £20 each, and see how you go.

I wouldn’t skimp on microphones, though: as they say in Hollywood, “your ears never blink”, and you can probably get away with significantly worse video quality if the audio is up to scratch.

Here’s two clips I recorded using the old camera-mounted RØDE shotgun mic:

and here’s some recorded with the AnkerWork mics:

So, that’s your lot. Good luck with it, and let me know if you found this useful.

Custom Validation Attributes in ASP.NET Core 8

2024-01-24T12:00:00+00:00

ASP.NET Core 8 rocks. It’s fast, powerful, cross-platform… and, yes, includes jQuery.

Chill. jQuery’s fine. It’s solid, it’s proven, it works, and we’re not here to impress hipsters hiring frontend devs for their Web3 startup, we’re here to build stuff that works (and will probably still work next year.)

Out of the box, ASP.NET Core includes jQuery, jquery.validate and jquery.validation.unobtrusive, which together provide client-side validation that integrates really closely with the server-side validation provided by ASP.NET and System.ComponentModel.

Most of the stuff you’ll find online about how to extend ASP.NET Core validation still works, but there’s a few new things in .NET 8 which make it a little cleaner. Specifically, IDictionary<TKey,TValue> in .NET 8 includes a .TryAdd(key, value) method, which means the MergeAttributes helper method you’ll find in lots of examples of custom validation attributes isn’t required any more.

Here’s how to extend ASP.NET with a custom validation attribute that’ll make a checkbox a required field – the classic “You must accept the terms and conditions” scenario.

The attribute itself:

public class MustBeTrueAttribute : ValidationAttribute, IClientModelValidator {

	public override bool IsValid(object? value) => value is true;

	public void AddValidation(ClientModelValidationContext context) {
		var errorMessage = FormatErrorMessage(context.ModelMetadata.GetDisplayName());
		context.Attributes.TryAdd("data-val", "true");
		context.Attributes.TryAdd("data-val-must-be-true", errorMessage);
	}
}

You’ll need to add two lines of custom JavaScript to your pages - I add these to the end of _ValidationScriptsPartial.cshtml.

<script>
	jQuery.validator.addMethod("must-be-true", (_, element) => element.checked);
	jQuery.validator.unobtrusive.adapters.addBool("must-be-true");
</script>

Don’t put this code inside the jQuery onload handler (the one that’s normally wrapped in $(function() { }) – it doesn’t rely on any DOM elements and the call to jquery.validator.unobtrusive.adapters.addBool() has to run before the validation code parses your form.

To use this attribute in your view model:

public class SignupPostData {
	[MustBeTrue(ErrorMessage = "You must accept the terms and conditions")]
	public bool AcceptTerms { get; set; }
}

and then in your Razor view:

@model SignupPostData

<form method="post">
    <label asp-for="@Model.AcceptTerms">
        <input type="checkbox" asp-for="@Model.AcceptTerms"/>
        I accept the terms and conditions
    </label>
    <span class="form-text text-danger" asp-validation-for="@Model.AcceptTerms"></span>
</form>

@section Scripts {
	@{ await Html.RenderPartialAsync("_ValidationScriptsPartial"); }
}

or, if you prefer using the HTML Helper syntax:

@model SignupPostData

@using (Html.BeginForm(FormMethod.Post)) {
	@Html.ValidationSummary()
	@Html.LabelFor(model => model.AgreeToPayment)
	@Html.CheckBoxFor(model => model.AgreeToPayment)
	@Html.ValidationMessageFor(model => model.AgreeToPayment)
	<input type="submit"/>
}

@section Scripts {
	@{ await Html.RenderPartialAsync("_ValidationScriptsPartial"); }
}

Choosing a Chat App For Your Tech Event

2024-01-18T12:00:00+00:00

WhatsApp, Telegram, Slack, Discord, Signal, Teams… seems like every event you go to these days has at least one “official” chat app, and probably at least half-a-dozen unofficial chat groups set up by attendees and speakers.

If you’re running a conference or seminar, should you set one up? And if so, which one should you use?

Let’s get the inevitable out of the way first: even a small community conference is going to attract a few dozen people, and there’s no way they’re going to agree on which chat app is best. Microsoft Teams is great when all the people in it work for the same company; for guests, even signing in can be a showstopper. For everybody who likes WhatsApp because it’s ubiquitous, there’s somebody who refuses to install it because they won’t use Meta products. For everybody who likes Slack because it’s already installed on their company laptop, there’s somebody who hates Slack because of its convoluted authentication system (hi!). For everybody who loves Discord because it’s where all their friends hang out, there’s somebody who can’t use it on their laptop because corporate IT think it’s only for gamers and crypto.

That said, having a chat system for folks participating in your event is an excellent idea. For first-time attendees who don’t know anybody, it’s a great way to connect with people, find out where people are having drinks or meeting for dinner. It’s an easy way for organisers to share updates and last-minute agenda changes.

If you’re running coding workshops as part of your event, setting up a dedicated channel for each workshop works brilliantly for sharing code snippets, URLs and diagrams. When I’m teaching workshops, I always ask the organisers to set up a chat with all the attendees, using something people can run on their laptops – there’s no point sharing 25 lines of JavaScript with somebody if the only place they can see it is a WhatsApp notification on their iPhone.

So, here’s a few guiding principles I’d encourage you to consider if you’re creating a chat for your event.

First, ask yourself: are you supporting a physical event, or creating an online community?

It’s lovely to think that people can use your chat to keep in touch after the event, but they don’t need your event chat to do that. Social media exists. Attendees can spin up their own chats and groups any time, on whatever platform they want to, and keep the conversation going there – somewhere where it isn’t your responsibility to answer questions, provide event support, and potentially enforce your event’s code of conduct. Doing those things well for three days is hard enough; do you really want to do it all year round, for free?

Online communities are inclusive: anybody can join, just send them the link. In-person events are exclusive: you’ve actually got to show up. If you’re not there, you’re not part of it. Sure, there’s a huge amount of overlap, but the kind of chat that happens when twenty people are together in an unfamiliar city trying to find a good sushi place is not the same as the chat that happens when those same twenty people are back home, spread across a dozen timezones, and somebody’s asking if anybody knows a good way to show Git commits in PowerPoint.

Consider creating a workspace just for your event: one of the big advantages of creating a new instance/server/workspace/whatever for each event is that it’s easy to invite the right people, and easy to shut it down when you’re done.

Use channels sparingly

You probably need a general chat, a speaker chat, an announcements channel, and if you have separately ticketed events like workshops or seminars, create a channel for each of those. Don’t create fifty different channels for web, JavaScript, ethics, .NET, IOT, crypto, Java: you’ll overwhelm people with choice, and when they see most of those channels are empty and nobody’s posted anything, they’ll lose interest.

Use familiar tools

Online platforms like Pine and gathertown serve a purpose, but the majority of your participants won’t have used them before, which makes them far less likely to get involved in discussion. Stick to something people already know.

During the COVID lockdowns when many events went entirely online, I saw far more engagement and participation during events that used Slack and Discord than I did in events relying on proprietary platforms - we’re talking thousands of chat messages over a few days, vs fewer than a dozen on some of the dedicated event platforms. Familiarity is important.

Encourage your team to get involved

Folks will use chat to ask questions, ask for help, ask if anybody has an extension cord or a Macbook charger or where the vegetarian food is served at lunchtime. Make sure your event staff are around to give helpful answers.

Don’t rely on it if something’s urgent

One of the things I do before I give a talk is to put my phone in airline mode. You ping me on WhatsApp five minutes before I give a presentation? I’m not going to see that until after I’m done. If you need to talk to somebody urgently, call them – and if that doesn’t work, go and find them.

Shut it down when you’re done

This is the one that I personally find the most frustrating. I’m still in Slack workspaces for events that took place in 2020, and like the Hotel California, I can sign out any time I like but I can never leave. Whenever I set up Slack on a new device, they’re all still listed there. I have Telegram chats and WhatsApp groups and Signal chats for old events… and, once in a while, somebody will try to get hold of me via one of those chats. You know if you archive a WhatsApp chat it doesn’t tell other people you’ve archived it… so they have no idea you’re not seeing their messages?

Go on. Delete your old event chats - and I mean delete. Gone. Forever.

Think how much better you’ll feel.

If it was up to me?

Set up a Slack a few months early. Use it to coordinate programme committee, crew, volunteers. Create an announcements channel, a speaker channel and a hallway track for general chit-chat. A few days before you open the doors, invite everybody who’s going to be there. Keep it open for a week or so after the event. Post an announcement that you’ll be shutting it down, give folks the chance to share their LinkedIn, Twitter, email - and then delete the workspace.

Fixing 'Version conflict detected for Microsoft.CodeAnalysis.Common'

2024-01-17T12:00:00+00:00

I’ve been upgrading projects from .NET 7 to .NET 8, and mostly it’s been remarkably smooth: edit the .csproj files, replace net7.0 with net8.0, then upgrade any 7.x NuGet packages to their 8.x counterparts, and so far, it’s just worked.

Today I got a weird error message:

error: NU1107: Version conflict detected for Microsoft.CodeAnalysis.Common. Install/reference Microsoft.CodeAnalysis.Common 4.5.0 directly to project Rockaway.WebApp to resolve this issue.

error:  Rockaway.WebApp -> Microsoft.EntityFrameworkCore.Tools 8.0.0 -> Microsoft.EntityFrameworkCore.Design 8.0.0 -> Microsoft.CodeAnalysis.CSharp.Workspaces 4.5.0 -> Microsoft.CodeAnalysis.Common (= 4.5.0)
error:  Rockaway.WebApp -> Microsoft.VisualStudio.Web.CodeGeneration.Design 7.0.10 -> Microsoft.DotNet.Scaffolding.Shared 7.0.10 -> Microsoft.CodeAnalysis.CSharp.Features 4.4.0 -> Microsoft.CodeAnalysis.Common (= 4.4.0).

I’m a big fan of error messages that tell you how to fix the error which caused them, like “Install/reference Microsoft.CodeAnalysis.Common 4.5.0 directly to project Rockaway.WebApp to resolve this issue.” in this one… except, if you look at the notes for Microsoft.CodeAnalysis.Common, it quite clearly says:

“Do not install this package manually, it will be added as a prerequisite by other packages that require it.”

Here’s what’s actually going on.

Before upgrading, my project uses two NuGet packages, both version 7.x because they shipped with .NET 7:

Microsoft.EntityFrameworkCore.Tools 7.0.12
Microsoft.VisualStudio.Web.CodeGeneration.Design 7.0.10

Both of these packages have an indirect dependency on Microsoft.CodeAnalysis.Common 4.4.0 – this doesn’t appear anywhere in the .csproj file, so I’m guessing it’s controlled by this line:

<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>

So you try to upgrade Microsoft.EntityFrameworkCore.Tools to 8.x and it fails, because that package requires Microsoft.CodeAnalysis.Common 4.5.0, but your project includes Microsoft.VisualStudio.Web.CodeGeneration.Design 7.x, which requires Microsoft.CodeAnalysis.Common v4.4.0

The solution that worked for me was to remove the other package (v7.x), upgrade EF Core Tools, then install the other package (which installs the latest version, 8.x):

dotnet remove package Microsoft.VisualStudio.Web.CodeGeneration.Design
dotnet add package Microsoft.EntityFrameworkCore.Tools --version 8
dotnet add package Microsoft.VisualStudio.Web.CodeGeneration.Design

The Hang of Thursdays #7

2023-08-17T12:00:00+00:00

Hey folks. So… you know the thing where you agree to do something, because it’s literally months away, and you’ll totally have time to get it all prepared by then — and then somehow it’s not months away any more, it’s two weeks… and it’s not one thing, it’s about five things?

Welcome to my week! I’m updating the course materials for the workshop I’m giving at the Copenhagen Developers Festival, all about how to build web apps with C# using all the amazing new shiny stuff that ships with .NET 7. (Did I mention that tickets are still on sale, and as well as the workshops there’s an amazing programme of talks, shows and live entertainment?)

I’m also figuring out how to fit Guitaraoke on an aeroplane to take it to Denmark, preparing a new talk about how I created Guitaraoke (including a bit about how to fit it all on an aeroplane and take it to Denmark) — oh, and planning my 45th birthday party, which is not only going to be awesome, but is also going to be the public beta test of Guitaraoke: Portable Edition, including a bunch of changes to the software, which I’m going to do on ** Saturday morning, about eight hours before the show.

So, yes. Life is a little crunchy, and for some reason in the middle of all this I’m writing a newsletter to tell you all about how crunchy it is.

I may have chosen… poorly.

Moq, Sponsorlink, and Open Source Funding

If you follow any of the .NET content on Twitter, Reddit, Github, or just about any online community with a strong .NET presence, you’ll have caught last week’s drama about Moq.

If you find reading about open source makes you angry, please take a moment to contemplate this otter. Look at her little otter hands! Aaah… isn’t that better? Photo © Ondrej Chvatal / Shutterstock.

Whatever you think about the specifics of this scenario, it got a lot of people talking about open source funding again — and that’s a good thing, because open source is completely broken. Companies like Meta, Google, Apple — not to mention your bank, your airline, your car, and every single app you have installed on your smartphone — use open source software. It’s a fact of modern software development. But they’re not giving anything back. These companies use open source libraries to create products that turn over hundreds of millions of dollars in revenue, and the developers who create and maintain those libraries are struggling to make ends meet.

David Whitney wrote about this way back in 2021, in his fantastic essay “Open Source Exploitation”, and he recorded a .NET Rocks episode on the same topic with Carl Franklin and Richard Campbell at NDC Oslo last year. SixLabors, the creators of the excellent ImageSharp library, have been similarly vocal — and eloquent — about making license changes they consider necessary to create sufficient sustainable income to support the development of their product.

I’ve been involved with open source since the 1990s, and one of the most consistent tenets of open source evangelism is the principle that if the project maintainers do something you don’t like, you can keep using the code you’ve already got. That’s one of the key value propositions of using open source software in the first place: it gives you control. Nobody can take it away, nobody can stop you using it, and nobody can force you to upgrade to a version you don’t want to use.

The folks who created Moq owe you absolutely nothing unless you are in possession of a commercial support contract that says otherwise. It’s open source. You don’t like what they did with 4.20? Fork 4.19. Maintain that fork. It’s yours now; you can do whatever you want to with it. You need a feature? Build it. You find a bug? Fix it.

The people saying we all need to remove Moq from our code right now? How about just… don’t upgrade? Nobody’s forcing you to. If Moq 4.19 solved your problems last month, Moq 4.19 will still solve your problems this month, and next month. Besides, I’d wager good money that the people obsessing about a minor point upgrade are probably the same people who are still running .NET Framework 2.0 in production.

This is a topic I’ll come back to in a future article, but for now, I wanted to leave you with a number to think about. The creator of Moq, Daniel Cazzulino, aka kzu, wrote a great article in February in which he compared open source licensing models to music subscription platforms. I thought this was an interesting idea, and I wondered what the numbers would look like.

NuGet has a page at https://www.nuget.org/stats/packages where they publish the total package downloads for the past 6 weeks. At the time of writing, Moq is the 12th most popular package on NuGet, with 19,992,598 downloads.

Spotify pays around $0.003 per stream.1

If NuGet paid as well as Spotify, Moq would have earned $63,576.46 in revenue in the past six weeks. That’s over $10,000 per week.

Just putting the number out there.

London .NET September Meetup

The next London .NET meetup is going to be on September 13th, at Accurx’s office in Shoreditch.

We’ve got Lotte Pitcher coming along to give us an introduction to Umbraco for .NET developers. Umbraco’s the leading content management system (CMS) on the .NET platform; it’s free, it’s open source, and it’s supported by one of the most committed developer communities I’ve ever seen. Lotte’s going to give us the lowdown on all the latest Umbraco features, like headless content management and EF Core support, as well as everything a .NET developer needs to know to get started with the latest release.

Our second guest speaker is David Whitney — yes, the same one — talking about how he built a Gameboy emulator, for fun, in C#. I’ve seen a version of this talk before and I absolutely loved it: building an emulator means tackling some properly hardcore tech problems, and David’s an excellent speaker who always manages to strike a great balance between engaging storytelling and gritty technical detail.

If either or both of those sounds like your kind of thing, registration’s open now — sign up and come along!

https://www.meetup.com/london-net-user-group/events/295373351/

ULTRARAM

So, this week I learned that (1) there’s a Flash Memory Summit in Santa Clara, (2) it has an awards ceremony, (3) one of the awards is for “Most Innovative Flash Memory Startup”, and (4) it’s just been won by something called ULTRARAM, invented at the University of Lancaster, which sounds like it could be an absolute game-changer.

We’ve always had to choose between two kinds of digital storage: volatile and non-volatile. Volatile memory — DRAM — is fast, but it’s expensive, and it only works as long as it’s drawing power. You turn off the power, anything you haven’t saved to disk is gone forever. Non-volatile memory — flash memory, SSDs, mechanical hard drives — is orders of magnitude slower, but it’s also orders of magnitude larger, and doesn’t require power to remember what’s stored on it. ULTRARAM sounds like the best of both worlds: high-capacity, non-volatile memory that’s as fast as DRAM.

It’s early days yet, but if it turns out to be viable, ULTRARAM would mean we could build devices that drew literally2 zero power while they were in sleep mode, and woke up instantly when you pressed a key or moved the mouse.

Their official site is at ultraram.tech, and Dave James has a good write-up over at pcgamer.com.

This week I’ve been…

Reading: “The Calculating Stars” by Mary Robinette Kowal. There’s something wonderful about finishing a book by an author you’ve not read before, and that you thoroughly enjoyed, and then discovering there’s all this other stuff they wrote — and so, after finishing “The Spare Man”, I’m working through her back catalogue. “The Calculating Stars” is the story of Dr. Elma York’s endeavours to become the first woman in space, set against an immaculately realised alternative history of the United States of America in the mid-20th century.

Listening to: “Drive” by The Defiants, featuring current and former members of Danger Danger. “Drive” is apparently their third album — I honestly had no idea they existed until a review popped up in one of my social media feeds earlier this week, so I looked them up on Spotify, and I’ve had this album on repeat ever since. It’s not challenging, it’s not progressive, it’s not revolutionary — it’s just a big fat slab of very good, very slick, 80s-inspired melodic hard rock, and in a few places Paul Laine’s vocal sounds uncannily reminiscent of Jon Bon Jovi circa “New Jersey”.

Watching: Nothing. Seriously. Which is a little weird… when I decided to include a weekly rundown of my media habits, I honestly didn’t think there would be weeks when I watched literally zero minutes of film or TV, but here we are; there’s not been a whole lot of downtime this week and what little I’ve had has been spent reading books and playing guitar. Mind you, last year when I had COVID I watched the whole of Stranger Things in a week — I binged season 1 on Monday, season 2 on Tuesday, and so on, because I wanted to know why everybody was going on about “Master of Puppets” — so I guess it all averages out in the end.

And that’s a wrap. Have fun, take it easy, be excellent to each other (yes, especially when talking about open source on Reddit), and catch you next week.

Dylan

According to producerhive.com, Deezer pays $0.0011, Apple pays $0.008 and Tidal pays a whopping 1.28 cents per stream.

Yes, literally, as in literally literally.

The Hang of Thursdays #6

2023-08-10T12:00:00+00:00

Hey folks. I’m still down in Camber, on the south coast of England, but the weather this week has cheered up significantly; blue skies, sunshine, a nice gentle breeze… which means it’s been a perfect week to sit at a laptop doing workshop prep, booking travel for the autumn conference season, designing new stickers and swag, and generally being a bit of a nerd.

Dr. Timezone and the Airport Mayhem: Autumn 2023 Edition

Europe kinda shuts down over the summer. When I first set up my own company back in January 2020 (yeah, I know, timing, LOL), I talked to a lot of folks I knew who’d done something similar, and one thing that really stuck with me was a great bit of advice from Kevlin Henney:

“Don’t worry too much about getting bookings in July and August, because even if the person who wants to hire you isn’t on holiday, the accountant who’s supposed to pay your invoice probably is.”

That’s turned out to be great advice so far: budget for two quiet months in summer (and two quiet months over Christmas), and just accept you’re not going to be doing much. In fact, one of the things I enjoy most about being self-employed is the absence of busywork. When I’m busy, I’m busy — but when I’m not? No big deal. Chill out, design some stickers, write a few songs, do a little carpentry, and only occasionally wake up in a cold sweat at 4 am wondering what I’ll do if things never pick up again. But so far, they always have.

Here’s what picking up again looks like. On Thursday 24 August I’m flying to Denmark. The Linebreakers have booked a long weekend in Rågeleje, on the north coast of Zealand, where we’ll get to actually do some proper rehearsal for a change. Then on Sunday we head to Copenhagen for the CPH Developers Festival. Two days of in-person workshops about full stack web development with C# and .NET, followed by three days of conference/festival. Home for six days, then off to Nottingham, where I’m running a speaker workshop for the folks who have been selected to speak at DDD East Midlands this year. I love DDDEM; it’s a fantastic conference run by awesome people, and offering this kind of opportunity to their speakers is just one of the many, many ways they’re helping to support the developer community.

I’ll get home from Nottingham late on Saturday night, and have a day at home. Monday 11 September I’m on a Eurostar to Rotterdam and then the Intercity to Utrecht, where I’m a keynote speaker at the Tweakers Developer Summit. Then I’m heading back first thing on Wednesday morning to host the London .NET User Group meetup.

The following week I’m at Digit 2023 in Tartu, Estonia, where I’m presenting a keynote, running a seminar, and doing the music at the after-party. I spoke at Digit last year and had a great time, and Estonia is a wonderful country to visit. The only slight complication is that, although Tartu does have an airport1, as of September 2022 it doesn’t have any scheduled commercial flights — so getting there involves flying to Tallinn via Helsinki and taking a local train.

Two days at home. Back on Eurostar to Brussels, onwards to Antwerp to run a day of speaker training with the crew from Axxes, then on to Amsterdam for DrivUn on 27 September, where I’m doing a keynote and the Linebreakers are performing at the conference party.

After that, we have a Schrõdingerian bit of scheduling where I might be coming home or I might be going to Malmõ2. Then from the start of October through to December I’m doing a talk at the University of Sussex, DDD East Midlands in Nottingham, a talk at the .NET group in Birmingham, then I’m off to NDC Porto in Portugal, CodeCamp Iasi (pronounced “yash”) in Romania, XPand 2023 in Jordan, probably another trip to Copenhagen, BuildStuff in Lithuania, and then at the start of December I’m doing the YOW! Tour of Australia: back-to-back conferences in Melbourne, Brisbane, and Sydney, and a few days in Perth because it turns out I know a surprising number of excellent people who live there.

There’s about 50 hours of Zoom training and online stuff sprinkled in there, too, and that’ll probably be a wrap for 2023… but hey, if you see a gap somewhere that lines up with your event (or, even better, you want to book me to run a workshop or give a talk in a country I’ll already be visiting!), get in touch.

On a completely unrelated note, my latest hot startup idea is a service where you drop off your suitcase full of dirty laundry when you land at your home airport, tell them what your next flight is, and they unpack, wash, dry, repack, and check your suitcase in for your next flight for you. If anybody wants to invest £50 million in my startup, I promise to think about really hard for a few months and then explain why it didn’t work.

New Stickers

I love parodies, I love stickers, I love rock bands… and I love banging things together to see what happens. Here’s the latest from the bit of my brain that likes to play with Adobe Illustrator:

Welcome to the jungle, baby, we’ve got video games.

I’ll be ordering a bunch of these designs from the lovely people at StickerApp very soon, along with a few other new designs and fresh batches of perennial favourites like the “certified Rockstar developer” stickers. For now, the only way to get them is in person, so come along to one of the events I’m at and ask nicely — but I’m hoping to have some kind of online ordering available later this year, for all you folks out there who think a set of parody rockstar developer stickers would make an ideal Christmas gift.

Incidentally, the “rose shaped like a zero” that appears in this design marks the first time I’ve ever used AI in any of my sticker artwork; it was generated by Midjourney. Asking for a “digit one number 1, steel, gun metal, polished metal, photograph”, and countless other variations on the same theme, didn’t even come close to producing anything usable, so the rest is 100% good old organic human creativity.

Want this in your inbox? Sure you do. Then you can read it in your email and it’ll look like you’re working! Go on. Subscribe. It’ll be fun.

This week I’ve been…

Watching: Honestly? Not a lot, unless walking on the beach watching juvenile herring gulls counts as watching. They’re very cute, and extremely squeaky.

“Mum! Mum! I want chips. Can we get chips? I’m bored. I want a snack! MUM! That man has chips. I’m gonna steal his chips. MUM WATCH THIS… Mum?”

Listening to: One from the vaults this week: “Dead Air for Radios” by Chroma Key. I was a huge Dream Theater fan in the 1990s — or perhaps it’s more accurate to say that I’m still a huge fan of the music Dream Theater released in the 1990s; “Awake” is one of my favourite albums of all time, and a big part of that is down to the musical influence of their original keyboard player, Kevin Moore.

Kevin left the band after releasing “Awake”, and “Dead Air for Radios” was his first solo project, released under the band name Chroma Key. It’s an album of textures: haunting, lyrical soundscapes, stark piano over rich waves of ambient synth; the sound of a man who walked out of one of the most promising prog metal bands in history, sold his belongings, packed what was left into a station wagon and hit the open road.

Reading: “The Spare Man” by Mary Robinette Kowal. Another one from this year’s Hugo Awards shortlist, it’s a rollicking good murder mystery disguised as science fiction. There’s enough hard science here for the premise, a murder on board a passenger spacecraft in transit between Earth and Mars, to ring true. The fun, though, is in the characters, the plot twists, and the occasional sense that you’re enjoying a literary cocktail that’s one part “Spaceship Medic”, one part “Murder on the Orient Express”, and a generous dash of something you can’t quite place but it’s really rather good.

And that’s a wrap, folks. Take it easy, have fun, and be excellent to each other.

Dylan

1 The international airport code for Tartu is TAY, and before Finnair shut down the only commercial route late last year, you could fly from TAY to HEL and back.

2 A certain tolerance for uncertainty is very much a prerequisite for this kind of job… it turns out that the most mysterious part of being an International Man of Mystery is never knowing whether a provisional booking is actually going to happen or not.

The Hang of Thursdays #5

2023-08-03T12:00:00+00:00

Hello! This week’s THOTH is coming to you from Camber, a tiny village on the south coast of England, where I’m staying at my parents’ house for a couple of weeks while they’re away. This week so far has been torrential rain and 40 mph winds — aka “traditional English seaside holiday weather” — so the usual Camber activities of walking on the beach, swimming in the sea and cycling into Rye to enjoy its many excellent pubs have been postponed in favour of sitting indoors poking the internet.

The Framework 16 laptop

My main travel laptop these days is a Framework 13. It’s a fantastic machine that I love dearly, and I was really excited when Framework announced earlier this year that they’d be launching a 16” modular laptop. Looks like I wasn’t the only one: the Framework 16 opened for pre-orders last week, and the first batch sold out in less than 12 hours.

substack.dylanbeattie.net is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Sean Hollister at The Verge has a great write-up about what makes it special — and how Framework are hoping to succeed where so many other brands have failed in delivering the first laptop that genuinely supports a modular, replaceable gaming GPU.

I haven’t ordered one yet… but I know a few folks who have, and I suspect that after I get my hands on one of theirs to try out, it won’t be long before I’m putting in an order for my own.1

Cree# and Ancestral Code

This has been around for a few years, but I’d never heard of it until somebody pinged me a link last week: Jon Corbett has created a programming language based on Cree, an Indigenous language spoken by around 120,000 people across Canada. Cree# isn’t just another esoteric language, though; it’s been explicitly created to help preserve the Cree language and culture. As Jon puts it:

“My primary target communities here are Cree communities that are looking for new (and exciting) ways to encourage students (especially in the K-12 grades) to use their heritage language as much as possible, and resist using English as their primary language.”

Jon Corbett’s design for a keyboard (Creeboard?) based on the Cree star chart, from https://esoteric.codes/blog/jon-corbett

Cree#’s syntax and conventions reflect the visual art and storytelling traditions of the culture that inspired it, and it’s not quite like anything I’ve seen before.

Interview with Jon Corbett at esoteric.codes
Cree# on Hacker News (April 2021)

Fun with Printers

For somebody who works with code, web pages and digital media, I print a lot. I use printed notes and schedules when I’m running online workshops, I rely on printed checklists when I’m packing for conference trips and gigs, and I print proofs of my sticker designs and other odd bits of artwork.

Printers get a bad rap. Sure, you can buy a colour inkjet printer for £50. That’s ridiculously cheap. Printers are big and they’re packed with moving parts; you can’t sell a printer for £50 and make a profit. No, a £50 printer is a loss leader. A £50 printer is a way to get you to spend another £50 on cartridges every few months — cartridges that have microchips in them, which in some cases have been used to store encryption keys so that anybody trying to reverse-engineer them to create compatible low-cost replacements can be prosecuted under the Digital Millennium Copyright Act.

Last month, my printer died. Something something print head won’t align beep beep no more prints for you. It was a Canon Pixma iP4900 that I’d salvaged from an office clear-out nearly 10 years ago; apart from costing me a small fortune on ink cartridges over the years, it was fantastic.

My new printer’s also a Canon. It’s a Canon Pixma G6050. It didn’t cost £50; it cost £240. It’s an all-in-one printer, scanner and colour copier; it’s got built-in wired and wireless ethernet; but most important of all: it doesn’t use cartridges.

If you really want, you can mix all the inks together to make Brown Flavour!

See that? Bottles of ink. Just ink. No microchips, no DRM. Just a couple of big old plastic tanks full of ink. I particularly love that instead of angry messages about how it can’t print because it’s out of cyan, there’s just a message in the UI that says “Hey, ah, please check there’s ink in the tanks before printing and if there isn’t then maybe don’t print right now?”

It’s been a month, and I’ve printed about 150 pages; so far I’m delighted with it, and I’m sure that when I eventually have to fork out the princely sum of £19.99 for a full set of ink refills I’ll be even more delighted.

NDC London Call for Papers

NDC London, one of my favourite conferences, will be back in January 2024, and the Call for Papers is open until 1 September. If you’d like to be part of it, this is your chance. I’m on the programme committee; we’re looking for talks on everything from embedded systems to ethics to web assembly to language models. What are you and your team working on? What problems are you trying to solve? How did you get there?

NDC London Call for Papers at Sessionize

I wrote a blog post a while ago about how the selection process works, which is still very much relevant, so maybe take a look at Why Your Talk Wasn’t Picked For NDC London before submitting anything.

Kevin Mitnick RIP

I was sad to see the news that Kevin Mitnick has died from pancreatic cancer. Mitnick was probably the world’s most famous computer hacker; his exploits were legendary in the fledgling infosec community when I was an undergraduate back in the 1990s, and following his release from prison in 2000 he reinvented himself as a security consultant.

My favourite Mitnick story has nothing to do with computers. As a teenager, he noticed that when the buses in his home town of Los Angeles broke down or were taken out of service, passengers would be issued with transfer tickets so they could switch to another bus and continue their journey. Mitnick found a stack of unused transfer tickets in a dumpster near the bus station, and realised that if he could find a way to validate them, he could ride the buses for free — so he told staff he was working on a school project about transportation and wanted to buy a ticket punch for his project, and voilà! Free bus rides. I always thought that was a particularly elegant hack: observe the system, find the loopholes, wait for a lucky break, and round it all off with a little social engineering.

Want this in your inbox? Sure you do. Then you can read it in your email and it’ll look like you’re working! Go on. Subscribe. It’ll be fun.

This week I’ve been…

Watching: Barbie — at the first completely sold-out cinema showing I’ve been to since 2019. It’s good. It’s very good. Charming, visually stunning, and laugh-out-loud funny.

Special shout out to the most unlikely credit you’re likely to see in a motion picture soundtrack any time soon, for “I’m Just Ken” — written by Mark Ronson, performed by Ryan Gosling, and featuring Wolfgang van Halen and Slash. Now that’s a supergroup I’d pay good money to see in concert.

Listening to: “Wild in the City”, the new single from Nitrate. Founded by songwriter Nick Hogg, Nitrate are a melodic rock band from Nottingham whose last album “Renegade” has been top of my Spotify playlist for months. Slick, polished 80s-style rock, luscious waves of shimmering synthesizers and screaming guitars, and some seriously catchy hooks.

Reading: “DOOM Guy: Life in First Person” by John Romero. John’s autobiography is as much the story of id Software as it is the story of his own life, from the infamous Super Mario PC demo to Doom, Quake, Daikatana and beyond. I think it’s fair to say that John is better at writing code than he is at writing prose; there’s a wealth of detail here, but the writing can get a little clunky from time to time. Nevertheless, John remains one of the world’s most celebrated programmers, and the story of how a bunch of long-haired D&D fans revolutionised the gaming industry is compelling.

And that’s a wrap, folks. Take it easy, have fun, and be excellent to each other.

Dylan

1 It’s my eyes, you see. 13” screens just aren’t quite big enough for all the really important programming work I have to do. Honest.

The Hang of Thursdays #4

2023-07-28T12:00:00+00:00

Grüße aus Berlin! I’m in town for the WeAreDevelopers World Congress (WADWC), so this week’s newsletter is all about what’s happening, who’s here, and what they’re talking about. And, yes, it’s Friday. I know. Relax. It’s fine.

This event is huge — somebody at the speaker dinner last night said there were 12,000 people here, and having been on site for two days, I can easily believe it. Fifteen stages, hundreds of speakers and sessions, two expo halls; it’s so big it fills both buildings of the Berlin Messe convention centre, and outside in the plaza area between the building there are more stages, food trucks and a few exhibitor stands. It’s absolutely colossal, and I can’t hope to do more than scratch the surface… but here are the bits I did manage to see.

Tim Berners-Lee and the Future of the Open Web

The big name on this year’s programme at WADWC is Sir Tim Berners-Lee, inventor of the World Wide Web, whose opening keynote was an engaging riff on the history and future of the open web. Tim’s main research project for the past few years has been SOLID, an initiative to add authentication, identity management and standardised data APIs to the web, and he talked about the challenges of decentralising identity, the problems with the data silos that have emerged post web-2.0 — “If you want to share a LinkedIn post with your Twitter followers, you can’t; you have to create a Twitter account to do that” — and the kinds of apps that we could build on top of decentralised identity and authentication.

He also talked about AI, which is very much a recurring theme at WADWC this year, and in particular the idea that we should have access to AIs that work for us — “the way your doctor works for you, or your lawyer works for you; they have access to your private data, and you trust them not to share it.”

The keynote was followed by a “fireside chat” session with Tim and Sead Ahmetovic, the CEO of WeAreDevelopers, about the web, AI, and the career challenges which developers face in 2023. A couple of quotes really stood out for me. On whether he’d do anything differently if he was creating the web over again, Tim said:

“I used the domain name system, and at the time it was managed by people like Jon Postel, and managed in a very responsible way; it wasn’t just a fighting ground for investors. So maybe that was a mistake, or maybe it’s something we can fix.”

I love DNS, and I think it’s a fantastic system that works incredibly well; I’ve always thought the design of DNS was one of the key factors behind the success of the web, so it was interesting to hear that the creator of the web thinks DNS was a weak point.

On whether developers should generalise or specialise:

“From the point of view of running open source projects, as the CTO of a company that needs talent: it’s good that there’s a variety of talent. We need generalists, we need specialists, we need T-shaped people, or pi-shaped people, where you’ve got two deep specialities.”

That’s the first time I’d ever heard the phrase “pi-shaped”, and I think it’s delightful.

The quote that stuck with me the most, though, was when he was asked about social and ethical responsibility:

“I suppose… one way to think about it is when you write your line of code, when you write a function: work local, but think global. Code it as though you are creating part of a huge, wonderful, very powerful system, that people are using to build effective democracy, effective science, to allow society to function much better than it does.”

I loved that.

John Romero on Creating Doom

The afternoon keynote speaker was John Romero, sharing the story of how Id Software created Doom in 12 months.

It’s no exaggeration to say that Doom revolutionised computer gaming. It introduced the world to multiplayer network gaming. It pioneered the free-to-play model by making episode 1 a free download, back in the days of dial-up bulletin board systems. The 3D engine was more sophisticated than anything we’d ever seen before, the music still triggers pangs of nostalgia in an entire generation of gamers — and the word “deathmatch” was created to describe Doom’s competitive multiplayer mode.

John Romero at WeAreDevelopers World Congress 2023

I’ve geeked out over the history of Id Software before; my talk “The Web That Never Was” even has a whole alternate-history riff on what might have happened if the two Johns had been hired by Nintendo after they ported Super Mario to the IBM PC. Even so, I learned a few things watching John’s talk that I’d never heard of before, like the story of John Carmack walking through the snow to the bank to withdraw $11,000 in cash, so he could pay cash-on-delivery for his new NeXT workstation.

John also revealed that 20th Century Fox offered Id the “Aliens” licence — cue a few astonished gasps from the audience — “but it only had aliens; there weren’t any demons. We talked about it for, like, half an hour, and decided no”.1 We also learned that Carmack’s decision to use BSP trees as the game’s main data structure came after a sunken octagonal staircase caused a recursion bug that slowed gameplay down to a crawl and sent Carmack digging through computer science papers looking for a solution.

Other Highlights

I caught Gift Egwuenu’s session “Going Beyond Passwords: The Future of User Authentication”, which was a great overview of the history of password-based security, what’s wrong with it, and the current state of authentication — single sign-on, multifactor auth systems, and how security standards like FIDO fit into your system’s authentication flow. I “met” Gift at a Zoom workshop I ran back in 2020, when we were all doing everything online; she’s been doing all sorts of awesome things, so it was great to finally meet her in person.

Stack Overflow has a huge presence here. They sponsored the speaker dinner/reception event on Wednesday night, and they’re taking advantage of WADWC to announce a whole load of new features — basically their vision for incorporating generative AI into the Stack Overflow platform. I must say that Prashanth Chandrasekar’s product announcement session didn’t really grab me, but then later Prashanth returned to the stage alongside Joel Spolsky (you know, Joel on Software, founder of Stack Overflow, FogBuz, CityDesk, etc.) and Sead Ahmetovic for a Q&A session about the past and future of Stack Overflow. One comment that stood out for me was Joel talking about embracing the idea of Google as a user interface:

“When we built Stack Overflow, we knew that Google was going to be the UI. We weren’t expecting people to come to Stack Overflow; we expected them to Google their problem and find the answer on Stack Overflow. Now, ChatGPT, and to a lesser extent Copilot, have become the frontend to Stack Overflow’s knowledge base. You don’t just get the answer, you get the answer modified to suit your codebase, ready to copy and paste into your solution, and I think that’s great; it makes life so much easier for developers.”

They talked a great deal about trust as well. Prashanth shared the statistic that, according to the most recent Stack Overflow developer survey, “70% of developers want to use AI tools, but only 40% of them trust AI”. He discussed the work they’ve been doing to close that gap, particularly with regard to sources and citations so a developer can find out exactly where the AI found the solution to a particular problem.

Personally, I’m optimistic about it all. Code reuse is clearly a good idea. Prashanth used the example of parsing phone numbers, which I thought was great: as a user, it sucks using platforms that keep rejecting your phone number as invalid, but it also doesn’t make sense for every dev team out there to build their own algorithm for parsing and validating phone numbers. We’ve had healthy package management ecosystems for long enough now to have a pretty good idea of what they can do, and there’s clearly still a whole class of problems out there that we don’t want to solve from scratch every time, but that don’t really lend themselves to importing a package or standard library.

The challenge, of course, is that AI is easily capable of producing an infinite amount of crap, and if we train the next generation of language models on the SEO clickbait produced by the previous generation, we’re in a race to the bottom in terms of quality. We’ll see how we go.

Rocking in the rain

And then there’s my bit — two live sets of developer comedy rock’n’roll. I was playing on the Airstream Stage, a converted Airstream trailer outside next the food trucks and beer tents… which was absolutely fantastic until it started raining about an hour before I was due to go on… but you know what? I’m British. I’ve survived dozens of English summers and five Glastonbury Festivals, and it takes more than a bit of rain to stop the show.

I think I figured out where “Wet Wet Wet” got their name.

Thankfully, the audience agreed. Somebody even came up to me afterwards and said “you got our lead developer to sing along — we didn’t think we’d ever see that!”

On stage with Hila Fish and Keren Kenzi at WeAreDevelopers World Congress

Huge thanks to everybody who came along — and especially to Hila Fish and Keren Kenzi for some outstanding (and unrehearsed!) guest vocal performances. And a shout out to Framework, whose 13” laptop turns out to be mostly showerproof as well as fast, light, modular and generally awesome.

And that was that, apart from a few well-earned beers and the obligatory post-gig currywurst and Fritz-kola.

Time to grab lunch, say hello and goodbye to a few people, and head to the airport.

Catch you next time.

Dylan

A few years later, the “Alien Quake” community mod turned the Quake engine into aliens vs space marines, and did a much better job than the official Alien vs Predator game.

The Hang of Thursdays #3

2023-07-28T12:00:00+00:00

Coding Streams on Twitch and YouTube

I spent a substantial chunk of last week building a web app for running guitar karaoke nights. I streamed the whole thing live1, which means if you really want to, you can watch almost the entire development of the app on video — all 19 hours and 31 minutes of it. (And for the record: I ran it on Saturday night, and it worked flawlessly. There were a few features I’ll add for next time, and I definitely need to add some kind of security to make sure we don’t get internet weirdos signing up and messing with the stats, but it worked. Which is nice.)

I really enjoy those kind of live-streaming coding sessions. It’s fun, I enjoy the folks asking questions in chat, but I also find that knowing there’s even a handful of people out there watching what I’m doing really helps my focus.

I’d like to do a lot more streaming, but averaged over, say, a year, I probably only spend about 30% of my working time actually writing code — and at least half of that is stuff I’d never stream because it’s full of customer API keys and other stuff I don’t want going anywhere near a Twitch stream.

I do a lot of other stuff that might work well on a stream, though. Designing stickers in Illustrator; preparing workshop material and demos, making PowerPoint decks; I figure maybe some folks out there might like to see how it all comes together, so keep an eye on my Twitch or YouTube channels if you’re interested.

When I hit a thousand subscribers, I’ll release a new version of Rockstar. Go on. Sign up. Make me regret saying that.

“Is this React or Nuxt?”

Among the many fascinating questions that viewers shared in the chat during last week’s live stream, one the really stuck with me was somebody who dropped in while I was adding some client-side JavaScript to handle a few specific interaction scenarios, and asked “is this React or Nuxt”?

I often wonder if there’s a generation of developers out there now who have been so immersed in frameworks and abstractions for their entire careers that they literally don’t grok what JavaScript actually is. Then I wonder if there was a generation of machine code programmers who felt the same way about those young whippersnappers with their assembly languages, and then a generation of assembly hackers who didn’t trust C compilers, and C programmers who didn’t trust Java, and that maybe that’s just the way the world works. I suspect that Spolsky’s Law of Leaky Abstractions remains as relevant now as it was 20 years ago. You want to do React or Nuxt? Cool — but you probably need to know enough JavaScript to understand what’s happening when the abstractions break down.

On the other hand, I watched Ben Smith’s presentation at Joy of Coding in Rotterdam last month. Ben codes demos, in web assembly, by hand: less than 2 kilobytes of hand-coded instructions that’ll run a maze race game right in your web browser, without a framework or compiler in sight. Yep, turns out that hand-coded assembly is alive and well right here in 2023. Pointless? Maybe. But sometimes, the point of a good demo is to remind you that not everything needs to have a point.

We Are Developers World Congress in Berlin

July was supposed to be downtime. No conferences, no gigs, no travel, just a nice quiet month at home — catch up with friends, work on some new talks, workshops, write a couple of songs… you know the bit in the agile manifesto where it talks about “responding to change over following a plan?”

Yep, next week I’m off to Berlin to do my developer comedy rock music thing at We Are Developers World Congress. This event is huge — not only in terms of attendance, but it’s also the first time I’ve been on a conference bill alongside names like Tim Berners-Lee, John Romero, and Joel Spolsky. Y’know, the guys who invented (or co-invented) the web, the first person shooter, and Stack Overflow, respectively. Plus a whole bunch of other amazing speakers — and did I mention it’s in Berlin, one of the coolest cities in the world? Tickets are on sale now — €579, but they’re doing a €299 ticket for early-stage startups and a €199 ticket for students. I’ll be rocking out on the Airstream Stage on Thursday evening; if you’re there, come & say hi. I might even be giving out stickers.

The Horizon IT Scandal

In my talk “Failure is Always an Option”, I share the story of the Horizon IT scandal which led Post Office Limited, the UK company that administers high street post office branches, to falsely prosecute more than 700 employees between 2000 and 2014 after a faulty IT system reported accounting discrepancies and financial shortfalls. Horizon is back in the news this week, after the chair of the inquiry expressed concerns about whether the compensation programme set up in the wake of the scandal would be able to deliver on their commitment to fully compensate the affected parties by August next year.

Liability has always been a tough question when it comes to software engineering: compared to industries like healthcare and aviation, even the most cautious software projects still look like four drunks wrestling for control of a stolen car. We can’t build perfect software; in a world where we write abstractions on top of abstractions on top of abstractions, there are just too many variables — often both literally and figuratively. What I believe sets the Horizon scandal apart, though, is the managers’ and executives’ absolute refusal to acknowledge the possibility that the software might be to blame.

Software has bugs, and if somebody is prepared to testify under oath that it doesn’t, they’re either lying, or they’re stupid2 — or quite possibly both. It’d just be nice to see the executives behind these kind of fiascos face some consequences once in a while.

This week I’ve been…

Listening to: “SIX” by Extreme. I suspect I’m not the only Extreme fan who loves every moment of their second album3 but then finds most of their subsequent output rather less compelling. SIX is growing on me, though. A little whimsical in places, and a track or two that wouldn’t be out of place on a Muse album, but when SIX kicks into gear, it kicks hard. Nuno Bettencourt remains one of the finest guitar players the world has ever seen — check out this video of Queen’s Brian May reacting to Nuno’s solo on Get The Funk Out; it’s simply wonderful — and on tracks like BANSHEE and lead single RISE, the guitar work here is just sublime. No idea why they set all the song titles in CAPITALS, though.

Watching: Indiana Jones and the Dial of Destiny. Yes, they made another Indiana Jones film. Yes, Harrison Ford is 81 years old. Yes, the fourth one was a horrible mistake. Yes, that sound you can hear is taurus pecunias going “please, father, I’m so tired… let me sleep now?”

The thing is, I love cinema. A film has to be very, very bad indeed for me not to enjoy it on the big screen, and Indy V is not bad at all. The story is endearingly daft, there’s the odd spot of slightly wonky CGI, and plot holes you can fly a plane though — quite literally, in one case. But the cast are on cracking form; it has charm, and pace, and is packed with the kind of set-pieces that are so much fun to watch you don’t even think about how daft the whole thing is until afterwards.

Reading: Nettle and Bone by T. Kingfisher. I knew nothing about this book other than that it made the shortlist for this year’s Hugo Awards. I’m planning to read — or at least start reading — all six shortlisted novels between now and the awards ceremony in October; I’d already read and thoroughly enjoyed John Scalzi’s The Kaiju Preservation Society, but T. Kingfisher is a new name to me, and so far I like it a lot. If dark fantasy with witches and dustwives and bone dogs and even an actual fairy godmother sounds like your kind of thing, check it out.

Yeah, I know. You never subscribe to things. Go on. Break the chains of your conditioning. Dare to dream the impossible dream. SUBSCRIBE!

And that’s the week, folks. Next week’s issue will be a special THOTH4 from We Are Developers World Congress in Berlin, and quite probably a review of #barbenheimer.

Until then, stay safe, have fun, and be excellent to each other.

Dylan

1 At the moment, I’m using restream.io to stream to both Twitch and YouTube; Twitch generally seems to be a better platform for this kind of streaming, but I have way more followers on YouTube so I’m hedging my bets for now.

2 Ok, or maybe they work for NASA.

3 Which I shan’t name here, because the title contains a naughty word and the spam filters won’t like it.

4 The Hang Of Thursdays. THOTH — you know, the Egyptian god of maths, science and magic.

Travelling with Tiny Guitars

2023-07-24T12:00:00+00:00

Last week Jakub Gutkowski asked on Twitter:

@dylanbeattie hey, what travel guitar (when you fly etc) are you using? Thanks!

Sounds like a really good opportunity to geek out about guitars for a bit.

The current Steinberger Spirit guitar line-up. Gotta catch ‘em all… right?

My travel guitars — I currently own two — are Steinberger Spirit GT-PROs, and there’s a bit of history behind them. They were originally designed by a guy called Ned Steinberger in the 1980s. Steinberger was a true innovator; while most guitar companies were churning out the same wooden solid-body guitar in various different shapes and colours, Steinberger’s guitars were made of carbon fibre. To eliminate the headstock, he moved the tuning machines into a special bridge, and built his guitars to use specially built strings with a brass ball at both ends.

Personally, I think Steinberger’s greatest innovation was a thing called the “TransTrem”. See, the tremolo on most guitars will let you wobble the strings around a bit, get some gentle vibrato or a screeching “dive bomb” effect — but when you do, every string detunes by a different amount. The TransTrem used calibrated strings so when you grabbed the trem bar, the strings stayed in tune relative to each other — if you want to see one in action, check out “Get Up” by Van Halen, from the 5150 album; here’s a live clip from a 1986 show showing Eddie van Halen playing one of his custom red-striped Steinbergers, complete with liberal use of the TransTrem:

Steinberger sold the company to Gibson in 1987, who continued to manufacture the guitars until the mid-1990s. Then grunge happened, everybody got really miserable, the music industry unilaterally decided that the 1980s had been a horrible mistake, and Steinberger guitars went out of production for a while.

Life, the Universe, and… Guitars?

I had no idea any of this was going on; I think I might have seen a Steinberger on Top of the Pops once or twice, but I couldn’t have told you what it was called or anything more about it. Then, during the 1990s, the Guardian newspaper in the UK ran a short-lived column called “Me and My Gizmo”. They’d interview celebrities about a particular tool or piece of technology they loved — explorers talking about their Swiss Army knife, chefs talking about a piece of kitchen equipment. One week, the featured guest was Douglas Adams, talking about his favourite guitar: a white, left-handed Steinberger.

I’ve been a huge fan of Douglas Adams since I first read The Hitch-Hiker’s Guide to the Galaxy when I was a kid. I love his writing, I love his fascination with technology and gadgets… and it turned out he was also a huge guitar nerd. I clipped out the column, and it was pinned up above my desk in my bedroom for years. That original clipping went missing somewhere among my many undergraduate house moves, but with a bit of help from the lovely folks at ZZ9 Plural Z Alpha, I managed to track down a copy via the newspapers.com online archive. I’ve reproduced it here:

“ME AND MY GIZMO No 6: Douglas Adams’s guitar”, The Guardian, June 23, 1994

THE STUDY AT the top of Douglas Adams’s house owes less to Ford Prefect than it does to Spinal Tap. The author of the Hitchhiker’s Guide to the Galaxy and other ingenious comic and scientific observations works in a room festooned with guitars. Acoustic guitars, a brand new Santa Cruz guitar customised with exploding earths and dolphins in mother of pearl, ancient electric guitars, and a frightening aquamarine green guitar. Admittedly, it is a techie’s study, with a powerful Macintosh Quadra 950, a couple of lesser discarded Macs, a PowerBook, video entryphone and enough mixing and recording equipment to supply a modest studio, but it is the guitars that dominate. Adams owns about 30, but the one that really got him playing again is a white headless Steinberger. Adams began when he was 12, but the music dwindled as he travelled in his quest for the oddly named endangered fauna recorded in his book Last Chance to See.

Three years ago, he bought the Steinberger secondhand in Hollywood, for about $3,000, from the better known of the world’s two extant left-handed guitar shops (the other is in East London). It has full-length strings, but everything else is pared right down so it looks like a truncated toy. It plugs in to a Walkman-sized amp attachment, which feeds into tiny unfolding headphones, so it is perfect for Adams to take on his travels. He can be a guitar hero in hotel rooms worldwide, and even admits to practising mid-flight, “though you do get some strange looks from other passengers”. The Steinberger is made of graphite, so it is extremely strong. “If Pete Townshend had used one of these,” Adams says, “he would have done more damage to the stage than to the guitar.” Rocker lore is weighing heavy on Adams’s mind, as his buddy Dave Gilmour made him the birthday present of inviting him to join Pink Floyd for one number at Earl’s Court in October. Adams is already frantically practising the tricky full tone bends in Wish You Were Here.

That Floyd influence extends to Adams’s great ambition: to write an album. “Concept is the term I’m trying to avoid, but I’d like to write a whole piece, with a collection of songs that would make a story.” He messes about on his bank of synthesizers, electronically tweaking the sounds and storing them on disk. Performer software analyses each sound – its exact pitch, length and volume – and can transcribe any noise onto a musical score. “Singing is not the high point of my act,” he says, but he demands a whole-hearted and harmonious performance from friends at his annual Christmas carol party. “We don’t go out on the street shaking our tins, but we have a really good sing. We only invite friends who are proper singers.”

Still, Adams is stronger on the qwerty keyboard than the ivories. “I do a bit of Net-surfing, to newsgroups on music, computers, palaeontology, anthropology. And there’s a Douglas Adams fan group, so I have a look there.” His latest plaything is a virtual landscape building program, sort of cyber Capability Brown, for creating arty archipelagos, tormented seas, grassy undulations and other twiddly bits. Another handy piece of software is Now Up-to-Date, a calendar manager that schedules appointments into his wife and sister’s computers. The Adamses live in a well-wired household; when they were redecorating to make way for their imminent baby, Douglas stopped his wife from getting rid of the room’s Ethernet cable. “She’ll be needing that soon”, he predicts, “I’m just trying to work out what computer to get her.” The child will clearly be born with a mouse in her hand.

- Ruth Shurman

Douglas Adams died in 2001. Then in 2015, his original white Steinberger showed up for sale on eBay and somebody sent me a link going “hey, look, you can buy Douglas Adams’s guitar!”

There were only two problems: one, I wasn’t left-handed1, and two, I didn’t have three and a half grand knocking around. It got me thinking, though. This was around the time that I started travelling to tech conferences abroad, and writing tech parody songs, and putting them up on YouTube; I’m not saying I needed a travel guitar, but I really, really wanted one — and if it could be a white Steinberger like Douglas’s one, then so much the better. So I did a little digging.

The Steinberger Spirit revival

Turns out Gibson had started making Steinberger-style guitars again in the early 2000s, under the “Spirit” model name. Now, the Steinberger Spirit is a very, very different beast from the original 1980s guitars. They’re made of wood rather than carbon fibre, they don’t have the transposing tremolo, they don’t have active pickups, and they’re not made in the USA… and they cost a few hundred pounds, rather than several thousand.

My white Steinberger Spirit GT-PRO.

I bought my white Steinberger from Guitar Guitar in Epsom in 2016 for £299. It was love at first strum: I plugged it in, played it for about thirty seconds, and that was it. That guitar’s been to Australia twice, seven trips to Russia, six trips to Ukraine, the USA, Belarus, Israel, and dozens of shows all over Europe — if all those trips were in a straight line it’d be three quarters of the way to the moon by now. I’ve swapped out the regular strap buttons for Straplocks, and moved the neck strap button on to the top shoulder of the body, but apart from that it’s completely stock.

The first few trips I did with this guitar, I took it in the cabin with me, but flying with a guitar — even a tiny travel one — is a bit of a grey area. Technically, it’s not allowed, because it exceeds the cabin baggage allowance. I’ve never been stopped, but I’ve also never been explicitly told it’s OK… it was always a case of smile, don’t ask, and hope it works out, and I was never entirely sure what I would do if it didn’t work out.2

These days, it goes in the hold. I have an Eastpak roller duffel case which is big enough to stash my guitar in the bottom, solid enough to keep it safe in transit, and small enough to check as regular baggage, so most trips the guitar goes in the big case with clothes packed around it, and the aluminium flight case with my mixing desk, wireless gear, etc. shoved in the top.

Yeah, checked baggage doesn’t always arrive in the right place at the right time. I’ve been remarkably lucky with air travel — the only time my bag ever went missing was on the way home from St. Petersburg a few years ago, and I got a phone call less than an hour after I left the airport to say they’d found it. Still, it’s a risk, so a year or so later, I went shopping for a backup guitar in case the primary guitar went walkabout.

Turns out that the supply chain for these guitars can be a little volatile: none to be had in guitar stores, for love nor money. Eventually I tracked down a second-hand one, but trying to keep it in tune gave me all kinds of headaches. Turns out there was a huge crack in the tremolo block, which was slowly bending out of shape under the tension of the strings, and eventually snapped off altogether.

That one hung on the wall for a long while, until earlier this year, I decided to overhaul it: a new EVH-inspired paint job, new electrics, a replacement tremolo block, and a headstock adapter so I could use regular strings with it instead of special Steinberger double ball-end strings.

Overhauling a Steinberger Spirit GT-PRO guitar.

I’m really happy with the new paint job and the electrics… but the new trem block and the string adapter, not so much. It’s not easy to find authentic Steinberger replacement parts, so I ordered after-market replacements which are apparently made by OVERLORD OF MUSIC, and, well, they’re terrible. The string adapter lasted about four hours before tuning problems drove me to put the original headstock back on, and I ended up completely dismantling the new tremolo assembly and rebuilding the original around the replacement block from the new one, and I’m still not entirely convinced.

Replacement part woes aside, they are genuinely wonderful guitars. They’re fun, they look fantastic, they sound great and play remarkably well considering they retail for under £500. Given that the famously fickle supply chain appears to be firing on all cylinders right now, I suspect that before long I might end up buying another one.

After all, no matter how many guitars you already have, the ideal number of guitars to own is N+1.

1 I’m still not, in case you’re wondering.

2 On my fourth, maybe fifth, trip to Russia, the attendant at the airline check-in desk told me that “guitars were not allowed on flights to Russia”. I explained I’d taken the same guitar on the same airline several times before. She said that wasn’t allowed. I thanked her for the information and said I’d remember it for next time. She handed me my boarding pass, and that was that. Even at the best of times, Russian bureaucracy never made the slightest bit of sense.

The Hang of Thursdays #2

2023-07-21T12:00:00+00:00

The MVP Renewal Cycle #mvpbuzz

This week my networks have been buzzing with people sharing the news that they’d been renewed as a Microsoft MVP. MVPs — “Most Valuable Professionals” — are “technology experts who passionately share their knowledge with the community”. People who work with Microsoft technology and who run meetups, speak at conferences, write books, write blogs, maintain open source projects; activities which aren’t really your job, but which benefit the wider community.

Microsoft aren’t the only company that does this; I know a few Google Developer Experts and Java Champions, and I even met a Docker Captain once. I know there are some mixed feelings about these kinds of programmes in general, and the Microsoft MVP programme in particular — usually sentiments along the lines of “wow, you did all this free marketing for a trillion-dollar company and they sent you a certificate!”

But, whatever you think about Microsoft, corporations, and these kinds of programmes, just about every Microsoft MVP I’ve ever met has been a thoroughly excellent person, and that alone makes it a programme I’m proud to be part of.

So congratulations to all the Microsoft MVPs out there who got renewed, and particular congratulations to everybody who’s just been awarded for the first time. For me, this will be my seventh award, but I can still remember vividly what it felt like when I got my first MVP award: I was absolutely delighted. Recognition is a big deal.

London .NET Meetup with Angeliki Patsiavou and Nick Chapsas

Thanks to everybody who came along to our London .NET User Group meetup last night — and especially to our hosts Codat for providing the venue, drinks, pizzas and snacks.

Our first speaker was Angeliki Patsiavou, who presented a brand new talk about building and enabling cross-functional teams inspired by “The Avengers”, packed with insight and humour — not to mention one of the best slide decks I’ve seen in a while.

London .NET meetup at Codat, July 12 2023

Next up was Nick Chapsas, with a great talk which was ostensibly about logging in .NET, but actually took us on a deep dive into the gritty internals of how strings are managed by the .NET runtime and how something as seemingly innocuous as formatting your log messages can actually have a big impact on your application’s performance and memory consumption.

For all the folks asking “did you record it?” — not this time, sorry! (although there’s a version of Nick’s talk from NDC Oslo available on YouTube.)

There’s good news on that front, though. LDNUG is taking a break in August, but when we come back in September, we’re planning to video all the talks from our meetups and share them online, so those of you who weren’t able to join us in person can catch up afterwards.

Rejected GitHub Profile Achievements

Flet has created this list of rejected GitHub Profile Achievements, which made me laugh out loud. I’ve definitely earned a few Procrastinators, two Secret Santas that I’m aware of, and at least one Sith Lord from the days when my team was switching from Subversion to git — but I think my most interesting achievement would have been a Tee Hee. Many years ago, I had a web project that I was working on using both Windows and macOS, and I thought it’d be a good idea to share the project folder from macOS via SMB and then map a Windows drive to it.

Turns out that if you clone a Git repo on macOS and then push it on Windows, you mess up every single line ending in the entire project.

So… yeah, don’t do that.

Flet’s list is at https://github.com/Flet/rejected-github-profile-achievements if you fancy a laugh. 😁

You know what would be awesome? Getting enough subscribers on here that I could justify deleting all my other social media accounts. Yeah. That would rule.

Guitar Karaoke at the Copenhagen Developers Festival

At the end of August, I’ll be heading to Denmark along with a lot of other familiar faces from the conference circuit to speak (and play, and sing, and generally be Loud and Extroverted) at the Copenhagen Developers Festival, a new event from the folks behind NDC Conferences, which combines technical tracks during the day with live music, karaoke, science, gaming, and other fun stuff in the evenings.

I’m going to be sharing the story of how I created Guitaraoke — so between now and August, I need to figure out how to virtualise a bunch of the hardware I’ve been using to run it.

One big part of that is replacing paper sign-up sheets with a web app, which I’m working on this week and streaming live at twitch.tv/dylanbeattie. Like all the best software development, the stream so far is about 50% actually adding features, and 50% going “but why is that test failing? why!?!?”

I’ll be running the new app for real at the next Guitaraoke night, which is this Saturday 15th July, at Ignition Brewery in south-east London. After all, beta testing is best done live, in a brewery, with a guitar in one hand, a beer in the other, and room full of drunk people cheering you on.

Guitaraoke at Ignition Brewery in London, 20 May 2023

Modern Web Development with C# and .NET

I’m also going to be running a two-day workshop in Copenhagen on modern web development with ASP.NET Core. The web is wonderful, .NET rocks, and… well, I don’t like JavaScript frameworks.1 So I’m going to be showing folks how to build a website entirely in .NET, from frontend features like TagHelpers and SASS compilers, to minimal APIs and controllers, to some of the things .NET doesn’t really do well out of the box — email, timezones, y’know. Nice straightforward easy stuff.

Check out the workshop here: Modern Web Development with C# and .NET

And if you want to know more about how I prepare these kinds of workshops, you might find this post interesting.

“Prompts are unsafe, and that means language models are not fit for purpose”

I really liked this article by Baldur Bjarnason on the challenges of avoiding injection-style attacks when using large language models like ChatGPT. We’ve known about SQL injection attacks for literally decades; avoiding them is trivial, but they’re still one of the most common vulnerabilities seen in the wild. Securing language models against “jailbreaking” is orders of magnitude more complex than avoiding SQL injection attacks; Baldur’s basically saying we have no idea how to do this so we shouldn’t be integrating language models with external services or exposing them to external users. Interesting and unsettling in equal measure.

It’s been an entertainingly chaotic few weeks in the wonderful world of social media. My thoughts on the whole thing are here; I also enjoyed Alex Kirshner’s “Meta’s New Threads App is Terrible. It Might Just Bury Twitter” on Slate.com, but only time will tell whether this is the beginning of the end for social media, or just the end of the beginning.

This week I’ve been…

Reading: “Code That Fits in Your Head” by Mark Seemann. I’ve met Mark at a few conferences over the years, I find the way he talks about software complexity really refreshing, and the book is no exception: a step-by-step, case-by-case walkthrough of building a restaurant reservation system. My personal highlight so far is what Mark calls the Devil’s Advocate technique: given a perfectly reasonable-looking unit test, what’s the weirdest, most off-the-wall code you can write to make the test pass — and then how can you improve the test so that it catches those loopholes and edge cases?

Watching: South Park, season 25. I signed to Paramount+ to watch Star Trek: Strange New Worlds, which is excellent — and then found every season of South Park is also on there and I had a bit of catching up to do. I’ve been a fan of South Park since watching season 1 on RealPlayer 25 years ago, and it still makes me laugh until milk comes out of my nose.

(On a South Park-related note, the show’s creators, Trey Parker and Matt Stone, have recently spent literally millions of dollars refurbishing and reopening La Casa Bonita, the Colorado restaurant where they spent their childhood birthdays — complete with waterfalls, cliff divers, puppets, and a “person in a gorilla costume being chased by a sheriff”. I think that all sounds rather lovely.)

Listening to: Iron Maiden, live at the O2 here in London last Friday. A fantastic live show by a band who show absolutely no signs of slowing down; triple guitar harmonies, thundering backline, and Bruce Dickinson happy bouncing between joking with the crowd, belting out those high notes like an air-raid siren, and exchanging pyrotechnic laser fire with Eddie, Maiden’s monstrous zombie mascot who would occasionally prowl the stage in the form of a giant 12’ puppet.

Shout out to opening act Lord of the Lost as well; as somebody who remembers the Eurovision Song Contest being the least credible thing imaginable, I find it wonderfully weird that an industrial goth rock band (!) can represent Germany in Eurovision (!!), finish last (!!!) — and then open for Iron Maiden at the O2 in front of over ten thousand people less than two months later. But we do, and they did, and it was very good indeed.

And that’s the week, folks. Take it easy, stay safe, and be excellent to each other.

Dylan

1 Apparently “server-side React” is a real thing that people do now. Just… wow.

The Hang of Thursdays #1

2023-07-14T12:00:00+00:00

Hello! Welcome! So, this is a completely new thing and I really don’t know yet what kind of shape it’s going to take, but as yung pueblo said:1

main goal for july: no more waiting.

don’t get stuck in the planning stage.

you know what you need to do.

be bold and start making moves, even if the plan is not clear.

That hit a nerve. I’ve been saying for years that I need to write more. Folks who know me, or follow me online, know I do all kinds of stuff — tech, computers, music, travel. Lots of what I do is quite good fun, and even the bits that aren’t normally throw up the odd funny story along the way. I do a lot of stuff, and I like to write about what I’m doing, but recently a lot of the stuff I write ends up on platforms like Facebook.

…but hang on, don’t you already have a blog?

Yeah, I do! I got a whole website over at dylanbeattie.net. It’s mine, I own it, I built it myself using Jekyll, it’s currently hosted for free using GitHub Pages.

But… after five years, I’ve decided that blogging with Jekyll doesn’t work for me. Jekyll and GitHub Pages are brilliant for hosting static web content, but I want to be able to pull out my phone at the airport, connect my Bluetooth keyboard, write some words, paste a couple of photos, push a button, and boom, it’s live. That’s why so much of what I write ends up on Facebook instead of my own site, but I’ve been following a few folks who publish on Substack, and I like what I hear, I decided to give it a shot.2

So here it is: a weekly update on where I’m going, what I’m doing, and anything I thought was interesting. Like it? Want more? Here’s a button!

Subscribe now

What’s Happening?

July is delightfully quiet, and that’s just fine. May and June were back-to-back conferences and tech events - Bucharest, Essen, Hamburg, Berlin, Oslo, Stockhom, Athens, Odense, Rotterdam, Utrecht and Amsterdam, bouncing home in between for just long enough to do the laundry, host a London.NET meetup and run a guitar karaoke night. Being on the road is fun, but it’s nice to be home for a while.

Berlin, Hamburg, Bucharest, Stockholm, Oslo, Athens, Rotterdam… I need a vacation.

Email vs Capitalism, or Why We Can’t Have Nice Things

At NDC Oslo last month, I presented a completely new talk about email, spam, and the challenges of running a free, open, decentralised communication network in a world that’s increasingly controlled by billion-dollar multinationals.

I’m pretty happy with how it came out, but watching the recording back, I’m seriously thinking this one might warrant splitting into two separate talks - a fun, accessible talk with all the weird history, the RFC edge cases, the story of junk mail, and then a second one aimed at developers that gets into the gritty detail of things like DMARC records and how to mash up Mailjet and Razor to build a .NET mail templating engine. Or maybe keep the live presentation as the fun talk, and move all the gritty detail into some sort of video course folks can find online.

And yes, as several commenters have already pointed out, I screwed up one of the “let’s break email!” experiments you see in the video.

Oops.

I’ll fix it for next time.

London .NET Meetup at Codat on July 12th

After a long hiatus during the pandemic, the London .NET User Group is very much back up and running. We’re going to be meeting at Codat’s offices in Farringdon on Wednesday 12th July; Angeliki Patsiavou is going to be talking about what “The Avengers” taught her about working with developers, and then Nick Chapsas will be talking about logging in .NET and why we’re all doing it wrong. Nick’s basically an internet superstar with about a bazillion followers on YouTube — and as of right now, we’ve got 100 people RSVP’d for next week and 40 of them are first-timers who’ve never been to a London .NET meetup before, which is fantastic.

There’s a few places still available if you want to come along:

https://www.meetup.com/london-net-user-group/events/294304925/

And, note to myself: I am absolutely going to remember to print name badges this time, because the number of folks I recognise but can’t remember their names is getting embarrassing.

This week I’ve been…

Reading: “Season of Skulls” by Charles Stross. It’s the latest novel set in the world of the “Laundry Files”, which started life as a wonderfully macabre mashup of HP Lovecraft and the Jargon File, and now gleefully bounces from trope to trope trashing genre conventions as it goes. Daft, delightfully written, and enormous fun.

Listening: “Rosalie Cunningham” by Rosalie Cunningham. I saw Rosalie and her band at Prognosis Festival at the O2 back in April, and have had the album on constant repeat ever since. It’s — for me, anyway — a rare example of music that sounds genuinely progressive without just being “complicated heavy metal with lots of time signatures”, and I’m thoroughly enjoying it.

Watching: “Still: A Michael J. Fox Movie” popped up unexpectedly on Apple TV, and it’s excellent. It’s the story of Michael J. Fox, his acting career — Family Ties, Back to the Future, Spin City — and how his diagnosis with Parkinson’s disease at the age of 29 affected his life and his work. It’s brilliantly put together, combining interview footage with Fox today with staccato clips cut together from his movies and TV shows — watch out for the montage showing how he’d always find something to do with his left hand so viewers wouldn’t notice the tremors; it’s quite amazing.

And that’s a wrap. Drop a comment below, let me know what you think?

Until next time: rock on, have fun, and be excellent to each other.

Dylan

1 or maybe didn’t say, but somebody shared it on Facebook with his name on it and that’s basically the same thing, right?

2 I figure any publishing platform that’s hosting writing by Tom Cox, Cat Valente and my mum is probably doing something right.

The Social Meltdown

2023-07-12T12:00:00+00:00

I just joined Threads. If you’ve not been following the news, Threads is the new social media app from Meta, the company behind Facebook, WhatsApp and Instagram. It’s basically a carbon copy of all the worst bits of Twitter, right up to the capricious billionaire chief executive who’d probably kill their own users if they thought they could sell advertising space on the headstones.

Not dead, only Thread.Sleep()ing?

Threads will never replace Twitter, though, because what Twitter was, at its peak, was something remarkable. It was not a microblogging service, it was the microblogging service. If you wanted to post something publicly, you’d post it on Twitter, whether you were the President of the United States of America, or a parody account based on a sitcom about a burger van in Scotland.

Twitter had clout. It had influence. For better or worse, tweeting could actually change the world. The Arab Spring, Obama, Trump, Brexit… sure, it wasn’t the biggest, most popular, or most profitable social media platform. But ask yourself: have you ever seen a “serious” newspaper use LinkedIn or Facebook as a source when quoting somebody? Nope. Twitter drove the news.

I joined Twitter in 2008 (here’s my first tweet!), around the same time I started getting involved in tech community events. Twitter was perfect for the kind of loosely-coupled connections and friendships that arise from those events. Facebook was too personal, LinkedIn was too corporate, email too time-consuming. But tagging somebody on Twitter was a wonderfully unobtrusive way to say “hey, we’re talking about this thing you might find interesting.”

It wasn’t always like that. Before microblogging, there was blogging. Blogging was great. Google Reader was a great product, Blogger and WordPress were easy enough to get up and running, and writing a blog became a de facto rite of passage for anybody trying to establish a name for themselves in tech, journalism, art, music.

Social media killed that, and I believe that’s because it changed the dynamic of what people do when when have five minutes to kill at their desk between meetings — not to mention the paradigm shift from desktop to mobile that was taking place around the same time. I used to check my blog roll a few time a day. I’d post in the comments, reply to threads, write my own follow-up posts. Then social media came along, and blogs got relegated to something I’d check when I’d caught up on Facebook and Twitter… and before long, that became impossible. Too many people, too much content, and if you’ve read everything your friends posted, the algorithm can always find something else to show you.

I’m as guilty as anybody of falling for the convenience of social media. Over the years, I spent more and more time posting on Twitter and Facebook, and less and less writing on my own blog. The occasional tweet going viral was a thrill… the most vapid, inconsequential kind of thrill, a completely arbitrary and unverifiable number on a web page going up every time the my post racked up another page impression to sell to their advertisers — but hell, they had some incredibly smart people getting paid an obscene amount of money to make going viral on those platforms feel like an accomplishment, and it worked.

Then, in November last year, Elon Musk bought Twitter.

I don’t know anything about electric cars, so based on the media coverage of Tesla’s success, I assumed Musk must be some sort of genius.

I do actually know a thing or two about rockets, and when I saw SpaceX landing two reusable launch vehicles side-by-side like something out of a 50s science fiction movie, I continued to assume that Musk must be some sort of genius.

Then he bought a software company — and as it happens, I know quite a lot about software, and about what’s involved in running complex software platforms.

I no longer believe Elon Musk is a genius. Not even close. I think a ginger cat with one brain cell would have done a better job running Twitter than Elon Musk has, unless he’s actually trying to destroy the platform and bankrupt himself in the process.1

Twitter is dying. Sure, it’s still up, I’m still using it, and so are lots of other people. But it feels like nobody’s in it for the long haul any more. It’s like when you realise your favourite bar isn’t bothering to fix the broken washroom taps any more: we’re still visiting, but we’re waiting for the day we show up to find the doors locked and the repossession notice stuck in the window, and when the day comes, nobody will really be surprised.

The problem is: there’s no compelling replacement.

In the immediate aftermath of Musk’s acquisition, a lot of folks I know set up accounts on Mastodon. Quite a lot of us discovered we already had accounts on Mastodon, which we’d completely forgotten about… and quite a few of those, myself included, discovered that the Mastodon instance we’d signed up to wasn’t a Mastodon instance any more and was now hosting some, ah, specialist adult content.

At the time of writing, I have active accounts on Mastodon, BlueSky and Threads — not to mention Substack’s own Notes platform, which looks suspiciously like Twitter but with better typography and no users. Plus all the platforms I was already on: Facebook, Instagram, LinkedIn, Telegram, WhatsApp, 43 different Slack workspaces (don’t ask), Discord, Signal, Meetup…

In a way, it’s a blessing. The sheer overload of networks means that social media no longer feels remotely convenient — and as much as I wish a few of them would hurry up and actually shut down, I suspect that’s about as likely as Nadine Dorries actually resigning.

Fundamentally, though, the process of re-evaluating Twitter’s role in my online career has led me to realise that I use — used? — Twitter for four things.

First: I use it to tell stupid jokes. No big deal. Jokes move easily onto another platform.

Second: I use it to publicly engage with companies whose behaviour I think is worthy of comment. Sometimes because they do something I think is exceptionally cool; sometimes because they do something I think is exceptionally uncool. Companies who aren’t answering the phone or replying to email will often respond to a tweet — especially from somebody with thousands of followers. I suspect this is just gone (thanks Elon): I honestly can’t imagine an airline paying as much attention to Threads, Mastodon or BlueSky as they did to Twitter in its heyday, and as Twitter itself becomes increasingly irrelevant I’m sure customer service departments all over the world are breathing a huge sigh of relief.

Third: I use it to talk about tech. Fifteen years ago, when I didn’t know anybody to talk to when I needed help with .NET and Firebug and jQuery, posting on Twitter had an uncanny knack of connecting with the right people. For me, that’s not a factor any more; my network of helpful tech people has reached critical mass and is no longer beholden to a specific channel or platform.

Finally, I use Twitter to talk to people. Sometimes, this is personal: I talk to a lot of people via Twitter DMs, and in many cases, they’re people I have no other way to reach - no email, no phone number, no common Slacks or Discords. Over the next week or so, I’ll be pinging all of those people saying “hey, here’s how to reach me when this whole place finally burns down”.

Often, though, it’s more of a broadcast. It’s announcements about meetups I’m running, conferences, community events, workshops, Linebreakers concerts — and what I’d really like is to be able to use email for this.

Email is far from perfect, but what makes it unique is that it’s portable. When Twitter finally goes dark, I’ll lose the 15,000+ followers I’ve built up over the last fifteen years — they’ll just be gone. There’s no way for me to download a list and sync it to Threads or BlueSky. But if I have a list of email addresses, I can take that with me - and, if you’re running your own domain, so can you. I’ve moved dylanbeattie.net from my friends’ Qmail box, to my own mail server, to Fastmail; it’s not trivial, but it’s possible.

I’ve never run any kind of newsletter or mailing list before; to me, email has always felt more personal than just posting stuff online and hoping the right people will see it. But, y’know, with everything that’s happening to social media at the moment, it feels like it might be time to give it a shot.

You wanna be involved? Sign up, let’s see how this thing goes.

It’s just the same dumb stuff I was doing already, but in your inbox instead of on somebody else’s advertising platform.

And if not? I’m on Twitter, BlueSky, Threads, Instagram, Facebook, LinkedIn, GitHub, TikTok — and let’s face it, whatever daft thing launches next week, I’ll probably be dylanbeattie on that as well.

1 We shall leave aside, for now, the question of how moribund the automotive and aerospace industries must have been for a hack like Musk to disrupt them as successfully as he did.

What Should 'Cancel' Do?

2023-07-11T12:00:00+00:00

Good user experience design is about balance. Software lives in this weird liminal space where a perfectly innocent word has two contradictory and completely obvious meanings: anybody who’s ever written a Java program to to manage school enrolment knows that creating a class isn’t the same as creating a class, but most of the time, there’s a way to get the balance right.

Most of the time.

I got this absolute gem of a screen today, from some event management software I’m integrating with:

No, I’m not going to tell you where it’s from. They’re still figuring a lot of this stuff out, bless ‘em.

To be fair, if you end up on this screen by mistake and instinctively reach for the Cancel button, it won’t actually cancel the event unless there’s some text in the “CANCELATION REASON” box.

If there is? You’re one click away from accidentally refunding all your tickets and emailing all the ticket holders saying the event’s been called off.

I don’t know about you, but I’d have gone with a big angry red button labelled “Cancel This Event”, and perhaps a smaller, friendlier one labelled “Go Back”… y’know, just to make it obvious which one is which.

And, just in case you ever think the software industry is paying attention and learning from our own mistakes, here’s a post I wrote over a decade ago about SagePay doing exactly the same thing: “Should ‘Cancel’ cancel the cancellation, or just cancel the cancellation cancellation?”

What's in a (Version) Number?

2023-01-19T12:00:00+00:00

My Mastodon feed has been buzzing for the last few days with discussions about version numbers.

First, there was this post from Martin, asking what JS folks think about < 1.0 versions:

And then this one:

Now, this kind of rough’n’ready survey can be great to kick off some entertaining conversations, but this is software. The correct answer in both cases is “it depends”. Because it always depends.

Turns out the specific scenario they’re dealing with here is a project that’s basically ready to go. It’s running on production systems, it’s solid, it’s feature complete, but for political reasons the team behind it isn’t prepared to stamp it as version 1.0.0 - and probably won’t be for a year…

So the actual question here is something closer to:

We have a solid, finished product that can’t be shipped as 1.0.0 because of politics. What version number would mean you were comfortable using our package on your projects?

▢ 0.9.x

▢ 1.0.0-rcx

▢ I can’t/won’t use either of those versions

▢ Either would be OK for me

Now that’s a much more interesting question. I’ve run plenty of stuff in production that uses 0.9.x packages, but I have only very occasionally deployed a prerelease package as part of a production release, and only under very specific circumstances.

Let’s say I have a project which uses Foobar 2.1.0. I know Foobar, I’ve used it on a bunch of projects, it’s solid, I trust it. I find a bug. The bug will be fixed in the forthcoming 2.1.1 release… but I need it now. In that situation, I’ll install 2.1.1-rc1, but as a temporary fix. This is reinforced by the way the NuGet package management ecosystem usually requires an explicit opt-in to find and install prerelease packages: It’s not something you’re likely to do by mistake:

To me, checking that “Include prerelease” box is about getting the latest version a few weeks before everybody else, and it’s completely different thought process to what happens when I install a 0.9.x package.

To me, 0.9.x means “we’re still working on it.” There are TODOs in the source code, there are open issues on GitHub we haven’t triaged yet, and there are specific features which will be part of a 1.0 release which are not available yet.

That said - and as counterintuitive as this might seem - I’m probably more comfortable depending on a 0.9.x package than I am depending on a pre-release version. I’m certainly far more likely to install 0.9.x, take a bit of time to satisfy myself that’s doing the right thing, and then run with it until 1.0 eventually ships. I think this is probably because I associate pre-release packages with “coming soon” – but I don’t think I’ve ever actually encountered the specific scenario here, where the entire product is a 1.0.0-rc and is likely to be like that for a while.

There’s also a strong implication that the difference between 1.0.0-rc and 1.0.0 is technical: no new features, no new documentation, just give it a week or two for folks to kick the tyres and flush out any bugs we missed and then we’ll ship 1.0. Whereas the difference between 0.9.x and 1.0.x could be down to any number of reasons - code, branding, politics, documentation.

If you’d asked me before I started writing this post, I’d have said that the right approach in this specific situation would be ship 0.9.x releases. And remember - semver is numeric, so if you add significant new features, you can ship 0.10.x, 0.11.x, 0.12.x and keep going until the stars align and you’re ready to bump the major version to 1. The process of writing it has made me think hard about why that’s the case, though… and after careful reflection, I realise that it doesn’t actually make a huge amount of sense. But I suspect most folks out there won’t have thought about it in this much detail, which is why I think that the vast majority of developers working in .NET would be more likely to install and use a 0.9.x release than a 1.0.0-rc release.

And at the end of the day, though, it’s a version number. Your team might have taken weeks of tests, checks, reviews and sign-off procedures to finally stamp a release as 1.0.0, while the package that’s next to yours in the NuGet interface was one dev wired on Red Bull and synthwave YOLO’ing their code to NuGet at 4am and marking it 1.0.0 because they hadn’t found any bugs for, like, hours… and to all us folks out here, they look exactly the same. It’s what happens after we install them that actually matters.

Modern Frontends Live

2022-11-22T12:00:00+00:00

A lot of people have been sharing their experiences of the Modern Frontends Live conference that ran in London last week – an event that was advertised as having “25+ workshops” and “3000 developers”.

Well, here’s mine. I thought long and hard about posting this, for reasons which will become apparent, but here it is. I’m not going to comment on any of the things I’ve heard or read. This is my own personal experience of being part of this event.

June 2022: Gen Ashley contacted me via Twitter to ask if I would give a talk and/or run a workshop at a new conference she is working on, Modern Frontends Live, taking place at ExCeL in London in November. I’ve known Gen for over a decade; I wouldn’t say I knew her particularly well, but I’ve spoken at a few of her meetups over the years and seen her around at various tech events. I assume that the organisers have hired Gen to handle speaker liaison because of her connections in the London tech community. I submit a workshop and a talk – actually a placeholder “to be confirmed” talk, because Gen explains that they want to get speakers listed on the website as soon as possible. I include a note with my workshop submission asking to discuss revenue share arrangements before anything is published.

July, August… nothing happens. Total silence.

September 7th: I notice that my talk and workshop are both listed on the event website. I contact them to ask what’s going on; I have a call with Gen. She suggests a ticket price – £899 early bird, £1099 full price. That’s broadly in line with what I would expect for a two-day in-person workshop at an event on the scale they’re talking about.

Then Gen asks me if I’ll “donate” my share of the workshop revenue to support another event she’s running, TECH(K)NOWDAY, which is a “non-profit”.

I decline, politely, because I do this for a living. A substantial part of my income comes from running workshops at conferences, and while I don’t believe that Modern Frontends Live is going to reach the 3000 attendees their website is advertising, I think they’ll probably get 1500, maybe 2000. That should translate into 15-20 tickets for my workshop; after deducting overheads – room hire, catering, marketing costs – that should leave a respectable chunk of revenue for both of us.

October 31st: Gen asks me to “touch base” regarding my workshop. It turns out they’d sold three tickets. For most events, sales that poor would be grounds for cancellation. But Modern Frontends Live was happening in London, where I live; I’d already blocked out the time, and I didn’t want to disappoint the people who had bought a ticket.

November 10th: With a week to go until the conference, the agenda still hadn’t been published – I knew I would be speaking some time on Thursday or Friday, but had no idea exactly when. I got an email confirming that my workshop was going ahead – four attendees, two days of in-person training on Tuesday and Wednesday, 9am–5pm, in the “London Suite” at ExCeL – but no sign of the conference agenda. On Thursday morning, all the speakers received an email saying the agenda would be published later that day. Thursday came… Thursday went… no agenda.

Friday 11th: I received another email:

Subject: SCHEDULE UPDATE

Hi Dylan.

We’re having some technical issues with the platform we’re using, so there’s a bit of delay.

We also have a team member had a death in the family, and that has affected us a bit.

So bear with us!

The schedule was finally announced on Sunday 13th November at 17:09; my talk was scheduled for Thursday afternoon.

The week of the event: I arrived on Tuesday morning for the first day of my workshop. I couldn’t find the London Suite. I asked the venue staff; they’d never heard of it. No signage. No mention anywhere of Modern Frontends Live. I eventually find it – it wasn’t quite on display in a locked filing cabinet stuck in a disused lavatory with a sign on the door saying “Beware of the Leopard”, but it wasn’t far off.

Of the 25+ workshops advertised on the website, mine is one of only two that are actually running.

There’s no registration desk or attendee badges; Gen is there in person to meet people as they arrive. There’s no catering. No lounge, no break area, no coffee machine. We have the training room, and that’s it. After half an hour, Gen arrives with a tray of coffees that she’s quite clearly just bought from a place upstairs. Shortly after that she comes back with bottles of water.

It’s embarrassing. Considering how much the attendees have paid to attend the workshop, I’m absolutely mortified. The room we’re in has no windows or natural light. Two more attendees join unexpectedly at 10am – it turns out they couldn’t find the room either. But hey, no problem. We have plenty of space. We break for lunch – which turns out to be Gen paying for sandwiches at one of the concession stands because apparently London ExCeL is “too busy” to provide catering.

My workshop group is actually great: smart people with lots of really insightful questions. It’s a lot of fun working with them.

I discover later that most of the attendees in my workshop are only there because the workshops they’d booked and paid for had been cancelled. Chris Perry, one of the people who attended my workshop, has posted a detailed write-up of his own experience of the event.

After we call it a day, Gen takes me to one side. She wants to know why so many people are posting negative things about the conference on social media.

I explain, as politely as I can, that every single piece of communication and interaction I’ve had with the event has been shambolic – the emails, the schedule, the pricing, the sales, the lack of catering – and that if the other speakers’ experience is in any way comparable to mine, it’s no wonder they are angry. Gen is dismissive; she says that Twitter is not her problem and she needs to concentrate on running the event. I tell her she’s probably right but that if she ever wants any tech speaker to participate in one of her events again, she will need to follow that up with a full public explanation and an apology.

I think she thinks I’m joking. I’m not joking. I’m tired, I’m fed up, and I’m starting to suspect I will never get paid.

Wednesday goes much the same – except ExCeL is a lot busier: there are more events happening, with a lot more attendees; once Gen has paid for everybody’s sandwiches (again), there are no tables free anywhere to sit and eat lunch together. We go back to our windowless training room. If I’d been working with any other event, I would have taken the attendees out to lunch and charged it to the organisers… but, like I said, I’m already starting to doubt whether I’ll ever see any of the revenue share from this event.

On Thursday morning I head along to the actual conference. It’s underwhelming, to say the least. The sponsor “booths” are tables in a hallway. There are a few hundred attendees – nothing close to the 3000 developers advertised on the Modern Frontends Live website. Most of the rooms are laid out cabaret style – a trick that venues use when turnout is far below expectations: put all the chairs around big dining tables, so the room looks full even if there are only a few dozen people in it.

My talk goes pretty well. Maybe 50 people show up to learn about ray-tracing and 3D computer graphics. The projector works, the sound works. I don’t see a camera anywhere in the room. I cannot comment on the so-called “live stream” or the controversy about virtual tickets. I didn’t buy a virtual ticket, I didn’t try to access any streams of the event. All I know is that there was no camera in the Ada Lovelace room where I was presenting my talk.

I didn’t go back on Friday. After two days of training and a day of conference sessions, trying hard to create a positive experience for the paying attendees despite the logistical challenges, I was completely wiped out.

I’ve seen comments online about how polarised the discussion around the event has been, that everybody either loved it or hated it. Well, I think a lot of attendees actually had a great time – and a lot of that is down to the speakers, some of whom I now understand had paid their own travel costs in order to participate. But I’ve also chatted with the sponsors, and I have some idea how much they paid to be there.

By my reckoning, the total sponsorship revenue for this event was well in excess of £100,000. That’s over a hundred grand that is no longer available to support any other events. For some partners, that’s their entire event sponsorship budget for this year – wiped out for two folding tables in a hallway and a few hundred attendees. And wow, that pisses me off – especially when I consider that my participation, and that of speakers like me, was probably a factor in those partners’ decisions to sponsor the event.

And this brings us to the difficult part.

Modern Frontends Live owes me money. I don’t know how much, because I haven’t had any confirmation yet of the final ticket prices or the room hire cost. I’m guessing they owe me somewhere between £1000 and £2000 - and when somebody owes you that kind of money, it’s generally not good business sense to publish openly critical articles about them^{[citation needed]}.

Well, I’m not going to keep quiet and play nice about this. People who I know and respect have stuck their heads above the parapet to publicise what happened with this event. I care more about supporting them than I do about the money, and the more I find out about how this event was organised, the less comfortable I am with realising any kind of profit from it.

But I’m also not prepared to walk away and let Modern Frontends Live keep the proceeds, or claim that my speaking up about this justifies them not paying me my share.

So here’s the deal. I have asked Gen Ashley to work out the numbers and tell me how much my share of the revenue comes to. I expect to invoice Modern Frontends Live for the the full amount, I expect them to pay it – and in return, in the spirit of Gen’s request all those months ago, I will use that money to support the developer community. I’m one of the organisers of the London .NET User Group. Since 2002, our group has run free community meetups for developers working on .NET, web, and associated technologies.

If Modern Frontends Live pays me what I’m owed, I will use 100% of those funds to support London .NET User Group events during 2023, with a specific emphasis on improving diversity and inclusivity at our events. That would allow us to offer travel and accommodation costs to speakers from outside London; to invite more speakers from under-represented groups to participate in our events, and help us build connections with the wider developer community.

And if they don’t pay me? I think that’ll send a pretty clear message to any speakers, trainers, or sponsors who might be considering working with Gen Ashley or any of her conference brands ever again.

Your move, Gen.

Here’s what other people have been saying about Modern Frontends Live:

Niall Maher at Codú has posted a YouTube video “10 Tips on Running a Terrible Conference - Lessons from Modern Frontends”
Niamh McCooey: My Experience at Modern Frontends Live
Jo Franchetti: My experience of Modern Frontends Conference
Cassie Evans: Modern Frontends
JD Hillen: My Experience at Modern Frontends Live
Mike Hartington: Modern Frontends Live
Chris Perry: Modern Frontends: An Attendee’s Perspective
Hidde de Vries: My experience at Modern Frontends Live
Todd Libby: Modern Frontends

The Road To Guitaraoke, Part 1: Vamp, Chordino, ImageSharp, and ffmpeg

2022-09-19T12:00:00+00:00

One of my side projects at the moment is running a monthly karaoke night at my local brewery tap, the very awesome Ignition Brewery in Sydenham – but it’s karaoke with a twist.

As well as singing, you can get up and play guitar or bass; instruments and backline are provided, and I’ve created 5-channel mixes of all the backing tracks so we can fade out specific instruments if somebody wants to play them live.

I ran the first Guitaraoke night in August, and it went great (despite a heat wave and a train strike!) – but by far the most common bit of feedback I got was “it would be great it we could see the guitar chords on the video”. Well… yeah, it would. After all, the whole point of karaoke is that the singer doesn’t have to know the words… if we’re going to do Guitaraoke properly, the players shouldn’t need to know the songs.

So, that’s the requirement: guitar chords on screen on the night. For the last couple of weeks, I’ve been investigating possible ways of doing this – from editing the videos by hand in Premiere (which is way too time-consuming), to building some sort of custom video player that’ll read chord charts from a text file and overlay them onto the video (turns out building a video player is hard), to complicated systems involving OBS.

Today I hit a bit of a milestone, though. Here’s how I got there.

Vamp and Chordino

Vamp is “an audio processing plugin system for plugins that extract descriptive information from audio data”. I’d never heard of Vamp until a few days ago, and I have to say, it’s not the most user-friendly platform I’ve ever seen – the “documentation” for quite a few of the plugins is the scientific paper published by the authors describing what the plugin does. (No, really…)

Caveat: I spent a few days trying to get this stuff running on my M1 Macbook Pro, with limited success. Arm64 binary hosts can’t load x64 binary plugins, and a lot of this stuff involves getting program A to load library B which requires library C. Windows 10 has been straightforward, though - and I suspect Intel Macs and Linux would be just as easy.

To try it out and see how well it worked, I installed the Vamp Plugin Pack, and then used Audacity as a host application to play around with the plugins. Once you’ve installed the plugin pack, click Analyze, Add / Remove Plug-ins… and enable the “Chordino: Chord Estimate” plugin:

Then open an audio file, highlight the audio waveform, go to Analyze > Chordino: Chord Estimate… and you’ll get this:

This was my first “wow, this might actually work” moment… the plugin has done a pretty good job identifying the chords to “Man! I Feel Like A Woman” by Shania Twain, but it’s also extracted the exact timing of the chord changes. This is going to be useful.

The next step was to get this data out into a format I could actually work with. Turns out that the Vamp developer SDK includes a simple command-line host, so I grabbed a copy of that.

To use the command line host, I had to convert my input file to WAV audio (Audacity will quite happily analyze MP3 audio and MP4 video files if you’ve got the ffmpeg library installed). It took a bit of trial and error to get the exact syntax right; the command I needed is:

D:\projects> VampSimpleHost.exe nnls-chroma:chordino man-i-feel-like-a-woman.wav

VampSimpleHost.exe: Running...
Reading file: "man-i-feel-like-a-woman.wav", writing to standard output
Running plugin: "chordino"...
Using block size = 16384, step size = 2048
Plugin accepts 1 -> 1 channel(s)
Sound file has 2 (will mix/augment if necessary)
Output is: "simplechord"
 0.185759637: N
 0.464399093: Bmaj7
 3.854512471: Bb
 5.061950113: Gm
 5.619229025: Bb
 20.944399093: Eb7
 22.244716553: Bb
 28.653424036: Eb7
 29.907301587: Bb
 31.950657596: Gm
 32.554376417: Bb
 34.411972789: Ab
 34.690612245: Bb
 38.220045351: Ab
 38.545124716: Bb
 ...

Easy. That’s the timestamp (in seconds, as a decimal fraction), and the chord (“N” denotes “no chord”).

OK. Later, we can wire this into some kind of automated processing pipeline. For today, what I did was to redirect that into chords.txt and move on to the next part: turning this into video.

Turning Chord Data Into Video

There’s a bunch of ways to turn data into video, but there’s two things to remember. One – it always comes down to rendering individual frames and then sticking them together. And two – if ffmpeg can’t do it, you don’t need it. If you’ve not worked with ffmpeg before, it’s a command-line application that can read, write, convert and stream just about every audio and video format ever invented.

So, here’s my approach:

Create a program that’ll read that chord data and turn it into individual video frames
Feed those frames to ffmpeg to produce a transparent video file
Composite that video file onto the top of the original karaoke track (kinda like showing a subtitle track on a movie).

Rendering video in .NET using ImageSharp and FFMpegCore

I used .NET here because I know it, and I like it. To actually draw the chord names onto each frame, I’m using ImageSharp; each frame is then wrapped in an ImageVideoFrameWrapper, a class I wrote to pass data from ImageSharp to ffmpeg. I’m then using a .NET library called FFMpegCore to read those frames directly from memory and render them into the output video stream. Note that I’m using the .webm format here, and specifying libvpx-vp9 as the video codec – this is because I want my output video to support alpha transparency.

Here’s the program code:

using ChordMaker;
using FFMpegCore;
using FFMpegCore.Arguments;
using FFMpegCore.Pipes;
using SixLabors.Fonts;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Drawing;
using SixLabors.ImageSharp.Drawing.Processing;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;

const int FPS = 60; // frames per second
const float SPF = 1f / FPS; // seconds per frame

var chords = File.ReadAllLines("chords.txt").Select(line => new Chord(line)).ToList();
for (var i = 1; i < chords.Count; i++) {
    chords[i-1].Duration = chords[i].Time - chords[i-1].Time;
}

var frameCount = (int)((chords.Max(pair => pair.Time) + 5) * FPS);
const int width = 1920;
const int height = 1080;
const int speed = width/6;
var frames = CreateFramesSD(frameCount, width, height, chords);
var videoFramesSource = new RawVideoPipeSource(frames) { FrameRate = FPS };
FFMpegArguments
    .FromPipeInput(videoFramesSource)
    .OutputToFile("chords.webm", overwrite: true, options => options
    .WithVideoCodec("libvpx-vp9"))
    .ProcessSynchronously();
Console.WriteLine("Done");

static IEnumerable<IVideoFrame> CreateFramesSD(int count, int width, int height, List<Chord> chords) {
    var family = new FontCollection().Add(System.IO.Path.Combine("fonts", "ttf", "FreeSansBold.ttf"));
    
    DrawingOptions options = new() {
        GraphicsOptions = new() {
            ColorBlendingMode = PixelColorBlendingMode.Normal
        }
    };

    var playheadPosition = width / 8f;
    var chordBarHeight = height / 8f;
    var chordBarTop = chordBarHeight * 6.5f;
    var font = family.CreateFont(chordBarHeight * 0.6f);
    var chordTextTop = chordBarTop + chordBarHeight * 0.2f;
    var transBlack = Brushes.Solid(Color.FromRgba(0, 0, 0, 200));
    var transRed = Brushes.Solid(Color.FromRgba(255, 0, 0, 127));
    var white = Brushes.Solid(Color.White);
    for (var frame = 0; frame < count; frame++) {
        Console.WriteLine();
        Console.Write($"Frame {frame}/{count}: ");
        var time = frame * SPF;
        using Image<Rgba32> image = new(width, height, Color.Transparent);
        image.Mutate(x => x
            .Fill(options, transBlack, new RectangularPolygon(0, chordBarTop, width, chordBarHeight))
        );
        var lastChord = String.Empty;
        foreach (var chord in chords) {
            if (chord.Name == "N") continue; // No chord
            if (chord.Name == lastChord) continue; // No change
            if (chord.Name.Length > 4) continue; // ignore chords like Bm7b5 which Chordino sometimes returns
            if (chord.Duration < 0.5f) continue; // skip chords which are very, very short
            lastChord = chord.Name;
            var offset = playheadPosition + speed * (chord.Time - time);
            if (offset < -playheadPosition || offset > width) continue;
            var point = new PointF(offset, chordTextTop);
            Console.Write($"{chord.Name} ");
            image.Mutate(x => x.DrawText(chord.PrettyName, font, Color.White, point));
        }
        image.Mutate(x => x
            .Fill(options, transRed, new RectangularPolygon(0, chordBarTop, playheadPosition, chordBarHeight))
            .Fill(options, white, new RectangularPolygon(playheadPosition - 2f, chordBarTop, 4, chordBarHeight))
        );
        using ImageVideoFrameWrapper<Rgba32> wrapper = new(image);
        yield return wrapper;
    }
}

This is the Chord class:

class Chord {
    public float Time { get; set; }
    public string Name { get; set; }
    public float Duration { get; set; } = 0f;
    public string PrettyName => Name.Replace("b", "♭").Replace("#", "♯");

    public Chord(string line) {
        var tokens = line.Split(":");
        Time = float.Parse(tokens[0].Trim());
        Name = tokens[1].Trim();
    }
}

and the ImageVideoFrameWrapper<T> class that I’m using to get data from ImageSharp into the FFMPEG input buffer:

using System.Runtime.CompilerServices;
using FFMpegCore.Pipes;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;

namespace ChordMaker; 

public class ImageVideoFrameWrapper<T> : IVideoFrame, IDisposable where T : unmanaged, IPixel<T> {
    public int Width => Source.Width;
    public int Height => Source.Height;
    public string Format => "rgba";
    public Image<T> Source { get; private set; }
    public ImageVideoFrameWrapper(Image<T> source) {
        Source = source ?? throw new ArgumentNullException(nameof(source));
    }

    public void Serialize(Stream stream) {
        byte[] pixelBytes = new byte[Source.Width * Source.Height * Unsafe.SizeOf<Rgba32>()];
        Source.CopyPixelDataTo(pixelBytes);
        stream.Write(pixelBytes, 0, pixelBytes.Length);
    }

    public async Task SerializeAsync(Stream stream, CancellationToken token) {
        var pixelBytes = new byte[Source.Width * Source.Height * Unsafe.SizeOf<Rgba32>()];
        Source.CopyPixelDataTo(pixelBytes);
        await stream.WriteAsync(pixelBytes, 0, pixelBytes.Length);
    }

    public void Dispose() {
        Source.Dispose();
    }
}

That’ll create a transparent video file with moving chords on it in chords.webm. The final step is to composite this video onto the original backing video, which is another job for ffmpeg:

ffmpeg
  # input file #0:
  -i original.mp4
  # video codec for input file #1: libvpx-vp9
  -c:v libvpx-vp9 
  # input file #1
  -i chords.webm
  # video filter: scale video #1 to 1280x720, store that in [z], then overlay [z] onto video #0
  -filter_complex "[1:v]scale=1280:720[z];[0:v][z]overlay" 
  # video codec for output: libx264
  -c:v libx264 
  # video bitrate for output@: 2500kbps
  -b:v 2500k 
  # output filename
  composite.mp4

That’ll produce composite.mp4, which is the original backing video with the transparent chord overlay composited onto the top of it. It’s also a 4-minute video I don’t have permission to publish online, with a 5.1 surround sound mix that’s going to sound really weird unless you’re running it through the right audio gear, so just for all you folks following along at home, I ran it through ffmpeg one more time:

ffmpeg 
  # input file: 
  -i .\composite.mp4 
  # audio channels: 2
  -ac 2 
  # audio filter: render 5.1 down to 2.0 audio.
  #  FL (front left) = 0.7 * FC (front center), + 0.70 * FL (front left) + 1.0 * BL (back left)
  #  FR (front left) = 0.7 * FC (front center), + 0.70 * FR (front right) + 1.0 * BR (back right)
  -af "pan=stereo|FL=0.7*FC+0.70*FL+1.0*BL|FR=0.7*FC+0.70*FR+1.0*BR" 
  # skip start time to 45 seconds 
  -ss 00:00:45 
  # trim video length to 30 seconds
  -t 00:00:30 
  # use the codec for video: libx264
  -c:v libx264 
  # set the video bitrate to 2500kbps
  -b:v 2500k 
  # use the codec for audio: aac
  -c:a aac 
  # output filename
  guitaraoke-demo.mp4

That’ll produce a 30-second clip, in regular stereo, which I’ve uploaded to YouTube so you can see the results:

That’s a pretty convincing proof of concept… but there’s a world of difference between doing it once, with one clip, and being able to churn out the 50+ tracks you need to actually run a karaoke night. So next steps are to add beat detection, so I can quantise the timing of the chord changes to land on the beat exactly. Not essential, but nice to have.

I also need to automate the various steps - video > WAV > chords > overlay > composite - so I can churn this thing across a folder full of tracks and spit out dozens of videos at a time. I suspect some tracks will need some manual editing of the chord data before rendering the video, so I also need to figure out a way to run a high-speed version, probably 12fps at 640x360, to check the results, and then the high-quality 1280x720 60fps version to produce the final video.

And if you want to see it in action, come along to the Ignition Brewery and Taproom on the third Saturday of the month and check it out. Bonus points if you actually get up and play something. 🎸🤘🍻

Fun with Docker and /etc/hosts

2022-09-09T12:00:00+00:00

I was running my Distributed Systems with .NET workshop with a team in Bucharest this week – two days of learning all about microservices, HTTP APIs, message queues, gRPC, SignalR, and how they all work with .NET, and then on the third day we looked at the various ways teams can use Docker on these kinds of projects: hosting external services like RabbitMQ for local development; creating stable build environments, and, finally, hosting .NET apps and services directly in a Docker container.

All the exercises and demos went great right up until the last part of the third day… we went over how to package a .NET hosted service application as a Docker image, and then when we started up a container from that image… nothing. The service in question was a tiny component that would subscribe to messages coming in via a RabbitMQ queue, translate them into JSON, and forward them to a SignalR hub running in our web app. But when we fired up the component inside Docker, it couldn’t connect to SignalR. We tried a half-dozen different things - DNS, creating a bridged network… but none of it worked, and I was completely stumped as to why. I’d never seen anything like this before. I know that to get network traffic into a Docker image you’ve got to open ports and things, but getting traffic out? I don’t remember that ever even being a problem… it just works, right?

So we wrapped it up, without ever getting the final demo to work. Which I hate doing - it always feels like missing the end of the movie or something… so after I got home, I dug into what had gone wrong.

Turns out it was incredibly simple – but for complicated reasons.

There was absolutely nothing wrong. The code was fine, the Dockerfile was correct. And if I had run it on any machine, anywhere on the planet, other than the machine I use to run my workshop demos, it would have worked perfectly; I’d inadvertently created the diametric opposite of “works on my machine”.

For running workshops, I use a Windows 11 VM hosted in Azure as my demo machine, and I have a bunch of ports open from this machine to the internet. I point a real DNS record – workshop.ursatile.com – at it, and install a real HTTPS certificate, so that I can run apps in Visual Studio, and the attendees can connect to https://workshop.ursatile.com on various ports and get a real, live HTTP connection, with a proper certificate, with my demo/example code running on the other end of it.

Except… virtual machines running in Azure can’t see their own public IP addresses. If I want to connect to my own code, I have to connect to localhost - but because I’ve replaced the certificate, that doesn’t work.

So what I do is to edit the hosts file and add a record that says

127.0.0.1	workshop.ursatile.com

So when I connect to workshop.ursatile.com locally, it’s connecting to 127.0.0.1, but the rest of the world connects to the public IP in the DNS record.

But when I’m using that same VM to host Docker containers? Those containers can’t see the hosts file. Which means when I try to run one of the example apps in Docker, it does a public DNS query, gets the Azure external IP address for workshop.ursatile.com, tries to connect… and can’t, because as far as the Azure network is concerned, that traffic’s coming from inside the VM and so can’t see that external address.

So there you go. It’s always DNS, and when it’s not DNS, it’s probably the hosts file. Which is kind of also DNS. And for future reference, there is a very simple, standalone .NET application and Dockerfile here:

https://github.com/dylanbeattie/simple-dotnet-docker-service

which demonstrates how to build a .NET Hosted Service, run it inside Docker, and connect to external network resources.

Reinstalling Windows (again)

2022-07-08T12:00:00+00:00

In November last year I upgraded my main workstation to Windows 11. I didn’t like it much. Lots of unnecessary window dressing - rounded corners, changes to the Start menu, context menu, stuff that wasn’t a problem but didn’t really improve anything. I found Windows 11 slower than Windows 10. Booting my PC in the morning took nearly 3 minutes, most of which it was just sat there with a blank screen, doing something – I have no idea what? – before it finally brought up the login screen.

But the thing that clinched it was that something in Windows 11 couldn’t handle certain kinds of image data on the clipboard; copying images to the clipboard would cause parts of the UI to flicker, crash, redraw, and basically make Windows unusable. Most noticeably anything in Adobe Photoshop - there’s a support thread here all about it, but it’s not a Photoshop problem; various other applications caused the same behaviour, up to and including right-clicking the icon in the Google Chrome Help > About page.

So bye-bye Windows 11.

When it comes to reinstalling Windows, this is not my first rodeo. I keep everything important on a separate drive, with all my work either pushed to GitHub or backed up in Dropbox, so my D: drive just comes along for the ride completely intact, meaning it’s not that big a deal to reformat C: and put a fresh Windows install on it. So I took a backup copy of my C: drive – just in case – and then grabbed the latest Windows 10 Pro ISO, used Rufus to burn it to an 8Gb USB stick, and off we go.

Windows 10 install took about 15 minutes. Here’s what goes on afterwards:

Configure Windows

Explorer, View, Folder Options
- Check “Display the full path in the title bar”
- Uncheck “Hide extensions for known file types”
Set desktop wallpaper to something suitably atmospheric from kvacm (and if you like Michal’s stuff, support him on Patreon!)

The Essentials

I use Ninite to install Chrome, Firefox, Evernote, Steam, Zoom, Discord, Dropbox, Python x64 3, WinSCP, PuTTY, Visual Studio Code, Winamp, Audacity, Spotify, K-Lite Codecs, WinDirStat, and 7-Zip.

For the last few months I’ve been using Vivaldi as my default browser, but this time around I’m gonna stick with Edge for a while and see how it goes.

Sign in to Chrome, that syncs my bookmarks and browser extensions, including LastPass, which has credentials for pretty much everything I do online.

It always surprises me how usable the system is at this point. So much stuff is web/cloud based now that a connection, a browser and LastPass is enough to get into Google Drive, Gmail, Twitter, Facebook, GitHub

The Big Bag O’ Fonts

I have a lot of fonts. They’re all backed up in Dropbox. Ctrl-A, right-click, “Install for All Users…”

I like fonts.

The Command Line Stuff

Windows Subsystem for Linux
- installed by opening an Administrator command prompt and running wsl --install
Powershell
- Not to be confused with Windows Powershell, which ships as part of Windows 10
Windows Terminal
- not to be confused with Windows Terminal Preview, which is the same, but different.
oh-my-posh
- …the necessary nerd fonts are already installed at this point ‘cos they’re part of the Big Bag O’ Fonts I installed earlier.
Cygwin64
- Default selection, plus curl, wget, and openssh
- Everything goes into C:\Windows\Cygwin\bin, and I add this to the system path. This becomes my default location for any utilities I want to be available from the command line, like ffmpeg – like /usr/local/bin on a *nix system.
git for windows
OpenSSL for Windows
ffmpeg
- using the Windows precompiled binaries from https://github.com/BtbN/FFmpeg-Builds/releases
- Usually the master latest win64 lgpl build is the one to go for.

Languages and SDKs

.NET Core SDK 3.1
.NET Core SDK 5
.NET Core SDK 6
nodeJS 16.x
- and select the option to install build tools for building native C/C++ modules.
Ruby for Windows

GUI Apps

Then it’s a big long list of apps I use on a daily basis, which I’m listing here as much for my benefit as yours:

Zoom, Slack, WhatsApp, Signal, Telegram, Discord…
- yes, every single one of these platforms is the communication tool of choice for at least one friend, collaborator or online community that I’m connected to. And that’s not counting the ones who use Twitter DMS, Facebook Messenger, Facetime, Skype… or even good old fashioned email.
OBS Studio
- great for streaming, great for screen recording, great for doing cool stuff with overhead cameras and pens and paper during Zoom training sessions.
- Scenes and profiles are all copied over from the old system; they’re stored at C:\Users\dylan\AppData\Roaming\obs-studio\
SnagIt and Camtasia
- I set up SnagIt with Shift+Ctrl+\ as a quick capture hotkey.
Microsoft Office
- Powerpoint, Word, and Excel, I use on a daily basis. Access, maybe once in a blue moon. Teams when I have no choice. Outlook, OneNote, and Publisher, I don’t think I’ve ever opened except by mistake, but hey, they’re part of the installer now, and it looks like the screen in the Office installer where you choose the bits you actually want has been banished to the great recycle bin in the sky, so I guess I’ve got them too. Meh. Storage is cheap.
Microsoft Visio
- Visio rocks. For architecture diagrams, floor plans, presentation slides that are just a bit too complicated for PowerPoint, I find it indispensable.
Visual Studio 2019
- including .NET Framework 4.6.2, 4.7 and 4.8, for running training with clients using .NET Framework 4.x
Visual Studio 2022

After installing Visual Studio and Resharper, to into Tools > Options > Source Control > Plugin Selection and set it to None. Otherwise you’ll get an error message “Files still read-only” when trying to do various refactoring and renaming things with Resharper.

https://stackoverflow.com/questions/26715783/resharper-function-shows-files-still-read-only

Docker Desktop for Windows
- using the WSL2 subsystem, not the old HyperV system
SQL Server Management Studio
- these days I always run the actual server inside a Docker container, but the SQL client tools are still useful
Adobe Creative Cloud
- Photoshop, Illustrator, Premiere, AfterEffects, Dimension, Audition, Character Animator.
- …and remember to go into Preferences in the Creative Cloud Desktop app, enable “older apps”, and install the Extendscript Toolkit CC for running JS automation scripts in AfterEffects.
Axialis IconWorkshop and CursorWorkshop
JetBrains Toolbox
- and then DataGrip (for poking PostgreSQL stuff running in Docker containers), WebStorm (for nodeJS), Resharper, Rider, and Pycharm
NCrunch
NGrok
Open Hardware Monitor
Microsoft PowerToys
Typora
Beyond Compare
Instant Eyedropper
SysInternals suite
- unzip the whole thing into C:\Program Files\SysInternals\
Postman
qBittorrent
Protocol Buffers
Elgato StreamDeck

And there you go - fresh install to a usable system in less than a day. Sure, I’ll be finding odd bits that aren’t set up quite right for a week or two, and there will invariably be some app I reach for one day a few months from now and discover I forgot to install it, but WIndows 10 just feels so much snappier than 11 ever did – and best of all, I can use the clipboard in Photoshop again. 😊

Rebooting the London .NET User Group

2022-06-20T12:00:00+00:00

It’s been a difficult couple of years for tech events^{[citation needed]}, but in-person conferences are back, venues and event spaces are open for business as usual, and we’re really happy to announce that the London .NET User Group (aka LDNUG) will be running regular in-person meetings again starting in August.

Here’s a quick FAQ to get you up to speed on who we are, what we’re doing, and how to get involved.

What is the London .NET User Group?

It’s a monthly, free community meetup for people based around London who develop software using .NET and related technologies.

Who’s behind it?

Ian Cooper started the group in 2002, and he’s been running it ever since. Dylan Beattie (that’s me!) has been helping run the group since 2015, and various other people help out from time to time.

Why do you do it?

Three reasons:

It’s a great way to meet people who are working on the same things that you are. Maybe they can help you, maybe they can hire you, maybe you can hire them? Over the 20 years we’ve been running the group, people have found jobs, contracts, employees, open-source collaborators – and made quite a few friends along the way, too.
It’s a great way to hear first-hand from folks who are working with the latest & greatest tech – or who are applying more established technology in some interesting and unexpected ways. Listening to people talk about what they actually did, how they did it, what worked and what didn’t work, can provide a really valuable perspective if you’re thinking about trying out a new technology on one of your own projects.
It’s a platform for speakers to try out their talks and content in front of a friendly audience, figure out what works, and get constructive feedback on their presentations. Many of the people you’ll see speaking at big international conferences did their first talk at a group like LDNUG – and, just as importantly, many of those same “big name” speakers use the group to try out new talks and present new material before submitting it to those same big conferences.

What happens at the meetups?

Folks normally start arriving around 18:30. Around 19:00, we’ll kick off with a few announcements about forthcoming events and interesting stuff that’s going on in the wide world of .NET. We aim to have at least two speakers at each event; they’ll do 45-60 minutes each, with a break in the middle for drinks and chat. Talks should finish by 21:30, and there’s usually a few people heading to a nearby pub afterwards for drinks and chat.

Will your meetups be streamed online?

Probably not. For now we’re relying on companies to provide event space to host our meetups, and live streaming adds a whole extra level of complexity both for the organisers (us!) and for the venue that’s hosting us.

Will your meetups be recorded?

We’re looking into it. We’d love to be able to publish talks and videos on YouTube after each event, but we’re still figuring out the details.

What sort of things do you talk about?

Anything and everything, as long as it’s vaguely connected with building software using .NET. We’ve hosted lots of deep-dive technical talks about things like C# language features, .NET memory management, and performance optimisations. We’ve hosted introductory talks to technologies like Blazor and SignalR, we’ve had talks about related technologies like SQL, frontend development, and node.JS, and we’ve hosted speakers talking about all kinds of topics related to working in development – security, culture, diversity, recruitment.

Can I do a talk?

Sure you can. Submit your talk via our Sessionize, and we’ll be in touch to find a date that works and figure out some details.

I can’t join in person - can I do an online talk via Zoom?

Probably not. For now, we’re prioritising in-person events, with live speakers talking to an in-person audience; after two years of online events we’re all a bit burned out on Zoom talks and “hey, leave a comment in the chat”.

Can I come along and watch?

Yeah, absolutely! Join our Meetup group, follow us on Twitter, and we’ll let you know when we announce new dates.

Can my company host a meetup?

Probably! If you’ve got some venue or event space in central London (zone 1 & 2), and you’d like to invite a few dozen .NET developers to see your offices and meet some of your team, hosting a meetup is a great way to make it happen. We’ve done this many times with companies who are hiring developers, which tends to work out well, but even if you’re not hiring it’s a great way to support the London .NET community.

You’ll need to provide a meeting space, auditorium, or similar, equipped with a projector/TV screen, ideally 1920x1080 HD or better, with HDMI. We’ll need audio in case speakers have music/sound in their presentations, and speakers will need to have their own laptop on stage with them when they’re presenting so they can run live demos.

We’ll also ask you to provide refreshments: water, tea, coffee, soft drinks and snacks. (If you want to provide beer, wine, pizza, buffet snacks, etc. for attendees, that’s very much appreciated, but not required.) Guest access to wifi is normally a good idea too, although we can work around this as long as we know about it in advance. In exchange, you’ll get a chance to show off your offices, tell our attendees a bit about your company and what you’re working on, and feel good about supporting local community events. 😊

If that all sounds like something you’re interested in, drop us an email and let’s work something out. We aim to schedule events at least three months in advance.

Where do I find out more?

All our events are announced via our Meetup group:

https://www.meetup.com/london-net-user-group/

We’re also on Twitter:

https://twitter.com/LondonDotNet

If you’re interested in speaking, submit a talk via our Sessionize:

https://sessionize.com/london-dotnet-user-group/

If you want to get in touch with the organisers, you can email us here:

Dylan Beattie: dylan@dylanbeattie.net
Ian Cooper: ian_hammond_cooper@outlook.com

A VBA macro for storyboarding PowerPoint slide decks

2022-04-18T12:00:00+00:00

I’m working on a couple of new talks at the moment, and thought people might be interested in seeing a bit about how I actually put them together.

Step 1: Write the Talk

All my talks start life as a written transcript. I speak at 1,000 characters per minute, so for a 1-hour talk, that’s about 60,000 characters – between 10,000 and 12,000 words. But the first draft of a talk is normally more like 15,000-20,000 words, because this is where I throw absolutely everything at the wall to see what sticks. All the ideas, all the stories, everything that’s been rattling around in my brain or my Evernote library since I started thinking about writing the talk.

Once I’ve got everything drafted, I edit, edit, and edit again. I move things around, figure out how to connect the sections so the talk flows properly, and then go through and edit out the bombadil until I have something about the right length which I think hangs together nicely.

Good morning, everybody! I’m Dylan, and I’m here today to talk to you about yak shaving. You’re probably wondering what yak shaving is. Well… you know that thing where you need to do something really, really trivial? Like changing the year from 2021 to 2022 in the copyright footer on your website… shouldn’t be a big deal, right? So you check out the code, and change it, and commit the change, and then you go to deploy it to production and discover that the SSL certificate used by your deployment pipeline has expired. So you roll a new certificate… but you can’t install the new key onto the production server because your user account doesn’t have permission to do that. So you try to sudo onto the box, but you don’t know the root password, and the only person who does know it is backpacking in the Cordillera Huayhuash until March, so the only way to get it live is to physically log onto the server in recovery mode, but nobody’s been into the server room for so long that the lock on the door has rusted shut, so you go to the hardware store and get some WD40, and you squirt it into the lock, and whack it a few times with a hammer to try and loosen it up… and somebody walks past and sees you attacking the server room door lock with a hammer and asks what you’re doing, and you reply “CAN’T YOU SEE I’M TRYING TO UPDATE THE COPYRIGHT FOOTER ON THE WEBSITE?”

Well, that’s yak shaving.

I’ll run through that at least once, just me, talking into a camera. No slides or anything yet; this is just me trying all the words on and checking they fit properly, making sure I know how to pronounce them all (how do you pronounce “Cordillera Huayhuash”?), and double-checking the timing.

If that’s a train wreck, I’ll go back, edit some more, and do it again. Eventually, I’ll have a written transcript which I know is pretty much the right words, in the right order, and the right length.

Step 2: Work out what slides I want

Then, I’ll go through that transcript and put in slide markers. This is all real low-tech - I’m just hacking around plain text files here - but I’ll identify the exact points in the presentation where I want to show a slide (or add/change something on an existing slide), and put in text markers. Markers will often be right in the middle of a sentence – if that’s the moment I want the slide to appear, that’s where I put the marker:

[SLIDE: Title slide] Good morning, everybody! I’m Dylan, and I’m here today to talk to you about [SLIDE: that yak shaving cartoon from Ren & Stimpy] yak shaving. You’re probably wondering what yak shaving is. Well… you know that thing where you need to do something really, really trivial? Like [SLIDE: screenshot of website footer copyright message] changing the year from 2021 to 2022 in the copyright footer on your website… shouldn’t be a big deal, right? So you check out the code, and change it, and commit the change, and then you go to deploy it to production and discover that [SLIDE: expired certificate error message] the SSL certificate used by your deployment pipeline has expired. So you roll a new certificate… but you can’t install the new key onto the production server because [SLIDE: insufficient permisssions message] your user account doesn’t have permission to do that. So you try to sudo onto the box, but you don’t know the root password, and the only person who does know it is [SLIDE: picture of somebody hiking in the mountains] backpacking in the Cordillera Huayhuash until March, so the only way to get it live is to physically log onto the server in recovery mode and reset the root password, but nobody’s been into the server room for so long that [SLIDE: a really, REALLY rusty door lock] the lock on the door has rusted shut, so you go to the hardware store and [SLIDE: picture of WD40] get some WD40, and you squirt it into the lock, and [SLIDE: picture of a big old hammer] whack it a few times with a hammer to try and loosen it up… and somebody walks past and sees you attacking the server room door lock with a hammer and asks what you’re doing, and you reply [SLIDE: picture of me looking really really stressed, with a speech bubble] “CAN’T YOU SEE I’M TRYING TO UPDATE THE COPYRIGHT FOOTER ON THE WEBSITE?”

I save that as a text file.

Step 3: Storyboard the PowerPoint deck

I have a PowerPoint template that contains my preferred fonts and colour scheme – and which also contains a VBA macro I wrote, which will take the transcript TXT file and turn it into a slide deck. It’ll put the slide marker text as the headline on each slide, and put the relevant chunk of the transcript into the slide notes area:

Public Sub BuildSlideDeckFromTranscript()
    Dim filePath As String
    Dim contents As String
    
    filePath = PromptForFilePath()
    If (filePath = "") Then Exit Sub
    contents = ReadUtf8TextFile(filePath)
    
    Dim slideTexts As Variant
    ' vbTextCompare indicates case-insensitive string comparison
    slideTexts = Split(contents, "[SLIDE: ", , vbTextCompare)
    slideCount = UBound(slideTexts, 1)
    If slideCount <= 0 Then
        MsgBox ("I didn't find any slide markers in that file.")
        Exit Sub
    End If
        
    If MsgBox("Found " & slideCount & " slide markers. Do you want to create slides now?", vbYesNo) <> vbYes Then Exit Sub
    Dim pptLayout As CustomLayout
    Set pptLayout = ActivePresentation.Slides(1).CustomLayout
    SlideIndex = ActivePresentation.Slides.Count
    For Index = LBound(slideTexts, 1) To UBound(slideTexts, 1)
        slideText = slideTexts(Index)
        tokens = Split(slideText, "]")
        If (UBound(tokens, 1)) = 1 Then
            SlideIndex = SlideIndex + 1
            Set pptSlide = ActivePresentation.Slides.AddSlide(SlideIndex, pptLayout)
            pptSlide.Shapes(1).TextFrame.TextRange.Text = tokens(0)
            pptSlide.NotesPage.Shapes(2).TextFrame.TextRange.Text = tokens(1)
        End If
    Next Index

End Sub

Function PromptForFilePath() As String
    Dim filePicker As FileDialog
    Set filePicker = Application.FileDialog(msoFileDialogFilePicker)   
    Dim selectedItem As Variant
    Dim filePath As String
    With filePicker
        .InitialFileName = ActivePresentation.Path
        .Filters.Add "Text files", "*.txt", 1
        If .Show = -1 Then
            For Each selectedItem In .SelectedItems
                filePath = selectedItem
            Next selectedItem
        End If
    End With
    PromptForFilePath = filePath
End Function

Function ReadUtf8TextFile(filePath As String) As String
    Dim contents As String
    Dim stream
    Set stream = CreateObject("ADODB.Stream")
    With stream
        .Charset = "utf-8"
        .Open
        .LoadFromFile (filePath)
        contents = .ReadText()
        .Close
    End With
    Set stream = Nothing
    ReadUtf8TextFile = contents
End Function

That’ll give me a deck full of blank slides, with each slide’s title set to the slide marker text, and the relevant bit of transcript in the slides’ speaker notes area:

Step 4: Make the actual slides

Finally, I go through the entire deck and replace each placeholder slide with the image, video, quote, or animation. Some slides will go through a couple of iterations, especially if I’m making custom visualisations or animations. Sometimes, I’ll join 2-3 sections into a single slide with lots of animation; sometimes I’ll split a slide out into a couple of separate ones.

I find this a really efficient way to work, but it also means that when I’m thinking about what I want to show on my slides, I’m not thinking in terms of PowerPoint layouts and bullet points; I’m thinking about what I actually want people to be looking at. It also means if I share a PDF of the deck, I can include the speaker notes, so underneath each slide in the PDF you see what I was actually talking about during that part of the presentation. Which I think is rather neat.

“The Art of Code” Workshops with fwdays to support Ukraine

2022-04-02T12:00:00+00:00

Ukraine is a wonderful country. I’ve been there many times, speaking at conferences and meeting Ukrainian developers; it’s a fascinating place to visit, and one I was very much looking forward to revisiting when the pandemic was over. What is happening there right now is horrific. Vladimir Putin’s illegal invasion, and the death and destruction that his actions have inflicted on the people of Ukraine, are unforgivable.

Donations of money are very, very welcome, whether to help equip the Ukrainian military or to assist humanitarian efforts providing medical assistance, transportation, and support to the millions of people who have lost their homes and livelihoods.

But the people I know in Ukraine are also working really hard to keep the Ukrainian economy working: delivering services, paying salaries, paying taxes. Over the past few years, I’ve run lots of online workshops with the crew from fwdays. Theyre obviously not able to run any of their regular conferences and in-person training at the moment, so instead they’re organising a series of online workshops with all the profits going to support humanitarian and defence causes in Ukraine.

To help them out with this, I’m running a series of half-day workshops based around ideas from my talk “The Art of Code” and the idea of programming as a creative art form – and I’ll be donating all my fees from these workshops to support Ukraine.

We’ll be kicking off on April 14th with a workshop about ray tracing in JavaScript. During four hours of live hands-on coding, we’ll build a fully featured ray tracer, in pure JavaScript, that runs directly in your web browser. You’ll learn the principles of 3D computer graphics, and how to simulate lighting, shading, reflection, and visual effects to create photo-realistic scenes. Not only that, but along the way you’ll learn about some state-of-the-art JavaScript and web technologies, including the HTML Canvas API, web worker, ES modules, clamped arrays, and many more.

Tickets are on sale now, with prices starting from UAH2534 (about £65/€75). It will all be online, using Zoom, Slack, GitHub, and various web-based collaboration tools. Find out more and book now at:

https://fwdays.com/en/event/the-art-of-code-ray-tracing-in-java-script

Any questions about it? Find me on Twitter or drop me an email at dylan@dylanbeattie.net.

We’ll have more workshops coming up soon, covering everything from designing esoteric programming languages to fractal geometry and Conway’s Game of Life, so keep an eye out for those. You’ll learn some useful real-world programming techniques, you’ll discover fun, creative, and inspirational ways to apply those techniques – and you’ll be helping to support Ukraine.

Hope to see you there.

Database already exists when running EF Core migrations

2022-02-01T12:00:00+00:00

I’m working on a project that uses Entity Framework Core, and I’m using EF Core Migrations to manage database state. Earlier today, I grabbed a fresh (obfuscated) snapshot of the production database, copied it across to my workstation, and tried to run dotnet ef database update to apply the latest migrations from my current branch:

[14:37:27 INF] Entity Framework Core 6.0.1 initialized 'MyDbContext' using provider 'Microsoft.EntityFrameworkCore.SqlServer:6.0.1' with options: using lazy loading proxies
[14:37:27 ERR] Failed executing DbCommand (15ms) [Parameters=[], CommandType='Text', CommandTimeout='60']
CREATE DATABASE [my-database]
COLLATE SQL_Latin1_General_CP1_CI_AI;
Failed executing DbCommand (15ms) [Parameters=[], CommandType='Text', CommandTimeout='60']
CREATE DATABASE [my-database]
COLLATE SQL_Latin1_General_CP1_CI_AI;
Microsoft.Data.SqlClient.SqlException (0x80131904): Database 'my-database' already exists. Choose a different database name.

It’s failing because the database exists… but I know the database exists – I just created it! It shouldn’t be trying to create a new database at all… right?

It turns out that the database exists, but the user account doesn’t. My application’s appSettings.development.json contains a database connection string which specifies a username and password:

{
  "ConnectionStrings": {
    "MyDatabase": "Server=localhost;Database=my-database;User Id=my-user;Password=p@ssw0rd;MultipleActiveResultSets=true;Connect Timeout=1"
  }
}

That username/password is still a valid server login on localhost (and, hey, look at that super-secret password right there!), but there’s no corresponding user in the my-database database, so I’m guessing what happens here is EF Core connects to the server (which works), tries to open the database, can’t open it, and so assumes it doesn’t exist – and so attempts to create it, which fails because it already exists.

I recreated the database user linked to the login (and added them to the db_owner role so that EF Core has permission to create tables, indexes, etc.):

USE [my-database]
GO
CREATE USER [my-user] FOR LOGIN [my-user] WITH DEFAULT_SCHEMA=[dbo]
GO
USE [my-database]
GO
ALTER ROLE [db_owner] ADD MEMBER [my-user]
GO

and it worked just fine.

Securing Admin Pages with ASP.NET and Azure AD

2022-01-25T12:00:00+00:00

I recently had to implement an authentication pattern which I must have built a dozen times over the course of my career, on various platforms ranging from classic ASP to Ruby on Rails. This time around it’s in ASP.NET Core on .NET Core 3.1, and although all the individual components are easy enough to plug in, I couldn’t find a resource describing how to implement this exact use case, so here’s how I did it.

The Scenario

There’s a website at https://example.com/ that’s available to the general public. No security, no user registration, no authentication.
There is an admin area at https://example.com/admin which is only accessible to employees of Example Company Ltd
We can assume that anybody with an @example.com email address is an employee

What makes this a little easier than the Olden Days is that Example Company Ltd runs all their email on Office 365, so there’s an Azure OpenID Connect endpoint I can use to authenticate users. What makes it a little more difficult is that that although Example Company Ltd use Office 365 and they all have Microsoft accounts, they don’t actually have an Azure subscription (and don’t want to set one up), so the organisation that’s managing the Azure app registrations (i.e. me) isn’t the same as the organisation that employs the authorised users.

Anyway. Here’s how it all works. You’ll need to register your app in the App registration area of the Azure dashboard, which you should be able to find here. You’ll also need a section in appsettings.json called "AzureAd":

{
  "AzureAd": {
    "Instance": "https://login.microsoftonline.com/",
    "Domain": "example.com",
    "ClientId": "<YOUR-AZURE-REGISTERED-APP-CLIENT-ID>",
    "TenantId":  "common"
  }
}

The ClientId there is the Application (client) Id from the Azure dashboard.

Install the NuGet packages for the Microsoft.Identity.Web middleware:

dotnet add package Microsoft.Identity.Web
dotnet add package Microsoft.Identity.Web.UI

Then we need to add a few things to our Startup.cs:

Define a string constant for the policy name, and another one for the authorised domain:

private const string EMPLOYEES_ONLY_POLICY = "EmployeesOnly";
private const string AUTHORIZED_DOMAIN = "example.com";

Next, add a chunk of code to the top of ConfigureServices:

var azureConfig = Configuration.GetSection("AzureAd");
services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
  .AddMicrosoftIdentityWebApp(options => {
    options.ClientId = azureConfig["ClientId"];
    options.Instance = azureConfig["Instance"];
    options.TenantId = azureConfig["TenantId"];
    options.Events.OnRedirectToIdentityProvider += context => {
      context.ProtocolMessage.DomainHint = AUTHORIZED_DOMAIN;
      return Task.CompletedTask;
    };
});

There’s an overload for AddMicrosoftIdentityWebApp which lets you just pass in the config section, but I couldn’t find any way to do this and then register the event handler for OnRedirectToIdentityProvider, which we’re using to pass a DomainHint to the Azure/Microsoft login endpoint. This means if the current user has ever signed in using an @example.com email address, it’ll default to using that one in future, which makes for a smoother login process.

Where we add Razor pages to our app, we’re adding the Microsoft Identity UI, which gives us prebaked UI bits like the “access denied” page:

services.AddRazorPages().AddMicrosoftIdentityUI();

Next, we define our authorization policy, but we don’t actually attach it to anything yet. This means that all our app content will be available by default (so we don’t need to go around putting [AllowAnonymous] on all our controllers), but we can lock down specific areas, controllers or actions by attaching this policy to them. RequireAssertion here is the extension point we can use to run arbitrary code checks on the authorization context - in this example, verifying that the user’s identity name (which is their email address when using Azure AD) ends with the authorized domain.

services.AddAuthorization(options => {
  options.AddPolicy(EMPLOYEES_ONLY_POLICY, new AuthorizationPolicyBuilder()
    .RequireAuthenticatedUser()
    .RequireEmailDomain(AUTHORIZED_DOMAIN)
    .Build());

RequireEmailDomain isn’t one of the policies provided out of the box, but that’s no problem; we can add it using an extension method:

public static class AuthorizationPolicyExtensions {
  public static AuthorizationPolicyBuilder RequireEmailDomain(
    this AuthorizationPolicyBuilder builder, string domain
  ) {
    domain = domain.StartsWith("@") ? domain : $"@{domain}";
    return builder.RequireAssertion(ctx => {
      var email = ctx.User?.Identity?.Name ?? "";
      return email.EndsWith(domain);
    });
  }
}

ℹ️ Notice that we’re prepending an @-sign here. If we omitted this, you could sign in by registering a domain like fakeexample.com and setting up a Microsoft login using that domain, because "script.kiddie@fakeexample.com".EndsWith("example.com") will return true.

Finally, in the Configure method, we need to make sure the app is using both authentication and authorisation (as an aside, don’t you love how clear this syntax is?)

app.UseAuthentication();
app.UseAuthorization();

And then map a controller route for our secure area, which includes the authorization policy:

app.UseEndpoints(endpoints => {
    
    endpoints.MapControllerRoute("secure", 
        "{area:exists}/{controller=Home}/{action=Index}/{id?}")
        .RequireAuthorization(EMPLOYEES_ONLY_POLICY);
        
    endpoints.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");
    endpoints.MapRazorPages();
});

This means any controllers and actions that are part of the secure area will have this authorization policy applied, and the rest of the app is still available to unauthenticated and unauthorised users.

Here’s the whole Startup.cs with those extra bits in place:

// Startup.cs

using System.Threading.Tasks;
using Microsoft.AspNetCore.Authentication.OpenIdConnect;
using Microsoft.AspNetCore.Authorization;
using Microsoft.Identity.Web;
using Microsoft.Identity.Web.UI;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace WebApp_OpenIDConnect_DotNet {
  public class Startup {

    const string EMPLOYEES_ONLY_POLICY = "EmployeesOnly";
    private const string AUTHORIZED_DOMAIN = "example.com";

    public Startup(IConfiguration configuration) {
      Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    // This method gets called by the runtime. Use this method to add services to the container.
    // <Configure_service_ref_for_docs_ms>
    public void ConfigureServices(IServiceCollection services) {
      var azureConfig = Configuration.GetSection("AzureAd");
      services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
        .AddMicrosoftIdentityWebApp(options => {
          options.ClientId = azureConfig["ClientId"];
          options.Instance = azureConfig["Instance"];
          options.TenantId = azureConfig["TenantId"];
          options.Events.OnRedirectToIdentityProvider += context => {
            context.ProtocolMessage.DomainHint = AUTHORIZED_DOMAIN;
            return Task.CompletedTask;
          };
        });

      services.AddControllersWithViews();
      services.AddRazorPages().AddMicrosoftIdentityUI();
      services.AddAuthorization(options => {
        options.AddPolicy(EMPLOYEES_ONLY_POLICY, new AuthorizationPolicyBuilder()
          .RequireAuthenticatedUser()
          .RequireEmailDomain(AUTHORIZED_DOMAIN)
          .Build());
      });
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env) {
      if (env.IsDevelopment()) {
        app.UseDeveloperExceptionPage();
      } else {
        app.UseExceptionHandler("/Home/Error");
        app.UseHsts();
      }

      app.UseHttpsRedirection();
      app.UseStaticFiles();

      app.UseRouting();
      app.UseAuthentication();
      app.UseAuthorization();

      app.UseEndpoints(endpoints => {
        endpoints.MapControllerRoute("admin", "{area=admin:exists}/{controller=Home}/{action=Index}/{id?}")
          .RequireAuthorization(EMPLOYEES_ONLY_POLICY);
        endpoints.MapControllerRoute(
          name: "default",
          pattern: "{controller=Home}/{action=Index}/{id?}");
        endpoints.MapRazorPages();
      });
    }
  }

  public static class AuthorizationPolicyExtensions {
    public static AuthorizationPolicyBuilder RequireEmailDomain(
      this AuthorizationPolicyBuilder builder, string domain
    ) {
      domain = domain.StartsWith("@") ? domain : $"@{domain}";
      return builder.RequireAssertion(ctx => {
        var email = ctx.User?.Identity?.Name ?? "";
        return email.EndsWith(domain);
      });
    }
  }
}

One thing I haven’t figured out yet is how to override the built-in bits of UI for things like “Access Denied”. The policy works fine, but if you sign in with a valid Microsoft account that isn’t a @example.com email, you get this generic “Access Denied” message:

It’d be nice to replace this with something that explains only people with @example.com email addresses are allowed in.

I also have no idea yet how you go about testing this - this sort of federated authentication is very much in the “sufficiently advanced technology and therefore indistinguishable from magic” category, and security has always been hard to test even when it doesn’t involve exchanging encrypted tokens with Actual Microsoft Infrastructure.

But, hey, it works, and only adds a few lines of code. I’m quite happy with that – and the fact that all the authentication, password resets, locked accounts, setting up accounts for new users is all running on somebody else’s stack?

That, my friends, is what we call progress.

Copying Your Hard Drive With Robocopy

2021-11-12T12:00:00+00:00

Time to reinstall Windows - after 18 months of running Insider builds, hacking around with weird WSL/Docker network configurations, and trying out more weird video streaming hardware than you can possibly imagine, I’ve hit the point where certain things just don’t work any more.

The plan:

Copy everything from the C: drive to somewhere safe.
Do a clean reformat & reinstall.
Yay everything works now.

Turns out it’s actually way harder to copy everything from your C: drive onto another drive when you’re still running from that C: drive. Lots of files are locked because they’re in use, and a bunch of weird junction points (Windows-speak for symbolic links) that in some cases can actually lead to infinitely deep recursive nested directories.

Here’s the command that worked in the end:

robocopy /s /b /z /xo /xj /V /R:0 /W:0 /copy:DT C:\ G:\backups\c\

And here’s how that breaks down:

robocopy	
/s		   Copy all subdirectories (except empty ones - use /e for that)
/b		   Copies files in backup mode (overrides file & folder permissions)
/z		   Copy files in restartable mode
/xo		   Ignore older files - useful if you need to restart the whole thing
/xj		   Ignore junction points.
/V		   Verbose. List everything as it's copied. Handy to see what's going on.
/R:0	   Retry files zero times if there's an error
/W:0	   Wait zero seconds before retrying
/copy:DT   Copy only data and timestamps (i.e. do not copy permissions, attributes)

Check out the full Robocopy docs at https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy

On The Road Again

2021-08-17T12:00:00+00:00

This time last week, I was in Camber, a tiny village on the English coast. After two weeks of mostly vacation, including all the meteorological delights that you get for free when you holiday on the south coast of England, I was catching up on emails and looking at the next few months and beginning to, maybe, make some tentative plans around some in-person events and travel before the end of the year.

Then I got an email out of the blue from the folks at SaltPay – they were opening a new tech hub in Porto, and would I be able to give a talk? We set up a Zoom call, and after chatting for a few minutes I realised what they were proposing… “whoa, hang on a second, you mean in person? You’re inviting me to fly to Portugal on Monday?” Yes, that’s exactly what they were inviting me to do – and, it turns out that’s actually possible right now, thanks to SCIENCE! I’ve had both my COVID vaccinations, and Portugal is one of the European countries that recognises the UK vaccination certificate, which means I can travel without having to quarantine at either end, and TAP are flying direct from Gatwick to Porto a few times a week.

Now, on one hand, travel is a lot more complicated than it used to be. Between COVID and Brexit, the days when you could book a flight or Eurostar and be in Europe in time for dinner probably aren’t coming back any time soon. But on the other hand, it’s still a lot easier and cheaper than going to, say, Russia or Tanzania. The European Union is a wonderful anomaly; for most of the folks on this planet, travel is an expensive mess of visas, paperwork, bureaucracy - and vaccinations. Big deal. You sign the forms, you get the jabs, you pay the fees, and off you go.

The logistics are a bit complicated, partly because so much of this is time-sensitive, and partly because the rules and procedures keep changing. There are the lateral flow tests you do at home which aren’t valid for travel; there’s the old-style PCR tests that are valid for travel but results take 24 hours, and there’s the new lateral flow tests which are valid for travel if you get them done at a clinic or get a lab to certify the results… PCR tests have to be done within 72 hours before departure, LFD tests within 48, and there are passenger locator forms to fill out in both directions. Mainland Portugal is on the UK’s “amber list”, so to re-enter the UK you need to have booked (and paid for!) another COVID test that you’ll take within 2 days of arriving home - and, you guessed it, it can’t be a free NHS one.

I had to show my test result and vaccination certificate when I checked in for my flight, and again at passport control on arrival in Porto. although at no point did anybody ask me to remove my mask to check my face matched my passport photo, which I thought was a little incongruous. There was also a slight entertaining moment when the airline stewardess doing the safety demonstration reminded people to take off your COVID mask before you put on your oxygen mask, but apart from that, it was really nothing remarkable. Gatwick Airport is much the same - everything’s flying out of North Terminal at the moment, and South is completely closed while they’re doing some sort of building work. Porto’s much the same as it was when I was last here in January 2020, albeit a lot warmer and full of British holidaymakers.

One thing I hadn’t expected which really stood out for me was the sense of anticipation. Even with the best tech and the best organisation in the world, online events just sort of blur into the ennui of lockdown life. I’ve done dozens of online talks over the last 18 months, and the night before an online talk, I’m not thinking about the talk; I’m thinking about the tech; all the various cables and devices that need to work flawlessly for the presentation to work. But on Sunday evening, I was out in Porto having a francesinha and a Super Bock and I realised that for the first time in over a year, all I was thinking about was what I was actually going to talk about. I wasn’t worrying about microphones and screen-sharing and whether I’d be able to read the questions in the chat; I was thinking about the stories I wanted to tell. You know what? It felt great.

The event was excellent, and an interesting insight for me into what in-person events look like in the post-COVID world. Everybody wearing masks, everybody taking lateral flow tests on arrival, colour-coded wristbands showing how comfortable you were with social proximity.

SaltPay’s new tech hub is fantastic – and did I mention this was my first in-person talk since March 2020? ACTUAL AUDIENCE! IN THE ROOM! Well, mostly; it was a hybrid event – some folks joined remotely over Zoom, some folks had flown in specially to be there in person; some speakers were there in person, some were there online, and of course being a brand new venue this was the first time the AV tech had really been put through its paces and there was the odd technical glitch, but on the whole, it all worked beautifully.

I should probably mention that SaltPay are hiring for all kinds of tech and tech-adjacent roles in Porto, Prague, London, Lisbon and all over Europe. They’re a great bunch of people, doing some very interesting stuff with payment systems – if you’re interested, check out saltpay.co/about.html#jobs)

And now I’m at Armazem, a wonderful “barely converted warehouse” down by the river, opposite the old customs warehouse (which you might know as the venue we use for NDC Porto when that’s in town). It’s a weird mashup of a bar, a cafe, an art gallery and an antiques shop, and it’s absolutely delightful. They have wifi, they have cold beer, and believe me, after walking the long way round from Castelo do Queijo, down the coast to Foz and back into town along the Douro, the cold beer is very much necessary. But it definitely feels like things are slowly returning to some semblance of normality.

Of course, the caveat to all of this is that daily COVID tests are just fine unless one of them comes back positive. So far, so good, but if one of those little indicator strips suddenly shows a second red line, all bets are off and instead of catching my flight home I’m suddenly working out what’s involved in self-isolating in Portugal for 10 days. I keep repeating that line from the agile manifesto about “responding to change over following a plan”, because if there’s one thing we’ve all learned over the last 18 months, it’s that COVID-19 couldn’t give a toss about anybody’s plans. At the moment it definitely feels like things are opening up again; I’m hoping to make it over to Antwerp next month to spend a few days with the folks from Axxes, DDD East Midlands is still on track for an in-person conference in Nottingham in October, and November sees the return of GOTO Copenhagen, BuildStuff and NDC Oslo.

I’m sure that there are going to be some bumps on the road back to something approaching normality - all the event organisers I know are taking COVID very seriously indeed, but I won’t be surprised if one or two events find themselves having to improvise in a hurry if travel restrictions change or somebody reports a positive test at a speaker hotel or something. But, y’know, the unpredictability is part of what makes international travel exciting, and I have to say, the rest of 2021 is looking pretty damn exciting right now.

UK Road Signs (Redesigned by Google)

2021-07-25T12:00:00+00:00

Last year, Google redesigned the icons used for many of their web and mobile apps, as part of their “rebranding” of G Suite to Google Workspace. And they took a bunch of distinctive, easily recognisable icons and… well, they kinda made them all look the same. There is a much more in-depth discussion of it over on TechCrunch, but it got me thinking: what if you applied the same idea to something really familiar? Something that millions of people rely on every day, something that hasn’t changed in decades, something where clear signage you can recognise at a glance might literally be a matter of life and death?

What if UK road signs were redesigned by the Google Workspace team?

How (Not) to Respond to a CFP

2021-07-23T12:00:00+00:00

As the world tentatively relaxes COVID restrictions and we contemplate the prospect of actual in-person conferences again before the end of the year, it seems like a good time to revisit one of the perennial questions that people ask about speaking at conferences and technology events: how do I write a good CFP submission?

Many big technology conferences hold an open CFP¹, and competition for speaking spots can be fierce; events I’ve worked with typically accept less than 10% of the talks that are submitted. Your submission will be reviewed by the programme committee, who decide which talks to accept, but it may also end up in the conference programme, often in the exact form you submitted it, where the conference attendees will use it to decide which talks to attend. The talks that get accepted normally represent a diverse range of speakers, technologies, presentation styles and audience levels, and consequently they’re all different – there is no secret formula for writing a good proposal.

That said, there are some very common factors among the talks that don’t get accepted, so rather than telling you how to write a proposal, I’m going to share some tips about what not to do.

1. Too short

There’s a particular kind of Hollywood action movie that can be perfectly described in fewer than ten words. “Die Hard on a bus”, “The Matrix, but with vampires”, “Gangsters killed his dog… now he wants revenge.”

Well, dear readers, this isn’t Hollywood, and if you want to get your talks accepted, you’ll need to work a little harder than that. I have reviewed submissions before where the entire talk description was “this is a talk about blockchain”. Literally six words. No event is going to offer you a speaking spot – let alone cover hotel and airfare – based on a six word pitch. (Especially if the only one of those words that’s remotely related to technology is “blockchain”)

A good proposal should probably be around 200-400 words; long enough that you can establish some context, tell us a bit about who you are and what you’re talking about, and give the audience some idea what they can expect to see if they come to your talk.

2. Too long

At the other extreme are the essay submissions. You don’t see too many of these in the wild, because most events use platforms like Sessionize that can impose a character limit on the talk description, but I’ve seen proposals that looked like somebody had pasted an entire blog post into the submission form. Save the detail for the talk itself (or, y’know, publish a blog post?) – because when it comes to responding to the CFP, it is all about striking a balance between brevity and detail.

3. Nothing special

“In this talk I will show you how to get started with machine learning in Python”.

OK, great subject, interesting topic, and very relevant to a lot of folks working in tech.

Ask yourself: what would somebody get from attending this talk that they can’t find on the web, or by watching YouTube videos? Are you a key contributor to an open-source project in that area? Have you or your team used that technology to do something particularly unusual or innovative? Is there something interesting about how you personally got involved with machine learning in Python? If the answer to all these questions is “no”, your submission will probably be rejected in favour of something that offers more in the way of perspective and personality.

There’s another saying in Hollywood that a successful director’s first movie is always a movie that nobody else could have made. Think about your talks the same way; try to pitch something that nobody else could do.

4. Sales pitch

Also known as the “single vendor solution talk”, programme committees can smell these a mile away:

“GDPR is hard! In this talk, Bob will explain how OmniCorp DataMulch 3.0 will solve all your organisation’s data management headaches”

That’s not to say you can’t talk about your own company’s products or platforms, but the rule of thumb here should be that the audience should go away having learned something that’s useful even if they never buy your product. Explain the problem, explain the principles and ideas that are relevant – and by all means throw in a demo that shows how your product applies those principles and ideas.

But if your primary motivation for going to a conference is to sell licences, get a booth on the expo floor like everybody else.

5. Wrong audience

Every conference has an audience. Some are more diverse than others, and they change from year to year, but think about who’s likely to be attending and what they want to hear about. If an event attracts an audience who predominantly work with Java and JVM-based languages, your talk about building APIs with C# probably isn’t going to get accepted. Events with a frontend focus don’t tend to accept talks about relational databases, and vice versa. That doesn’t mean you can’t adapt your material; a topic like “Blazor for JavaScript Developers” will probably get a lot of attention, and if it’s delivered well it could inspire some of those developers to go away and experiment with some new tech. Again, it’s about what perspective and personality you can bring to the table beyond the videos and “getting started with X” guides that are available online.

6. Poor writing

As a native English speaker, I’m acutely aware that many of the brightest people in technology speak English as a second, third, or even fourth language, and so critiquing submissions on the quality of the written English can provoke a certain amount of controversy. Most programme committees I’ve served on have been relatively forgiving of grammar and colloquialisms; English is a horrifically complicated language and the odd grammatical oversight is usually no barrier to comprehension. The red flags here are things like mis-spelling technical terms, or confusing the names of specific technologies. I’ve seen proposals talk about the Kestral [sic] web server that ships with ASP.NET, about Microsoft Sequel Server (yes, that’s how we say it out loud, but nobody ever writes it that way), and one memorable submission about Angular, the popular Java framework.

In a perfect world, you’d follow up on every one of these - “hey, did you mean Angular, the popular JavaScript framework?” – but when a programme committee is trying to pick 75 talks out of 980 submissions, those kinds of errors indicate a lack of care on behalf of the person writing the proposal, and are often enough to justify an immediate rejection.

Finally, read through your submission and check it makes sense. I’ve seen proposals where I’m guessing somebody edited something for length, and removed a key sentence or phrase, leaving an abstract that really didn’t make a huge amount of sense. Careless editing can leave you with a proposal like: “machine learning is revolutionising the world of financial analysis. In this talk, we’ll look at how you can use FoobarJS to streamline your deployments.” OK, fine – but what does FoobarJS and deployments have to do with machine learning or financial analysis?

7. Lazy copy/pasting

Good talks have a shelf life; it’s normal to see the same material being presented at a dozen different events before the speaker finally retires it, and that’s OK. Preparing a talk is a huge investment in terms of time and effort, and there’s no reason you shouldn’t seek to maximise your return on that investment by presenting it to as many different audiences as they can.

When you’re submitting an existing talk to a new event, though, make sure you check the detail carefully. The most glaring error is a submission that still includes the name of another event: “Join Dylan Beattie at DDD East Midlands” doesn’t really work if it’s a submission to NDC Porto. Check dates, check product versions and codenames, check whether anything in the submission needs to be updated – whether it’s just a cosmetic detail, or it’s a whole chunk of your talk you’ll need to go away and revise.

Incidentally, it’s well known and generally accepted that certain parts of our industry move significantly faster than some conference selection processes… if you’ve submitted a talk about WombleJS version 2, and by the time the event actually takes place six months later WombleJS 2 is ancient history and everybody’s on WombleJS version 5, then (a) be prepared to update your talk, references and demos, and (b) let the conference organisers know, ideally with an updated abstract, a few weeks ahead of time so they can make sure it’s reflected in the schedule.

8. Too generic

You want to present “Introduction to Python” in half-an-hour? Good luck. That’s just about long enough to explain that Python exists, it’s a programming language, it comes in version 2 and version 3, and they’re different because… oh my goodness look at that, we’re just about out of time.

That’s not to say you can’t present broad intro-level talks – but your CFP submission needs to make it very clear what your angle is. Show the committee you’ve thought about who’s likely to be in the audience, what level of knowledge and experience you’re expecting them to have prior to your session, and exactly what it is that they can expect to learn by the end of your presentation.

9. Too specific

At the other extreme from the “introduction to Python” are the talks that aren’t likely to attract any audience because they’re only relevant to a handful of developers working in a very specialist area. You may be the world’s foremost expert on aspect-oriented programming in Object COBOL, but face it, if you drew a Venn diagram of people who are interested in AOP and people who are interested in Object COBOL, you’d probably end up with two very small circles, and the only person on the overlap is you.

If you’re submitting to an AOP conference, or an event that has an Object COBOL track, then hey, you might be in with a shot. But for events that target a broader audience, this kind of extremely specialist topic is likely to be rejected in favour of something that’s relevant to more of the audience.

10. No further resources

If your submission ends up shortlisted and the committee is trying to work out which talks to pick, they’ll often look online for anything else which supports your inclusion on the programme. If you’ve spoken at events before, include links to these – especially if it went OK and there’s video of it happening. If you’ve published books, training courses or open source projects related to your submission, say so, and include links where you can.

And if you haven’t, get started! Write some blog posts. Record a couple of short videos of you talking about something you’re passionate and knowledgeable about, stick them up on YouTube, start getting your name and your personal brand out there where people can see it.

The key thing is – whatever you’ve got, make it easy for the programme committee to find it. They’re unlikely to go googling your name when it’s 9pm, there are 143 more talks to review and the only snacks left in the committee room are Norwegian salted liqorice.

So what SHOULD I do?

Come up with an idea. Something nobody else can talk about; you’re unlikely to know more about F# than Don Syme, or more about Entity Framework than Julie Lerman, but I guarantee there will be something in your own experience, a chapter of your own story on which you are the undisputed expert.

Figure out what you want to talk about. Maybe you’re already written the talk; run through it, pick out the high points. Maybe it’s something new, in which case sketch out your narrative arc: where do you start, where do you end, what do you talk about along the way? Condense that down to 500 words. Go for a walk. Come back to it, take out another 100 words. Spell check it. Double-check the product names, acronyms, and technical terms. Ask somebody else to take a look. Read it out loud, check it all makes sense, then hit “submit” and cross your fingers.

Above all, don’t get discouraged. The acceptance rate for most public CFPs is somewhere below 10% – so if you’re getting 1 acceptance for every 10 submissions, you’re beating the odds.

There are literally hundreds of events going on all over the world, all the time. Two that you really don’t want to miss are NDC Oslo (closes Aug 1st) and NDC London (closes September 26), but there are hundreds of events going on all over the world, all the time; online, in-person, covering just about any technology you can think of. Check out sites like papercall.io and the Discover Events feature on Sessionize to see what’s open.

Get out there, good luck with it, and see you on a conference stage somewhere soon!

I run live, interactive workshops for technology professionals who want to get started with public speaking, or to improve their presentation skills. The next one is on 1 & 2 of September 2021; 2pm-6pm BST, online. using Zoom and Slack. Tickets are strictly limited and on sale now from just £165; check out https://ursatile.com/workshops/from-keyboard-to-keynote for more details and booking:

CFP stands for “call for papers”, a term we’ve adopted from academic conferences where presentations are usually based on scientific papers published in academic journals. I’ve also seen it retconned as “call for proposals” or “call for presentations”, but really everybody just calls them CFPs without worrying too much about what it stands for. ↩

Java Criminally Underhyped? Not Back in 1997.

2021-07-01T12:00:00+00:00

Earlier today, a fun little moment of Twitter serendipity alerted me to an article by Jackson Roberts, a computer science student at the University of Colorado, entitled “Java is criminally underhyped”. It’s a really interesting article, and Jackson’s observations correlate with a lot of my own thinking about languages and platforms, although I am squarely in the .NET / CLR camp on that particular front.

But Jackson ends his article:

I am curious why Java lost its hype in the first place. Programmer culture history is poorly documented and if you have insight, please email me or leave a comment.

And, well, the comment I was leaving got a bit long, so I figured I’d blog about it instead. You see, I studied computer science at the University of Southampton, from 1997 to 2000, and when I was an undergraduate, the hype about Java was unbelievable. Java was too important to ignore; it was released in January 1996, and by the time I started university in September 1997 it was a mandatory course for all first year students; it was used throughout my degree course, right up to the final year module on programming language design where our coursework assignment was to build a Scheme interpreter – in Java.

Java circa 1997 was probably the most hyped thing I have ever seen in my career.

For starters, it was the first really mainstream language to introduce a lot of features we now take for granted in most high-level languages. Memory management? Readers, even to a first-year comp sci undergraduate who had spent a grand total of six weeks working with malloc and free, Java’s automatic garbage collection was clearly a revelation. Concurrency? It has threads. They work. Job done. Exceptions? You mean I can just write functions that return what I want, instead of functions that leave the result in a char* pointer somewhere and return an arbitrary non-zero integer if it worked? Java introduced a lot of incredibly powerful ideas and patterns, and it’s hard to underestimate how much impact it would have on the next two decades of programming language design.

But Java wasn’t content with that. Java was going to solve ALL the problems. JavaScript? The language that would end up taking over the world wide web? JavaScript was a glue language. The idea was that all your serious applications would be deployed as Java applets, with JavaScript used for only the most superficial wiring between applets. Java applets were such a big deal in 1997 that some people seriously believed that they represented the end of desktop applications, the end of Microsoft Windows; even the end of the computer as we knew it.

Corel, who at the time were still a credible threat to Microsoft’s emerging monopoly, announced in 1996 that they were porting their WordPerfect office suite and their market-leading CorelDRAW! graphics software to Java. Check out this Infoworld article from 1996 to see just how deep the hype hole really went:

“The success of Java-based network computers, and perhaps Java itself, will likely require the availability of vertical productivity applications for word processing, spreadsheets and databases. At the JavaOne conference, Corel Corp. (www.corel.com) made a first stab at offering such products with the announcement that it plans to deliver a suite of office applications based on its QuattroPro and WordPerfect products. Both products should be available for sale by early next year.”

Java was going to revolutionise desktop development: Java Swing and the Java Native Interface (JNI) meant you could write your app once, and run it on Windows, Linux and Solaris.¹ And thanks to the amazing power of JavaBeans, once you had implemented a particular service or algorithm, you could create a “bean” for it and it could be reused anywhere, by anybody, and building software would be as easy as putting together Lego bricks. JavaBeans were often cited as the last component model we would ever need.

There was an industry spec for a “Network Computer”, a device profile that mandated support for HTTP, Java, JPEG and a handful of other standards… hardware companies would build and sell Network Computers, developers would use Java’s “write once, run anywhere” model to build software for Network Computers, and the entire future would be Java applets running on set-top boxes (or “net-top boxes” as they were often dubbed.) Sun Microsystems actually shipped a thing called the JavaStation, a device based on their SPARCstation platform that ran JavaOS.

The problem is: Java was good, but it wasn’t that good. For Windows desktop apps, Visual Basic and Visual C++ could run rings around it. For games and anything where performance was a consideration, C++ all the way, especially as we entered an age of affordable 3D graphics cards and OpenGL became a big deal. And for server-side web programming, we had Perl, PHP, Microsoft Active Server Pages, and good old cgi-bin, all of which would plug in alongside your hand-coded HTML website and let you gradually add interactivity without having to rebuild your whole site. We also had a handful of coursework assignments that bought into the hype, too. I remember having to build an interactive quiz game using Java Studio, a truly horrific piece of software where you had to wire together Java beans using a buggy IDE that had a frustrating tendency to crash and lose your work with it.

Java evolved. It continues to evolve. But that hysterical hype cycle of the late 1990s, when people seriously believed that the web was going to change the world and everything connected to it would run Java? That didn’t survive the first dotcom crash.

Java eventually succumbed to the inevitable dichotomy of software platforms: sooner or later, every software project has to decide whether it wants to be exciting, or wants to be important. By the mid-2000s, the things in Java that turned out to be important – garbage collection, checked exceptions, threads – weren’t exciting any more², and the things that used to be exciting – Network Computers, applets, office suites written in Java – turned out not to be important. Not even important enough to invest time and money in getting them right. I still dream of a world where Microsoft Office 97 was toppled from its throne by a cross-platform WordPerfect office suite running on Java, but that timeline’s well and truly deviated from ours and it isn’t coming back.

More than that, though, the best way to destroy hype is credible success. Hype is about excitement; it’s about the tantalising possibility that if you jump on board at just the right time, you’ll become part of something unprecedented and maybe end up rich and famous along the way. In the late 90s and early 2000s, a lot of people did exactly that – and, yes, many of them used Java along the way, and a fair few of those got rich and famous by getting in right at the beginning, and getting out before anybody realised their idea was never going to work. But by the mid-2000s, Java was being used in business and industry all over the world; not to build applets or to run CorelDRAW on set-top boxes, but to create stable, reliable, boring software. Java became the language of choice for calculating insurance premiums and booking railway tickets, and friends, nothing is ever going to be exciting to university undergraduates if people in suits are already using it to write pension management software.

The University of Southampton still teaches Java to first-year undergraduates, although I don’t see Java applets anywhere on the syllabus – but not because it’s new and exciting and they’re afraid of getting left behind. Java succeeded. It’s a solid, stable, reliable, cross-platform programming language with excellent tooling and a healthy ecosystem, but it’s also a known quantity. And in the last decade, Java has seen an astonishing resurgence thanks to first-class support on Google’s Android platform.

And, of course, the exquisite irony is that, with over 3 billion devices around the planet running Android, the Java dream of “write once, run anywhere” has damn near come true. It’s just that back in ‘97, we didn’t realise that would be because half the planet would be walking around all day with a JVM in their pocket.

Apple and macOS were having a bit of a crisis at this exact point in history. The blue iMac G3 in August 1998 marked the start of a slow journey back to the mainstream, that would continue with the release of macOS X in 2001 and conclude with the switch to Intel processors in 2006. ↩
well, except to Comp Sci undergraduates who’ve just spent two months learning malloc and free ↩

Introduction to Distributed Systems with .NET: A Live Online Workshop with fwdays

2021-06-21T12:00:00+00:00

On July 6th and 7th, I’m running a live online workshop with the folks from fwdays, all about designing and building distributed systems with .NET. It’s an introductory workshop aimed at developers who have some familiarity with C# and .NET; over two four-hour sessions, we’re going to look at:

Building HTTP APIs using REST and .NET WebAPI
GraphQL, SPAs, and the “backends for frontends” (BFF) pattern
Message queues, RabbitMQ, and the pub/sub pattern
Remote procedure calls with gRPC
Realtime browser communication using SignalR

Check out the video below for more info about what we’ll be learning and how it all works.

It’s a fully online workshop, using a combination of Zoom, Slack, and Visual Studio LiveShare; we’ll discuss the principles and patterns involved in designing distributed systems, and then actually implement them so you can see how they work in practice. Attendees will implement their own services and endpoints, and by the end of the workshop we’ll be running a live system, sending real requests and messages across the internet between us in real time to see how it all fits together.

Here’s what previous attendees had to say about this workshop:

“Extremely professional… I have been a developer for just two years so all of this is pretty new to me even though I’ve touched on some of the topics a little bit in the past. It was a great overview and it made me more confident that I’ll be able to actually use these technologies when the opportunity will rise.”

“I liked the fact that everything was put into action. Seeing the code helps us understand how should work.”

“The live coding is really great, it’s more engaging than just being shown some pre-written code. But generally enjoyed all of it, you explain things really well”

“The delivery is easy to follow, and the examples and code along bits are really helpful”

That’s happening on July 6th and 7th, 2021; 4pm-8pm Ukraine time / EEST (that’s 2pm-6pm if you’re in the UK)

Tickets start at 3900 UAH (about €120), and are on sale now from fwdays:

Buy tickets: https://fwdays.com/en/event/introduction-to-distributed-systems-second

I’ll be lining up some public course dates for the second half of 2021 soon – the prospect of some in-person events taking place before the end of the year; check out https://ursatile.com/workshops/ for details of all the courses I’m teaching, keep an eye on Twitter and LinkedIn for announcements about new course dates (or join my mailing list), and hopefully see you online.

The ONYX BOOX Note Air: Android 10.0 on an E-Ink Tablet

2021-06-11T12:00:00+00:00

I write a lot. Conference talks, blog posts, articles, song lyrics, training material – almost everything I do starts off as words. Maybe the odd heading or bullet list, but really it’s just getting a few thousand words out of my head and into an Evernote document, where I can review, edit, and eventually do something useful with them. For a long while, I’ve been looking for something that does for writing what Kindle does for reading. Something that’s digital, connected, lightweight, syncs to the cloud - but with an e-ink display I can use outdoors, in bright sunlight, and that doesn’t feel like using a computer.

There are some wonderful handwriting devices out there – I tried out a reMarkable tablet a few years ago and was blown away by how good the UX was. But I’m one of those people who vastly prefers typing to handwriting. I own many pencils, and an embarrassing collection of notepads, sketch pads and jotters, but when it comes to getting words out of my head and onto a page, give me a QWERTY keyboard every time. (I recall reading that Neal Stephenson wrote all 2,600 pages of the Baroque Cycle longhand, using a fountain pen, and I get carpal tunnel syndrome just thinking about that.)

I’ve owned every generation of Kindle, and loved them all dearly, but they’re closed devices. They run an “experimental” web browser that’s just about good enough to sign in to airport Wi-Fi if you’re patient, but there’s no filesystem access, no app store, no way to connect an external keyboard or install your favourite writing app. Ditto the reMarkable – although there’s an active hacker community working on extending the capabilities of those devices, they’re still a closed platform and for me that’s a deal-breaker.

Last week, after a particularly bright and sunny day which I spent indoors, with the curtains drawn, because I had work do to and every screen I own is so super-shiny and reflective, I asked on Twitter for any recommendations for an e-ink device that supported an external keyboard, and Scott Hanselman pointed me to a review he’d just done comparing the reMarkable 2, the ONYX BOOX Note Air and the ONYX BOOX Nova 3. The ONYX BOOX Note Air (which I shall refer to hereafter as just the Note Air) looked like exactly what I was after, and, well, it’s been a rough couple of weeks and buying gadgets is one of my coping mechanisms, so I ordered one from Amazon. I’ve been playing with it for a couple of days now and it’s delightful.

Check out the full video review at https://youtu.be/LaL8TQ-PzYs to see all the various features in action:

TL;DR:

It’s an e-ink tablet that runs full Android 10.0, and costs about £420 here in the UK.
It’s not Google “Play Protect” certified, so you need to jump through a few hoops to get the Google Play store on it.
Most apps run; whether they’re usable or not all depends how well they handle the e-ink display.
Bluetooth keyboards work just fine.
Bluetooth audio works beautifully, and it’ll run Spotify.
The pen stylus works brilliantly on the built-in Notes app, but most third-party apps have too much display lag to use with handwriting.
Evernote, Notion, Microsoft Word work (as long as you’re typing rather than handwriting)
Google Keep and Microsoft OneNote are unusable.
Battery lasts about 15 hours - that’s using various apps, with Bluetooth and Wi-Fi enabled the whole time. A full charge takes about 2 hours.

First Impressions

When you first open the box, it’s clear ONYX has paid a lot of attention to first impressions. The packaging is seriously classy, the device itself looks amazing - 6mm thick, fantastic build quality, and the e-ink screen is stunning. It’s crisp, it’s smooth, it’s perfectly even without any rogue bright spots.

This is the point where I should confess that this is the first Android device I have ever owned. I’ve run Windows, Linux, macOS, and multiple iPhones and iPads, but until today I’ve never actually owned a device that ran Android. So on one hand, I have absolutely no idea where anything is or how any of it works. But, on the other hand, the Note Air runs a heavily modified version of Android with a bunch of bespoke apps and stuff, so perhaps that’s not such a bad thing.

It also has a rather nice feature whereby the display has two separate backlights that you can adjust independently; one has a bluish light, one is more red, so you can adjust the colour balance of the display to get a more relaxed reddish light if you’re reading before bedtime, for example. It’s hard to capture on video (one of the tricky things about reviewing and comparing e-ink devices is that looking at picture of an e-ink display online is nothing like looking at the display itself in real life) but here’s a screen recording where I’ve animated the colour balance to show you how it works:

Second Impressions

Once you get over the initial wow factor, there’s a definite moment of “hang on a second… this smells a little odd”. The Note Air includes an “App Store”, but it’s not the official Google Play store, and the first time you fire it up you get a rather unsettling message about how “all of the proposed apps in the App Store are collected from the Internet, for study reference only” - and when the first app on that list is Amazon Kindle for Andorid9\10 [sic] the alarm bells start ringing just a little – mainly since most of the stuff I actually want to do with this tablet requires signing in to assorted cloud providers, and, well, I’m not 100% OK with typing my Google or Amazon credentials into a device that offers the Kindle for Andorid9\10 app “for study reference only”.

I did a little digging and found some instructions on how to enable the Google Play store. First, you need to enable the Google Play framework - there’s an option for this in the device settings. I hit the button, a dialog popped up saying “Checking info…”, and then nothing happened for fifteen minutes - but when I got bored of waiting and reset the device, when it booted back up, the Google Play store had appeared in the Apps menu. I opened it up and got another warning - “This device isn’t Play Protect certified”. Now, I’m guessing that’s some kind of Google certification program that ONYX hasn’t enrolled in; the workaround is to sign in to your Google account, and manually register the device ID as a trusted device.

The Note Air actually includes an interface for doing this - but to use it, you need to enter your Google credentials, and the form provided to do that is part of the Air Note UI – which means no address bar, no information about where they’re going, and no way to view or verify any certificate information. I wasn’t entirely happy with that, so rather than registering from the device itself, I opened up https://www.google.com/android/uncertified/ on my desktop, signed in to my Google account, typed in the GSF ID of the device, waited a few minutes for the cloud to catch up, and I was in business.

Using the Note Air as an e-reader

Once I’d installed the Kindle app from the Play Store and signed in to my Amazon account, it worked flawlessly. There’s not a huge amount to say beyond that - it just works.

Handwriting support and the Notes app

The Note Air ships with a handwriting stylus. It’s an EMR stylus (aka a “Wacom stylus”); it’s powered by an electromagnetic field generated by the tablet itself, so it doesn’t require any batteries, and it works incredibly well. Used with the built-in Notes app, it provides a really immediate, tactile experience, that feels closer to digital paper than anything I’ve ever tried.

The acid test of handwriting devices for me is whether I can use them to do a cryptic crossword, and this is the first device I’ve used that just works. It takes a bit of tweaking - to get to this point, I had to download the PDF of the puzzle from the Guardian website, open it up in Acrobat, grab a screenshot, and then insert that screenshot as an image into the Notes app – but once that’s done, it just feels like solving the puzzle on digital paper. The e-ink means it works outside in bright sunlight, the backlight means you can use it in the dark as well. The images and screenshots here really don’t do justice to how responsive it feels actually using the device.

One thing that I noticed is that the Note Air uses some sort of two-pass rendering. When you’re using the built-in Notes app with the stylus, lines and pen strokes appear immediately, but if you’re screen-casting or capturing a screen recording, the same pen strokes don’t appear in the recording for a few seconds. My guess is that there’s some kind of immediate feedback system built into the device itself that’s running on top of the Android display subsystem, so that when you move the pen, your line shows up on the screen immediately - before the operating system even knows you’ve drawn it.

The downside of this is that it means third party apps like OneNote that aren’t specifically optimised for the device just don’t work with the stylus for any kind of handwriting or sketching.

Writing with Evernote and a Bluetooth keyboard

The main thing I want to do with the Note Air is writing; I’ve previously used my iPad with a Logitech K380 Bluetooth keyboard and Evernote to write all kinds of things, so I wanted to do a side-by-side comparison of the two. Here they are, literally side by side, outside in my garden in bright sunshine:

The iPad is running at full brightness there - and it’s completely unusable; it’s just too reflective. The Note Air, on the other hand, is just like paper: bright sunlight makes it easier to read.

Like most apps, the Evernote UI is a little hit and miss – app menus with lots of nice photography and animation are never going to work well on this kind of display. But once you open the document you’re working on and make it full-screen, it works just fine.

Other Note-Taking Apps

I tried out a couple of other third party note taking apps to see how well they work with the e-ink display:

Notion: Works nicely with an external keyboard hooked up; I could easily see myself using it as an alternative to Evernote if I had to.

Microsoft Word: works pretty well. I downloaded the Office 365 edition from the Play store, and found it very usable apart from some UI glitches, like the “new document” icon being a white page, with a pale grey hairline outline, on a white background… on the e-ink display it just looks like blank space. But once you know where it is, it’s fine :)

Google Keep notes: Nope. It works, but the way it renders text onto the e-ink display is so poor as to be illegible.

Microsoft OneNote: OneNote was by far the worst of the apps I tried – it’s just completely incompatible with the e-ink display. The UI is illegible (lots of white-buttons-on-white-backgrounds), and as discussed above, the stylus doesn’t really work with third-party apps so you can’t use any of the handwriting features. (Well, you can, but you can’t see what you’re writing until you finish writing it, which means it’s effectively unusable.)

Termux, vim, ssh, nodeJS and git, oh my!

The Note Air supports Termux, a teminal emulator and Linux environment for Android – which means you can open a terminal, install nodeJS, Python, Ruby, git, vim; pretty much any text mode applications that will run on Linux. And it supports SSH, so you can spin up a Linux VM in the cloud, take your Note Air to the park or the beach, and ssh into the cloud from there – and yes, because it’s e-ink, bright sunlight isn’t a problem.

(Did I mention it works in direct sunlight? It works in direct sunlight!) 🌞

Music, video, and screen casting to ChromeCast

The Note Air has a built-in speaker and microphone. They’re nothing particularly special – the speaker is perhaps a little louder and clearer than the speaker in most smartphones, but we’re not talking high fidelity audio here. It supports Bluetooth audio, though, so you can pair an external speaker or Bluetooth headset; I’ve installed Spotify on mine; the UI looks a little odd but it’s usable and plays music just fine.

As for video: you’re probably wondering why anybody would watch video on a device that can just about manage 8 shades of grey at 4 frames per second… but because the Note Air runs Android, it can cast your display to a Google ChromeCast, which will give you full HD HDMI, in colour, on an external monitor or TV. I wouldn’t use it for anything particularly interactive – there’s a tiny bit of latency on the video signal – but for streaming Netflix or Amazon Prime to the TV in your hotel room, it works just great.

Remote Desktop

There’s an official Microsoft Remote Desktop client for Android, so I installed that just to see what it could do – and again, I was surprised at how usable it is. I’m writing this right now in Typora, via an RDP connection to my main Windows 10 machine, and it’s working remarkably well. Enabling Windows’ “high contrast” display mode makes a huge difference here, especially if you have dark mode enabled; those lovely muted navy-blue-on-midnight-blue colour schemes that look so good on a 32” TFT aren’t gonna work here, but nice bold black on white works just fine.

Miscellany

Yes, it runs Doom. Is it playable? Not remotely. It’s hard to dell the difference between a wall and a monster when you only have 8 shades of grey at 4fps. But it runs.
A few folks on Twitter asked about using it for sheet music: it’ll render PDFs just fine, and I tried it out with a PEDPro AirTurn Bluetooth footpedal and it worked perfectly.
Charging is via a single USB-C port, which you can also use to connect peripherals. I hooked up a powered USB hub and was able to run a wired keyboard and external HD quite happily. Note that the USB-C port appears to support either charging or data, but not both - so if you’re taking an external HD on the road, you’re going to be running the drive off the Note Air’ internal battery, which probably isn’t a good idea.
WiFi and Bluetooth both seem pretty solid; there’s no SIM card support so for mobile internet you’ll need to tether to your phone or rely on public Wifi.
The screen scratches easily, so be really careful about protecting it with something when you’re carrying the device around.
The stylus attaches to the edge of the device with magnets, but it’s not particularly secure and if you lose it you’re looking at £50 for a new one. The stylus has a replaceable nib (£18 for 5 from Amazon) since they apparently wear down in normal use. When I’m holding a pen in my hand I tend to, y’know, scratch my nose with it, twiddle it between my fingers, that kind of thing, and I actually managed to accidentally remove the nib from mine a few times and drop it; the nibs are extremely tiny and the stylus doesn’t work without them, so might be worth carrying a few spares just in case.

Conclusions

First off, it’s a huge amount of fun. I’ve had a blast setting this thing up, seeing what I can get it to do. For specific applications – handwriting notes, crosswords, Evernote, reading books via the Kindle app – it works very, very well, but the vast majority of apps in the Android ecosystem aren’t even tested on e-ink displays, let alone optimised for them, and it shows. For travelling, the Note Air probably replaces the Kindle + iPad combo that I used to carry around with me everywhere, but I’ll still be taking a laptop and a smartphone along for the ride.

This is still very much an expensive gadget for people who like expensive gadgets and aren’t afraid to tinker with them. Some things work even though they probably shouldn’t, like Windows Remote Desktop; other things, like OneNote, should in theory work really nicely with this kind of device but in practice are completely unusable. There is also clearly a whole world of applications out there – music composition springs to mind — that could work beautifully on this kind of device if they were designed for it.

It’s an interesting device, though. Being just about old enough to remember the days of connecting home computers up to a portable black & white TV, there’s a certain amount of nostalgic novelty to it, and if you think back to the 4-colour LCD displays in the earliest laptops, this is infinitely better than those ever were:

Very cool!

And, you know… That screen refresh is possibly faster than my first Mitac 386 laptop had on its greyscale lcd 😂
— Stacy Cashmore (@Stacy_Cash) June 9, 2021

The key question I always ask about gadgets is: if I lost this one, would I buy another one? I’m not sure, but only because I think within a year we’ll probably see something twice as good for half the price. I’m looking forward to seeing what happens with e-ink over the next few years. ONYX are already shipping a colour e-ink device, the BOOX Nova3 Color, and I’m sure it’s only a matter of time before the industry fixes the problems with ghosting and refresh rate and we’re looking at e-ink displays that can handle 8-bit colour at 24 fps.

But, hey – summer is here, the sun is shining, the prospect of being able to sit outside in the sunshine and get a little writing done at the same time is too good to pass up, and this is the best device I’ve seen yet for doing that.

Guru Meditation and Bad Error Messages

2021-06-08T12:00:00+00:00

Today’s Fastly outage is a nice moment to reflect on the fact that developers are absolutely terrible at writing error messages. Earlier today, some of the biggest sites on the web were offline for nearly an hour, including the BBC, the New York Times, Reddit, Stack Overflow, GitHub, and many UK government sites. During the outage, visitors to those sites got a cryptic error message:

Error 503 Service Unavailable.

Service Unavailable

Guru Mediation:

Details: cache-lcy19240-LCY (and then a long, meaningless string of numbers)

Varnish cache server

Now, a quick recap about how to write good error messages. It comes down to remembering four things.

Be helpful. Explain what has gone wrong - and whether there’s anything the user might be able to do to work around or fix the problem. “The website you’re trying to reach isn’t available right now. Please try again in a few minutes.”
Be human. Don’t use meaningless jargon. “The network is unavailable” is always better than “Unhandled connection timeout 0x80004005 (WCOM SystemTimeoutException)”. The people who need error numbers and stack traces will know where to look for it.
Be honest. If you screwed up, you screwed up. It happens. Never make your error messages so ambiguous that the user might think they’ve done something wrong. Life is scary enough as it is.
Be humble. Apologise. Your software just broke and now somebody else is having a bad day and that’s your fault. “We’re really sorry. One of our systems has failed and so that website isn’t available right now. Please try again in a few minutes.”

(The jury is still out on whether the fifth H should be “humorous”… humour is difficult and extremely subjective. If you know your audience, and you know that whatever went wrong is probably just inconvenient rather than disastrous, then sure, be funny. But be careful with it.)

Now, let’s talk about “503 Service Unavailable Guru Mediation.” That’s a joke. No, really. To about a hundred people on the planet, that is incredibly funny, because “Guru Meditation” is what their old Amiga 500 used to say when something went badly wrong.

Now today’s outage was definitely caused by Fastly, but the actual error message is coming from Varnish, an open-source software application that’s many organisations, including Fastly, to improve website performance. Varnish is really, really good, but… let’s just say that “user friendly” wasn’t on their list of requirements. Varnish is a high-performance HTTP caching proxy and URL rewriting engine. You configure it by writing rules in a domain-specific language, VCL, which is based on C (yes, that C). Those rules are compiled into a shared binary that runs as part of the Varnish network pipeline. It’s an app built by nerds, for nerds, to do one specific thing, and do it as fast as possible - and it does that one thing very well indeed. But at some point, one of the Varnish developers had to make a decision: “hey, what should we return if something goes wrong?” – and they decided that the best thing to do would be to return an HTTP 503 error, and include a joke about Amigas (although keen eyes will notice that the original “Guru Meditation” has actually changed to “Guru Mediation” in the Varnish error message. Cute.)

UPDATE: It looks like the change from “meditation” to “mediation” isn’t part of Varnish, it’s part of Fastly. Somebody actually took the time to dig into the Varnish configuration they’re running, find the Amiga joke, and change it to a slightly modified version of the Amiga joke. Some folks on Twitter hypothesised that this is so Fastly can tell the difference between a problem with their own Varnish stack and a problem with one of their customers’ Varnish stacks, which kinda makes sense - but, hey, while they were in there, they could have, y’know, replaced it with something… helpful?

This morning, I reckon about 100 million people probably saw that error message. I reckon less than 1% of them understood it, and of the 1% who understood it, probably 1% of those even realised it was a joke, let alone found it funny. The other 99 million people who saw it? Most of them probably freaked out a bit, because it looked like half of the internet had suddenly stopped working.

If the whole internet stops working, you assume it’s your problem. Check the wifi. Reboot the router. Call your ISP. If one website goes down? Big deal. It’ll be back. But when the BBC, the New York Times, the Guardian, Reddit, GitHub, Stack Overflow and the UK government all suddenly start returning “503 Service Unavailable Guru Mediation”, people get a bit rattled. So they check Google and fast.com and Netflix, and those sites are all ticking along just fine - and that’s when people start really freaking out. Virus? Malware? Terrorist attack?

No. Somebody probably just messed up the Varnish config… and, probably, had no idea that an innocuous typo in a block of not-quite-C code was going to result in hundreds of millions of people going “argh… what the heck is a guru mediation?”

If you want to capture stack traces and correlation IDs and other diagnostic details when something goes wrong? Awesome. Log them. Don’t have logs? Get logs. Don’t have time to get logs? Go and set up some logging. Go on. Do it now. We can wait. But when it comes to the error message that ends up on your screen when something goes wrong… somebody else is going to see that. I guarantee it. And you have no idea who that will be, or what they’ll be doing when they see it.

So remember: Helpful. Human. Honest. Humble. Doesn’t matter if you think nobody except your colleagues will ever see it. Doesn’t matter if it’s an error that you’re absolutely certain is never going to happen. If you’re wrong, your carefully-crafted error message could make a huge difference to somebody else’s day.

And if you’re right? No big deal. You wrote a good error message anyway.

Online Workshops in 2021

2021-02-01T12:00:00+00:00

I don’t do New Year‘s Resolutions at New Year. January 1st is a terrible day to start anything important. I generally kick off new things at the start of February, and perhaps line up another project or resolution or two to kick in at the end of March when the clocks change. This time last year, I started my own company, Ursatile. I had a business plan and everything – but, as Mike Tyson memorably put it, “everybody has a plan until they get punched in the mouth”…

Yeah. 2020 was definitely a metaphorical punch in the mouth. Compared to a lot of people, I had it pretty easy: I already had a proper office set up at home, I didn’t have to worry about losing my job, I don’t have any kids to home-school. But I had to figure out, fast, how to make my new company viable in the new world of online events and remote working. For a while, I was running three online events a week, just to give us all a chance to try things out, figure out how online events worked, and what we could expect from them, and on the back of that, NDC hired me for a few months to help translate their developer conferences into online events. At the same time, I was developing my own workshops and online courses, giving talks at dozens of online events, and spending a small fortune on cameras, microphones and lights - some of which became instantly indispensable; some of which went into the cupboard within 24 hours and never came back out. By the end of last year, I’d racked up something like 50 days of online teaching, and lost count of how many online conferences and meetups I’ve spoken at. I’ve done online concerts, online quizzes, online panel shows – I even got a couple of bookings to do the entertainment at online Christmas parties, which turned out to be a lot of fun. And I rounded out the year with a live streamed online party show – two hours of music, comedy, greenscreen effects, special guests and festive entertainment, which was also a lot of fun.

But for me, the biggest revelation of 2020 was how well programming workshops work online. Hands-on, deep-dive technical training, with all the attendees writing and running code throughout the workshop. I’ve run my Introduction to Distributed Systems workshop online several times now, and it’s been absolutely excellent. Instead of being in a conference room, hunched over laptops and struggling with bad hotel wi-fi, everybody’s at home, with their dual monitors and their favourite mechanical keyboard; everybody’s running code on their own workstation, we’re connecting to cloud services and passing messages and requests between us. A day is long enough to build a working prototype; two days is long enough to turn the prototype into something you could actually use - monitoring, security, logging, integration tests. We build the whole thing, line by line, and as we go, I explain how it all works and why it’s built that way: what’s a subscriber ID, and why does it matter? What do those magic numbers in the proto file mean? What happens if one of those services fails?

It’s a great way to teach programming, and it’s a format that works really well for online workshops – and so, for 2021, I’ve put together several new workshops that are specifically designed to teach online. Lots of coding, lots of collaboration; things it would be impractical – or impossible – to do in a classroom.

I’ll be posting details of all these new workshops over the next few days, so keep an eye out for that.

I also created a workshop last year about how to run online workshops, which proved really popular and had great feedback from everybody who attended it, so I’m going to be running that regularly throughout 2021 as well. It’s a half-day, hands-on workshop, four hours or so, covering everything from how to set up your mic and webcam to using online tools like Miro and Jamboard to run remote training courses, collaboration and design sessions. Dates are on sale now for February and March; find out more at https://urs.tl/htrow.

Adding Events to a Google Calendar via a Link

2021-01-12T12:00:00+00:00

It’s the 12th January 2021. Or the 43rd of December, 2020… or maybe the 317th of March, 2020? With so much of our lives being forced online, people all over the world working from home, it’s becoming harder than ever to keep track of what day it is and what’s happening.

I’m absolutely dependent on Google Calendar to keep track of what I’m doing – events, appointments, social calls, birthdays, even which day I need to put the bins out. I know I’m not the only person who does this, and so for a long time I’ve wanted a way to publish a link that will allow other people to trivially add an event to their own Google Calendar. This isn’t the same as actually sending an invitation – this is for things like meetups, live streamed performances; the kind of events where you’re not hugely concerned about whether a specific person can make it or not, but for folks who are interested, you want to help them remember that it’s happening.

Well, I found out recently that, through an undocumented feature of the Google Calendar platform, you can actually create a link that will do exactly this. And since it’s undocumented, I had to do a little digging to figure out the exact format of the URL you need to use.

First, the root URL: https://calendar.google.com/calendar/u/0/r/eventedit. If you open that link without any query string parameters, it opens a blank “Add event” screen on Google Calendar.

Now for the query string parameters. These are the ones I’ve figured out so far; I suspect things like notifications and guests can probably be prefilled via the query string as well.

name	description	Example / Comments
`text`	The one-line summary of the event.	`Alice and Bob's Weekly Cocktail Night`
`dates`	The event start and finish times, in RFC 5545 date format, separated by a `/`	`20210108T170000Z/20210108T190000Z`
`details`	The longer event description. This field supports a subset of HTML, so you can include formatting, bullet points and links by URL-encoding your HTML code.	(see below)
location	The intended venue for the activity. This field correlates to the `Location` property defined in RFC5545 section 3.8.1.7.	Google Calendar integrates this field with Google Maps, so entering something like “Zoom” will give you a handy link to the ZOOM+Care on Martin Luther King Blvd on Portland. Probably best to enter the URL again if it’s for an online event.
`recur`	The recurrence pattern for recurring events, supplied as an `RRULE` pattern conforming to RFC5545 section 3.8.5.3	`RRULE:FREQ=WEEKLY;UNTIL=20210331T000000Z`

The details field can include a URL-encoded HTML fragment, like this one:

%3Ch2%3EWe%27re+having+a+Zoom+party %3C/h2%3E%3Cul%3E%3Cli%3EVirtual%20backgrounds!%3C/li%3E %3Cli%3ESilly%20stories!%3C/li%3E%3Cli%3ESnacks!%3C/li%3E %3C/ul%3E%3Cp%3EYou%20can%20join%20it%20 %3Ca%20href=%22https://example.com/some/link%22%3Ehere%3C/a%3E%3C/p%3E

Putting it all together, you get a link that looks like this:

https://calendar.google.com/calendar/u/0/r/eventedit?text=Alice+and+Bob%27s+Weekly+Cocktail+Night&dates=20210301T190000Z/20210301T210000Z&details=%3Ch2%3EWe%27re+having+a+Zoom+party%20%3C/h2%3E%3Cul%3E%3Cli%3EVirtual%20backgrounds!%3C/li%3E%20%3Cli%3ESilly%20stories!%3C/li%3E%3Cli%3ESnacks!%3C/li%3E%20%3C/ul%3E%3Cp%3EYou%20can%20join%20it%20%20%3Ca%20href=%22https://example.com/some/link%22%3Ehere%3C/a%3E%3C/p%3E&location=https://example.com/some/link&recur=RRULE:FREQ=WEEKLY;UNTIL=20210601T000000Z

which looks unwieldly, but, hey, stick it behind some nice friendly text, and you’ve got your add to Google Calendar link ready to include in your event page.

Nice.

Petrucci: A Rockstar interpreter in TypeScript: Part 1 & 2

2021-01-06T12:00:00+00:00

Happy New Year, readers. 2020 is done, 2021 is here – and it’s time to do a new thing. I had a blast streaming Advent of Code on Twitch during December, and so for the next month or so I’m going to be building a Rockstar interpreter, using TypeScript, live on Twitch, and then writing about it here. If you want to join and watch live, I’m on twitch.tv/dylanbeattie most days around 16:00 UTC – and if you want to catch up afterwards, there’s a YouTube playlist of the videos.

Why TypeScript? Partly because I want to learn it. I’ve not worked with TypeScript before, at all, but I hear lots of very good things about it, and given it was designed by Anders Hejlsberg, who created C#, I suspect I’ll quite like working with it. I also want something that transpiles to JavaScript, so I can ship code into the wild and stick it up on a website where anybody can run it in their browser or on their phone. And partly because I really enjoy working with JavaScript; it’s a fun language that can do a lot of very cool things but which also affords a lot of experimentation and exploration, and I’m really interested in the TypeScript philosophy of allowing you to enhance JS with types and compile-time type checks if you want to, and allowing you to run plain old vanilla JS when that’s what you need.

So… part 1 (YouTube) was literally getting up and running. Installing TypeScript, having a look at Deno, figuring out how to get a PEG.js parser working nicely with TypeScript classes. Part 2 (YouTube) took this idea a bit further, and got as far as a very simple grammar for a tiny language, a parser and an interpreter for that language.

What makes things a little more interesting is that PEGjs, the parser generator that I’m planning to use on this project, only understands plain JavaScript - no TypeScript. So the high-level architecture looks something like this.

syntax.ts is the abstract syntax tree node classes - AdditionNode, MultiplicationNode, NumberNode, and so on. This is referenced directly by interpreter.ts (which is fine, because they’re both TypeScript), but I also need to include a reference to these classes in the parser that’s generated by PEGjs, and because PEGjs can’t understand TypeScript, I need to use the TypeScript > JavaScript compiler, tsc, to transpile syntax.ts into syntax.js. PEGjs uses the require() syntax, so there’s a line in the header block of the grammar file to require('syntax.js'). The pegjs tool translates the PEG grammar definition, parser.peg, into a parser.js file containing the actual parser. Finally, interpreter.ts imports the parser and the syntax definitions.

Because each different kind of language feature – addition, multiplication, output, etc – is now implemented as TypeScript class, it’s possible to implement an evaluate() method on each of these node types, which makes the design of the interpreter itself beautifully simple.

The interpreter presented here supports the following language constructs:

Arithmetic: integer literals, addition, multiplication, and parentheses.
Output: the say keyword will print the subsequent expression to STDOUT.
Flow control: the return keyword will halt the program with the specified return code.

Here’s a sample program:

output 1
output (4+5)*(6+7)
return 0
output 9

The parse tree for this program should be something like:

output:
  number: 1
output
  product:
    lhs: sum:
      lhs: number: 4
      rhs: number: 5
    lhs: sum:
      lhs: number: 6
      rhs: number: 7
return:
  number: 0
output
  number: 9

The output from this program should be:

1
117

and it should exit cleanly with the operating system return code (aka error level) 0. The final output 9 should NOT be executed.

Here’s the full code so far - this is also on Github

syntax.ts

This file contains the TypeScript classes for building the nodes of our abstract syntax tree.

// syntax.ts: abstract syntax tree nodes
export abstract class LanguageNode {
    abstract evaluate(): Result;
}

export enum Action { None, Return }

export class Result {
    value: any;
    action: Action;
    constructor(value: any, action = Action.None) {
        this.value = value;
        this.action = action;
    }
}

export abstract class BinaryNode implements LanguageNode {
    abstract evaluate(): Result;
    lhs: LanguageNode;
    rhs: LanguageNode;
    constructor(lhs: LanguageNode, rhs: LanguageNode) {
        this.lhs = lhs;
        this.rhs = rhs;
    }
}

export class AdditionNode extends BinaryNode {
    evaluate() { 
        return new Result(this.lhs.evaluate().value + this.rhs.evaluate().value);  
    }
}

export class MultiplicationNode extends BinaryNode {
    evaluate() { 
        return new Result(this.lhs.evaluate().value * this.rhs.evaluate().value); 
    }
}

export class NumberNode extends LanguageNode {
    value: Number;
    constructor(value: Number) {
        super();
        this.value = value;
    }
    evaluate() { return new Result(this.value); }
}

export class OutputNode extends LanguageNode {
    node: LanguageNode;
    constructor(node: LanguageNode) {
        super();
        this.node = node;
    }
    evaluate() {
        let value = this.node.evaluate().value;
        console.log(value);
        return new Result(value);
    }
}

export class ReturnNode extends LanguageNode {
    node: LanguageNode;
    constructor(node: LanguageNode) {
        super();
        this.node = node;
    }
    evaluate() {   
        let value = this.node.evaluate().value;
        return new Result(value, Action.Return);
    }
}

parser.peg

The grammar definition for the language supporting in this prototype:

{ const Syntax = require('./syntax.js'); }

program = head:statement EOL tail:program { return [ head ].concat(tail) }
        / item:statement { return [ item ] }

statement = output / return

output = 'output' _ node:expression { return new Syntax.OutputNode(node) }
return = 'return' _ node:expression { return new Syntax.ReturnNode(node) }

expression = sum

sum = lhs:product _ "+" _ rhs:sum { return new Syntax.AdditionNode(lhs, rhs) }
    / product

product = lhs:primary _ "*" _ rhs:product { return new Syntax.MultiplicationNode(lhs, rhs) }
        / primary

primary = number
        / "(" expr:expression ")" { return expr; }

number = digits:$[0-9]+ { return new  Syntax.NumberNode(parseInt(digits)) }

_ = [ \t]*
EOL = '\r'? '\n'

interpreter.ts

Finally, the interpreter, with the program itself inline as a JavaScript multiline string literal:

import * as Parser from "./parser.js";
import * as Syntax from './syntax';

let program = `output 1
output (4+5)*(6+7)
return 2
output 9`

let ast = Parser.parse(program);
ast.forEach(node => {
    var result = node.evaluate();
    switch(result.action) { 
        case Syntax.Action.Return: process.exit(result.value);
    }
});

To run this via the command line, you’ll need to:

Install NodeJS. I’m running v12.16.2 but this should all work with any version > 12.
Install TypeScript: npm install -g typescript
Install PEGjs: npm install pegjs

To build and run the example, type node run start - or to run each step individually:

$ tsc syntax.ts
$ pegjs parser.peg
$ tsc interpreter.ts
$ node interpreter.js

In the next instalment, I’ll be setting up some tooling to make things a little easier to work with, and figuring how to start incorporating some of the existing test fixtures and coverage from Rockstar into the project.

Joe Satriani and John Petrucci on stage at the Portsmouth Guildhall. Photo © rmk2112 via Flickr

Oh, and the name I’ve chosen for this one is Petrucci… the current JS interpreter for Rockstar is Satriani, after Joe Satriani (JS), and, well, John Petrucci is definitely younger than Satch, he’s more technically ambitious, and folks who are so inclined could get into a long argument about whether he’s better or not. But the world is definitely a better place for having both of them in it – and as it is with virtuoso metal guitar players, so it is with Rockstar interpreters. 😎

Dylan's Advent of Cool Nerd Things Day 25: Ninite

2020-12-25T12:00:00+00:00

Happy Christmas, friends! Have we saved the best for last? Well, I’ll let you decide… we’ve had 24 days of fantastic tools, utilities, websites and apps.

In the first week of #nerdvent, we resized windows with Sizer, visualised hard drive space with WinDirStat, turned our iPhones into webcams using Camo, powered up Windows with the X-Box Game Bar, kept on top of timezones with WClock, managed all our social media apps with Franz, and tunnelled the internet to our local machines with ngrok. Week 2 brought us Instant Eyedropper, PowerToys and the Fira Code programming font; we edited audio files with Audacity, kept an eye on our systems with Open Hardware Monitor, jumped cacti and dodged pterodactyls with the Chrome Dino Game, and wrote beautiful Markdown documents with Typora. Week 3 we got colourful with Paletton, compressed files with 7-Zip, made beautiful images of source code with Carbon, got cracking with GitKraken, got creative with Canva and Unsplash, broadcast ourselves with OBS and got our volume controls under control with EarTrumpet. Finally, we met the incomparable Beyond Compare, gave Windows a spring clean with Autoruns, and tested our emails with Mailtrap. And yes, I did actually publish a blog post every day for 25 days. I’m as surprised as you are. Especially since I’ve also managed to do Advent of Code every single day as well, and stream it live on Twitch and YouTube.

Anyway. It’s Christmas! Let’s meet the last awesome tool I’m going to write about for #nerdvent: Ninite. Ninite is amazing. Open the Ninite website, tick the apps you want, and it’ll give you a single, bundled installer that will install all of them on your Windows PC, with sensible defaults, no spyware or bundled crap. And, er, that’s it. There’s not a huge amount to say about it other than that it’s very, very good indeed.

When I’m setting up a new PC or reinstalling Windows, I use the default browser to get my Ninite installer, and Ninite does the rest. Web browsers, image editors, utilities, developer tools - in fact, quite a few of the tools I’ve written about over the last few months are available via Ninite, so why not check it out by using it to check them out?

Dylan's Advent of Cool Nerd Things Day 24: Mailtrap

2020-12-24T12:00:00+00:00

When you develop websites, or just about any kind of internet application, it’s only a matter of time before you need to send email. I don’t mean sending an email manually - I mean writing a feature into your software that sends templated, personalised emails you’ve never seen, to people you’ve never met, who have put their email address into one of your web pages. Order acknowledgements, receipts, password reminders, registration confirmations… nerds like me build applications that use email for all kinds of useful things. On one hand, email is amazing – a secure, reliable way to send messages to anybody, anywhere on the planet, regardless of what device, platform or internet provider they’re using.

On the other hand… email is complicated. The SMTP standard that governs most internet email was ratified in 1982, and by the time it began to creak in the mid 1990s, there were already so many people using it that replacing it with something better was, and remains, basically impossible. We’ve added all kinds of security features, encryption and encoding improvements, but they’re all fundamentally built on the same set of protocols we’ve been using since Return of the Jedi first came out.

Integrating email into your applications is a headache. People expect slick HTML emails now, with formatting, images, and links – but, believe it or not, there is no standard for adding rich formatting to emails. Most email systems rely on a labyrinth of nested tables and old-school inline styles straight out of 2006, and that’s before you even start thinking about alternate body encodings, internationalization, security, validation… yep, emails are hard to design, hard to build, and hard to test.

When it comes to designing them, find a good designer who’s worked with email as a medium. Or find a design you like, and copy it. When it comes to building them, the best tool I’ve ever found for layouts is the fantastic Foundation for Emails 2 from Zurb; the MailChimp email designer is also pretty good, although it’s not quite so easy to integrate into your own apps and websites.

But when it comes to testing emails, my go-to is a web app called Mailtrap; I’ve used it for years, and it works beautifully. Mailtrap gives you a set of SMTP server credentials, along with sample code for connecting and sending email in just about every web development platform out there:

Plug the appropriate config into your test or staging application, send your email, then open up Mailtrap’s web interface and see what you’ve got.

You can preview your email as it will appear on desktop, tablet and mobile devices - it’s not 100% perfect, but it’s easily good enough to see any glaring omissions or problems with the layout. It also gives you a detailed technical breakdown of all sorts of useful metrics and stats about your emails

The free plan gives you 500 test emails a month and a single test inbox - which is more than enough for hobby projects and trying it out. The various paid plans unlock higher limits and features like auto-forwarding, so you can bounce your test emails through Mailtrap and forward them to your real mailbox (or your boss, or your client), and the Team Plan is available for free to non-profits and open source projects.

Check it out at https://mailtrap.io/ - it’s great. In fact, I have only one criticism of Mailtrap, and that’s their current branding. It’s OK, I guess, nothing special… but until a few years ago, their homepage had this absolutely fantastic illustration on it; a grinning octopus riding in a bin, chasing down stray emails with a fly-swatter and a rolled-up newspaper. I thought it was wonderful - memorable and distinctive and so much more engaging than just another bland corporate homepage… but hey, I guess they decided that “mad octopus riding in a bin” wasn’t really on brand for them. Thankfully, the Wayback Machine never forgets – and hey, Mailtrap is still an excellent platform, and highly recommended.

Dylan's Advent of Cool Nerd Things Day 23: Autoruns

2020-12-23T12:00:00+00:00

I’ve been using Microsoft Windows since version 2.1, and one of the things that has held true of almost every version of Windows I’ve ever run is that they get slower. A fresh installation of Windows is a joy - snappy, responsive, all those lovely acres of empty hard drive space… and then after a year or so, it’s just not so snappy any more. I’m sure it’s partly psychological, and I’m sure some of it is also related to security patches and essential updates (hey, it’s easy to go fast when you don’t care about data security) – but a large part of it is to do with clutter. Clutter is the stuff you don’t know you’ve got. The operating system equivalent of the drawer in your kitchen that was empty when you moved in, and now somehow it’s full even though you never really decided what to put in it, or that pile of dead biros and scratched CDs that lives in the glovebox of your car.

In most modern operating systems, including Windows, desktop search has got so good that cleaning up is easy. Just shove everything in a folder called Clutter (yes, really), and get on with your life. It’s out of sight, out of mind, and if you need it again, just search for it. But there’s some stuff that isn’t quite so easy to clean up.

Often, when you install an application or utility, it’ll configure something to run automatically when Windows starts. Sometimes, this is a good thing - I have apps like Discord and Dropbox that run automatically when I start Windows, because it makes sense for them to be running all the time. They sit in the background and do useful work. Sometimes, it’s not. You’d be astonished how many applications and devices install a program whose job is just to run in the background all the time, that serve no purpose at all other than occasionally pinging the company’s website to see if there’s a new version available. Then there are the drivers for devices you don’t use any more; the utilities that insist they want to run at startup even though you’re quite happy to start them yourself when you want to use them, and the various extensions that add features to things like the Windows Explorer right-click menu.

Well, it’s time to get Marie Kondo on your Windows install, and to do that, we’re going to use a tool called Autoruns. Autoruns is part of the SysInternals suite, an astonishingly powerful set of free tools for managing Windows computers. SysInternals was founded by Mark Russinovich, who also wrote most of the tools in the suite. Mark is now the CTO of Microsoft Azure, and the SysInternals suite has been maintained and distributed by Microsoft since they acquired the company in 2006.

SysInternals is absolutely packed with goodies, but let’s take a look at Autoruns first. When you fire it up, you’ll get a surprisingly long list of things:

Windows has a lot of different mechanisms for running things automatically, and each “group” here reflects one of those mechanisms - registry keys, system folders, scheduled tasks, and so on. I fire up Autoruns every month or so to see what new delights have been added to my system by over-enthusiastic installers. There’s normally a few that I obviously don’t need, and one or two I need to take a closer look at to work out what they actually are. What’s really useful here is that using the checkbox alongside each entry, you can disable it without deleting it. Untick the boxes you think you don’t need, restart your computer, see how you go. If you suddenly realise one of them was actually something really important, no problem - fire up Autoruns again, check the box to switch it back on, and reboot.

You can download the whole SysInternals suite, including Autoruns, from https://docs.microsoft.com/en-gb/sysinternals/ - but if you don’t want (or aren’t allowed) to install anything, the entire suite is available as standalone portable EXEs at https://live.sysinternals.com/ - grab the tool you want, save it to the desktop, run it, then delete it when you’re done.

Oh, and if you want to know what the CTO of Microsoft Azure does for fun… there’s a VM configuration in Azure with 420 CPUs and 24Tb of RAM, and a few weeks back Mark wrote a program for it that will max out individual CPU cores in turn so the CPU load meter display plays a game of Tetris:

Achievement unlocked: playing Tetris on Task Manager on the Azure 24TB Mega Godzilla Beast VM: pic.twitter.com/4ntTsRqHEi
— Mark Russinovich (@markrussinovich) December 6, 2020

Absolute legend. I bet the CTOs of AWS and Google Cloud never do anything this cool. 😎

Holidays in the Holodeck

2020-12-22T12:00:00+00:00

I’m throwing a party! It’s been a bit of a rough year and with the new lockdown restrictions that have just kicked in across London and parts of the UK, Christmas is basically cancelled for a lot of folks… so on Sunday 27th December, I’m going to stream a live show on the internet, including loads of my parody songs, some PubConf-style comedy lightning talks, a couple of musical surprises, and almost certainly a feline cameo or two from Lionel.

It's at Sunday December 27th, at 19:00 UTC. I'll be streaming for around 2 hours, then we'll have drinks on Zoom.

Want to add it to your Google calendar? Here you go:

There's also a Facebook event for it if that's your thing, or if you want an ICS file, you can find one here

It would be lovely to see you there. Well, I won’t see you, obviously, ‘cos that’s not how streaming works. But you’ll see me, and if you put a comment in the chat, I’ll know you’re out there, and that would make me (and Lionel) very happy.

Dylan's Advent of Cool Nerd Things Day 22: Beyond Compare

2020-12-22T12:00:00+00:00

Today’s #nerdvent entry is another one of my must-have tools, a tool that’s saved me so much time over the years that it’s probably paid for itself many, many times over. It’s a file comparison tool called Beyond Compare, it’s available on macOS, Linux and Windows, and it absolutely rocks.

Let’s start with the easy stuff. You’ve got two text files, you want to know if they’re different – and if so, how. No problem. Open them up in Beyond Compare and it’ll show you exactly what’s changed between them:

Beyond Compare has comparison tools and viewers for many different file formats, so you can compare images visually to highlight any differences between them - and it’ll automatically scale images so your sources don’t even need to be the same size:

It’s got a three-way merge feature, too. For readers who haven’t encountered the joys of the three-way merge, it works kinda like this… imagine you and a friend are writing a book together - a thrilling airport blockbuster. You get a first draft that you’re both happy with… then a few days later, you email your friend saying “hey, I’ve had a brilliant idea - I’ve moved the story from London to Chicago” - and they email you back going “Oh, crap… I also had a brilliant idea… the lead character isn’t a professor any more, she’s a helicopter pilot!”

Now you have three different versions of your story - the original base draft that you both started with, your version, with all the landmarks and street names changed from London to Chicago, and their version, where our protagonist is a helicopter pilot instead of a professor. The changes aren’t fundamentally incompatible - we can quite happily end up with a story about a pilot that’s set in Chicago; we just need a bit of help reconciling the changes. That’s how a three-way merge works - it’ll compare your changes with those made by your co-author. Any passages where they didn’t change the text will be updated to reflect your changes; any passages where you didn’t change the text will be updated to reflect their changes, and any passages that you’ve both edited will be shown side-by-side so you can cherry-pick the bits you want to keep and/or rewrite that passage so it makes sense.

Turns out that kind of scenario actually happens a lot in collaborative software development (although we’re normally writing websites and mobile apps, rather than airport thrillers) so having a decent comparison tool that supports three-way merges is incredibly useful.

It’ll compare folders as well as files, which is useful in all kinds of scenarios - particularly if you’ve ended up with two different versions of a project that contains hundreds of files and subfolders, and you just need to know which bits are missing from each one. You can do a “quick compare”, which just looks at the file paths, dates and sizes, or you can compare files based on their contents – and even configure the comparison rules to ignore things that don’t matter (like Windows vs Unix-style line endings).

Unlike a lot of the tools in my Nerdvent list, Beyond Compare isn’t free – but it does have a feature that I haven’t seen in any other software which I think is absolutely brilliant. When you install it, you get a 30 day free trial, but the trial only counts days that you actually use the software. All too often, I’ll set up a 14-day free trial of something, use it for a day or two, then get sidetracked by another project; come back to it after a few weeks to continue evaluating it and… ah, no, it expired. Oh well. But the Beyond Compare 30-day free trial can last for months if you’re not using it very often - which means you get to evaluate it based on 30 days when you actually used it. And by the time it eventually runs out, you’ll probably find it much easier to justify paying the $30 (Standard) or $60 (Pro) license fee.

Beyond Compare is available at https://scootersoftware.com/ for Windows, macOS, and several popular Linux distributions.

Dylan's Advent of Cool Nerd Things Day 21: EarTrumpet

2020-12-21T12:00:00+00:00

In the beginning was the PC, and the PC had a speaker, and the speaker went “beep”. And the PC users listened to the Amiga 500s and Atari STs, with their glorious 8-bit FM synthesizer chips, and got a bit jealous. And so we saved up and bought an Adlib card so that we could bask in the luxurious musical possibilities of 9 channels of FM synthesized music. And then there was the SoundBlaster II, which had an FM synthesizer and a PCM chip for playing digital sampled audio (speech! actual actors! in a computer game! amazing!)… and then there was wavetable synthesis, and SoundFonts, and the Gravis Ultrasound, and then sometime in the late 1990s PC manufacturers finally figured out that audio was a Thing People Wanted, and now every PC, laptop and motherboard on the market comes with some sort of audio capability, and most of the time you don’t ever think about it beyond turning the volume up and down and maybe pairing some Bluetooth headphones.

A few years back, Windows worked out that a lot of folks were using their PC to make voice & video calls, so they added a feature that let you use one device for normal audio (music, games, etc.), and a second device for calls (a “communications device”). For most people, that works pretty well, but if you’re an audio nerd - or you’ve just got more than two audio devices hooked up at the same time - you’ve probably found yourself wishing you had a little more control. And if you’re doing any kind of remote training or live streaming, the ability to route different audio through different channels is an absolute lifesaver… we’ve all heard the horrible echo you get when someone’s on a Zoom call on their speakers and you hear your own voice a second or two after you’ve said something; imagine that, only it keeps echoing and getting louder every time and you have no idea what button you press to make it stop, and you’ve got some idea what can go wrong. Imagine you’re doing a remote workshop; you’ve got a PowerPoint deck with some audio in it. You need to hear the audio from your presentation; you need to hear the other people on the call, and you want to hear Windows desktop audio so you can still get chat notifications and stuff. Your audience need to hear your voice, and your presentation audio, but you don’t want them to hear your chat notifications, and you definitely don’t want them to hear themselves. That’s the “easy” setting. Now imagine a presentation with audio in it, but now you have a co-presenter who’s on a Zoom call with you and your audience are watching live on YouTube, so you need to make sure you’re routing the Zoom audio into OBS to stream it to YouTube, and your co-presenter can see your presentation and hear the audio. Oh, yeah, and the YouTube stream has 25 seconds of latency so if you make a mistake, you won’t find out about it for 25 seconds, then the YouTube chat will suddenly fill up with messages like “AUDIO PROBLEMS! BAD ECHO!” – and will CONTINUE to do this for another 25 seconds because that’s how long it takes the audience to see that you’ve noticed the problem and you’re fixing it…

And… breathe. Let’s meet EarTrumpet. EarTrumpet is a volume control app – all it really does is let you pick which apps play through which audio devices, and how loud those apps should be. It’s technically possible to do this via the Windows device settings if you know where to find it, but EarTrumpet is simple, it works, and for doing anything complicated with multiple audio devices, it’s an absolute lifesaver:

What’s particularly useful is being able to pin any app to a specific audio device – for example, if you always want Zoom to use your headset, regardless of what else you’re doing:

Finally, one thing to remember – not just with EarTrumpet, but with any device that can mute audio: it’s way too easy to turn something way down, forget about it, and then come back to it next week and be absolutely baffled as to why your audio isn’t working. A few weeks ago I used EarTrumpet to mute my web browser. Worked beautifully. A few days later… “why isn’t YouTube playing any sound? What? I AM CONFUSED…” until I remembered I’d turned it off. But to be fair, I’ve done the same thing before with guitar audio pedals (“what’s wrong with my amplifier? Maybe it’s the cable. Maybe my guitar is broken. Maybe all my guitars and cables are broken. Oh, no, hang on, that pedal is turned all the way down… d’oh.”)

EarTrumpet is slightly unusual, in that it doesn’t actually have a website. Like many of the things I’ve shared during #nerdvent, the code is open source and it’s available on GitHub, but to actually install it you can either download it from the Microsoft Store, or get it via Chocolatey (choco install eartrumpet)

Oh, and if you want to know where the synthwave wallpaper came from, it’s Falling Sun by Michal Kváč, via DeviantArt.

Dylan's Advent of Cool Nerd Things Day 20: OBS

2020-12-20T12:00:00+00:00

In November 2019, I spoke at an event here in London (yes, a real one – you remember those?) where the venue “forgot” to mention that one of the rooms had a glass ceiling. Now, there isn’t a lot of daylight in London in November, but there’s definitely enough that if you’re in a room with a glass ceiling, your PowerPoint slides aren’t going to project terribly well. I watched a few talks early in the day, realised this might be a problem, and thought “hmm… maybe I can stream my slides to YouTube or something so that folks who can’t see the projector screen well can watch it on their phone?” So I googled “stream powerpoint to youtube” and found out about something called OBS - Open Broadcaster Software. I installed it, poked around a bit, managed to stream my PowerPoint presentation to YouTube, mentally filed it away under “OK, that might be useful one day”, and got on with my life.

Fast-forward to March 2020, and those emails start coming in… a trickle at first, then a wave: “We would like to announce that $conference is now a Virtual Online Event!” And I think “ok, this could get interesting. Perhaps that OBS thing can help.”

And it did. OBS is brilliant, and it’s been an absolute game-changer for me running online events and doing livestreamed shows during 2020. It’s an incredibly powerful, and flexible, set of software tools – and it can be a little confusing until you figure out what’s what.

First up: OBS is an open source project. This means you’re free to download it, use it, share it, modify it - and you’re even allowed to modify it and publish your modifications under a different name. That’s exactly what the folks at Streamlabs have done, which means there are actually two very similar apps out there, “plain” OBS, and Streamlabs OBS. The Streamlabs version adds a bunch of bells & whistles aimed at folks who do a lot of live streaming, but this article is about the original OBS, the one that’s available on https://obsproject.com/.

At a very high level, here’s how it works:

It takes input from video and audio sources. Cameras, microphones, screen sharing, local video files - even online sources like live webcam feeds. You use the OBS Studio software to arrange all those sources into what’s called a scene, and then record the results to a local file, and/or stream it live to services like YouTube and Twitch. You can create multiple scenes, set up hotkeys to switch between them, and get a pretty professional-looking stream going relatively quickly.

You can use OBS quite happily with just your laptop’s built-in webcam and a regular headset mic - capture your screen, put a “talking head” window in one corner using your webcam feed, enable your audio, and you’re good to go. It’s great for recording demos, training videos, or just doing some cheap & cheerful live streaming.

Things get really exciting when you add a bluescreen or greenscreen into the mix; OBS has built-in filters to do things like removing your greenscreen background so you can get a “floating head” effect (anyone old enough to remember the days when humans relied on television for weather forecasts will recognise this as the “TV weather person” effect). Here’s a shot from one of my programming live streams:

And if, like me, you get totally obsessed with things and end up buying four greenscreens so you can have greenscreen backgrounds on three walls and the floor of your office, you can do some fantastically clever things with OBS and multiple camera angles. Here are a few shots from the conference party show I did for BuildStuff back in November – me playing guitar and singing, live, on a virtual stage, with various backing videos and effects composited into the shot live via OBS:

OBS is absolutely packed with features, but one that’s particularly useful is the Virtual Camera - this used to be a separate plugin, but now it’s built in to to OBS Studio. This lets you take your composited video with all your effects and backgrounds and whatever else, and share it as a virtual camera that you can select in Zoom, Google Meet or whatever else you’re using. Use it carefully - remember that Zoom, etc. think that it’s a camera and will apply some fairly hardcore video compression that works pretty well on real cameras pointing at real faces, and if your OBS scene includes source code, diagrams or anything with a lot of detail, it will probably look dreadful, so make sure you test it out before doing anything important.

If you want to see somebody who’s really taken their OBS fu to the next level, take a look at LaylaCodesIt on Twitch. Layla’s partnered with Jim Mc̮̑̑̑͒G and his Silicon Orchid project to really push the limits of what’s possible on a live video stream - immersive 3D backgrounds, realtime raytracing, all kinds of cool effects; check out Jim’s blog post on how it’s all put together.

But like I said, you don’t need all that to get started - just a laptop with a webcam and a microphone. OBS is available at https://obsproject.com/ - and it’s free, so download it, play around with it, and see what you can do. And if this post inspires you to start streaming, give me a shout on Twitch or on Twitter and I’ll drop by and say hi :)

Dylan's Advent of Cool Nerd Things Day 19: Canva and Unsplash

2020-12-19T12:00:00+00:00

Visual design is one of those things that many people appreciate but don’t understand - and by “appreciate”, I mean that design can change the way we perceive things, even if they don’t necessarily realise why that’s happening. In a world of 4G, wifi and smartphones, we’re surrounded by information, all day, every day (although the fact you’re reading this blog post suggests you haven’t quite had your fix for today, and hey, I’m glad you’re here!). A bit of visual design can be what you need to make your message stand out from the background, whether you’re using social media to promote something you’re working on - a project, an event, a charity campaign - or you just want to wind up one of your mates by taking something stupid they said and turning it into an inspirational quote poster.

Canva is a web app for designing things. I found out about it from my friend Emma Dingle, who runs Social Results Ltd and manages social media for all sorts of big names - and thanks Emma for the heads-up, because Canva is brilliant. If you’ve ever played around with the clipart and templates in something like Microsoft Word or Publisher… well, Canva feels like it’s what those things were trying to be. Start by picking a template for the thing you’re trying to make, whether that’s an invitation, a poster, a banner to share on social media. Tweak the wording, maybe pick a different background, and you’re done.

Like a lot of creative tools, you can get sucked into a spiral of trying out different options - fonts, colours, backgrounds, layouts. The secret is to not worry about it too much. Find a template that’s close to what you’re looking for, tweak the wording, save the results, and get on with your life.

I want to mention another tool today that works beautifully with Canva, and that’s Unsplash.

Unsplash is a huge archive of free photography - beautiful images you can download, redistribute, and use freely in your own work. You can’t sell them, and you can’t use them to create a competing photo library, but apart from that you can do pretty much anything you like. Using a background I found on Unsplash and a Canva template, it took me about five minutes to put together this image, based on one of my favourite quotes ever:

It’s got some really nice touches, like automatically generating a colour palette from the tones in your background image, and if you do want to dive down the rabbit hole you can tweak just about every detail - layout, font, size, colours, effects. But primarily, it’s perfect for that sweet spot where plain text is a little too plain but firing up Photoshop or Illustrator feels like overkill.

Check out Canva at https://canva.com/ and Unsplash at https://unsplash.com/, and let’s fill the internet with beautiful things.

Dylan's Advent of Cool Nerd Things Day 18: GitKraken

2020-12-18T12:00:00+00:00

Git is a phenomenally powerful system for managing source code files - so powerful, in fact, that I’ve joked before that Git was actually invented in the future, and sent backwards in time as a way of teaching human minds how to deal with non-linear history and parallel timelines. Like a lot of developers, Git never really clicked for me until I started using GitHub to host my projects - but these days, working with branches, forks, pull requests and merge heads is so fundamental to how I build software it’s hard to imagine working without it.

(As an aside, I also talk to more and more up-and-coming developers now whose first experience of programming wasn’t sitting in a room by themselves writing little programs, but contributing a few lines of code or documentation to a significant open source project - and I think that’s a very good thing indeed, and I think GitHub deserves a lot of the credit for making collaborative coding so accessible. Nice work, GitHub.)

The Git data model is still incredibly complex, though, and… look, I know there are folks out there who are absolute virtuosos on the git command line; folks who can branch, stash, merge, pop, rebase, commit and push without even skipping a beat. I am not those people. I find it much, much easier to reason about complexity if I can see it, and working with git is one of those areas where I have a handful of familiar commands that I’ll run in a terminal, and for anything even slightly complex I’ll reach for a GUI.

My favourite git GUI tool for the last few years has been Axosoft’s GitKraken, and it just keeps getting better and better.

Fundamentally, it presents the history of all the branches, merges and tags in your project in a way that I find very easy to understand. It’s got built-in merge tools, and in recent versions has added first-class support for working directly with GitHub features like issues and pull requests. Git and GitHub are fantastic tools, but they definitely have a learning curve, and I’ve always found tools like GitKraken invaluable for helping me visualise what’s going on.

Oh, and their logo/mascot? He’s a kraken called Keif, and he loves to dress up - check out the Keif Gallery on their site for some wonderfully whimsical variations on the standard logo. I think this one here is my personal favourite:

Dylan's Advent of Cool Nerd Things Day 17: Carbon

2020-12-17T12:00:00+00:00

Source code is weird stuff. On one hand, it’s just regular text with a whole lot of weird punctuation. On the other hand, it’s indistinguishable from magic – arcane spells and incantations that we use to control the lightning that we trapped inside the rocks, solving hundreds of millions of calculations in a heartbeat, sending images and sounds and ideas flying around the world…

Code is hard to read. Any experienced developer will relate to the experience of opening up a file you’ve not seen before and staring at it blankly, trying to figure out what’s going on… why does a function called JoinStrings return an integer? What on earth does the HelperManagerFactory do? Why doesn’t Array.map return a Map?

Alas, we haven’t yet invented something that makes code readable… but today’s #Nerdvent tool can at least make it look good. It’s a website called Carbon, which I found thanks to Kevlin Henney, which lets you “create and share beautiful images of your source code.” And I mean, this thing is stunning. Just lovely. Here’s part of my solution to Advent of Code day 15, using the SynthWave ‘84 theme and the Fira Code font we saw the other day:

Isn’t that glorious? You can almost smell the rain sizzling on the neon lights…

Here’s Duff’s Device, with the Blackboard theme and the JetBrains Mono font:

Carbon will export your code images as PNG or SVG files, which is great for making presentation slides - but if you want to embed them online, there’s an IFRAME option as well, which gives you beautifully highlighted code that your readers can still copy & paste:

Carbon is over at https://carbon.now.sh/ - check it out, give them a shout out on Twitter @carbon_app, and have fun!

Dylan's Advent of Cool Nerd Things Day 16: 7-Zip

2020-12-16T12:00:00+00:00

I was working desktop support back in the days when email didn’t really work yet, and about once a week, somebody would call Helpdesk to say they couldn’t open an email attachment. By the time I eventually left that job, I knew more than you can possibly imagine about compression and archiving formats. ZIP, BZIP, GZIP, TAR, UUEncode, LZH, LZW… and those were just the ones we had back in 1997. These days, email has got a lot better, but once in while I’ll still get an attachment in a format I’ve not seen in a while. And now that almost everything is distributed online via the web, you’ll also find virtual CD and DVD images floating around in .ISO format, macOS installers baked into DMG files, VMDK files containing virtual hard drives, MSIs and CAB files containing software installers… isn’t progress wonderful?

That’s where 7-Zip comes in. 7-Zip is a free Windows application that will open just about any archive file format you can think of, as well as creating and managing archives in most popular formats. I’ve used it on every Windows computer I’ve run since Windows 2000, and found it absolutely indispensable – it’s one of the first things I install on any new machine I’m setting up, and one of the first things I notice is missing if I’m working on someone else’s PC and they don’t have it installed.

It’s free (GPL2), it’s incredibly useful, and it’s available at 7-zip.org - check it out.

Dylan's Advent of Cool Nerd Things Day 15: Paletton

2020-12-15T12:00:00+00:00

Colour (or color, if you’re an HTML parser, a .NET class specification or a human being from the United States of America) is complicated. Way complicated. I’m a pretty monotone kind of person - my clothes are black, my walls are white, most of my favourite foods are beige, and I’m still not entirely comfortable with the existence of green LEGO bricks.

But colour matters. There’s a wonderful infographic from The Logo Company called the Color Emotion Guide, that illustrates how dozens of popular brands use colour to evoke particular emotions and connections when consumers look at their logos:

When it comes to colour - and all sorts of aspects of visual design - I tend to group things into three categories. Category 1 is the things that just blow you away with how brilliant they are. Category 2 is the stuff you don’t really notice. And category 3 is the things that are so awful you can’t really ignore them.

If you want your projects to end up in category 1, find a brilliant designer, pay them, and trust them; if you want your projects to end up in category 3, let your database engineers design your user interface. But if you’re happy sitting quietly in category 2, you can do a lot worse than learning some basic design principles, and then relying on tools that can help you avoid common mistakes.

Paletton is such a tool. It’s an online tool for creating colour schemes, and it’s delightfully simple; you pick a colour you like, and tell it how many complementary colours you’d like to go with it and what sort of generator to use, and it’ll give you a palette. It’ll generate pretty much any kind of scheme, from bold contrasting primary colours to subtle muted pastels, but it’s based around mathematical colour models that should help you create usable colour schemes and avoid any horrible clashes and unpleasantness.

Formerly known as Color Scheme Designer, Paletton is online at https://paletton.com/. It’s ad-supported, but it’s much more usable with an ad blocker running, so hey, why not block the ads and throw them a few bucks via the Donate link if you find it useful?

Solving Advent of Code in Amstrad LOGO

2020-12-14T12:00:00+00:00

Advent of Code is an “Advent calendar of small programming puzzles”; every year, the creators come up with a set of 25 programming puzzles and release them each day from the 1st to the 25th of December. This year, I resolved to do Advent of Code every day, and to do it live on Twitch – most days I’m streaming at 16:00 UTC for an hour or two, so feel free to stop by! It’s been a lot of fun, and so far I’ve managed to solve both parts of every day’s puzzle live on the stream; I think the longest time to solve so far is nearly 2.5 hours for the nested luggage problem on day 7.

On day 12, “Rain Risk”, the puzzle was about navigating a ship – and the puzzle input was provided as a set of commands that looked a little reminiscent of the first programming language I ever used: LOGO. My first computer was an Amstrad CPC6128, and as kid I’d sit there for hours using LOGO to draw shapes and build little quiz games. When I created my talk “The Art of Code” last year, I wanted the intro sequence to reflect that, so I found an Amstrad CPC emulator (the extremely brilliant WinAPE), worked out how to boot LOGO on it, and how to edit LOGO programs and instructions in a proper editor on the host PC and then use WinAPE’s auto-typing feature to replay those instructions into the LOGO interpreter.

(As an aside, WinAPE is so detailed that it not only plays sampled audio of the 3” disk drive used on the Amstrad CPC6128 when you’re reading and writing files, it plays it through the right stereo channel because on the CPC the disk drive was on the right… the attention to detail that’s gone into this thing is just unbelievable.)

So when I looked at yesterday’s Advent of Code puzzle, my first thought was “hey… maybe I could solve this one using Amstrad LOGO”.

Want to know if it worked? Check out the YouTube video of the stream.

If you want to catch the rest of Advent of Code live, I’m on twitch.tv/dylanbeattie most days at 16:00 UTC – why not stop by and say hello?

Dylan's Advent of Cool Nerd Things Day 14: Typora

2020-12-14T12:00:00+00:00

Writing things on the web can be harder than it looks. At the one extreme, you can just publish plain text files – and, yes, I know there’s no such thing as “plain text”, but a UTF-8 file served with a text/plain MIME type header will be readable on just about any system that can render the alphabet it’s written in.

At the other extreme, there’s the rich tapestry of tags, layouts and formats provided in modern HTML – grid layouts, flow layouts, drop caps, web fonts, and all sorts of wonderful typographic things that we never even dreamed of back when I started hacking web pages together in the 1990s.

The problem is that they both kinda suck. Managing online web content is significantly harder than it looks - online content management is a multi-billion-dollar industry, and some very smart people have written entire books about managing the information and presentation architecture that’s required by modern websites. And at the other end, plain text is… well, it’s a bit plain, y’know?

Everyone has their own “sweet spot” – the point where you have just enough control to achieve the things you want to do, without incorporating so much flexibility that managing the options becomes a maintenance headache. For me, that sweet spot is a file format called Markdown, It’s plain text with just enough special sauce to plug in images, headings, code snippets and bullet points – the kinds of things I want to think about when I’m writing articles – but without having to think about layouts, fonts and typography, which are the kind of things I want to think about once every five years and then forget about until the next redesign.

The last big overhaul of this site was in August 2019, when I migrated the whole site from Blogger to GitHub Pages, using a publishing system called Jekyll. Jekyll loves Markdown files, and all of the posts and articles I’ve published since then have been written in Markdown, using an editor called Typora.

Typora is wonderful. It’s a lightweight, minimalist editor for Markdown files that is what I called “just WYSIWYG enough” - things like headings, bullets and code snippets get rendered on the fly, so you can get an idea of the layout of your article, but if you need to include chunks of HTML to do things like embedding IFRAMES for playing YouTube videos, just do it - if Typora knows how to render them, it will, otherwise it’ll just leave the HTML tags visible and you can check it in a browser.

Typora has a bunch more features I find really, really useful, like being able to use Jekyll frontmatter to define where to store your images, and then copy’n’paste images from the Windows clipboard and have them automatically converted to PNGs and dropped into the right folder.

It’s got a whole bunch more cool features that I’ve hardly scratched the surface of - sequence diagrams, tables, mathematical equations - but fundamentally, it’s just a really, really good tool for writing blog posts and articles.

The only open question about Typora is how much it’ll cost. At the moment it’s in beta, and it’s free-as-in-beer - it isn’t open source, but it doesn’t cost money. It’s been on version 0.9.9.something for a while now, though, so I’m expecting it to ship 1.0 any day now, and I’m curious to see what their pricing model is going to be when that happens. I have no problem at all paying for good software, it’s just a little disconcerting becoming this attached to a tool without knowing what it’ll end up costing. But hey, right now it’s in beta, the beta is rock-solid, and it’s free for Windows, macOS and Linux. And it’s excellent.

Check it out at https://typora.io/

Dylan's Advent of Cool Nerd Things Day 13: Chrome Dino Game

2020-12-13T12:00:00+00:00

You’ve probably seen this screen a few times… maybe you’ve been on your phone on the train and you’ve travelled into an area with no signal, maybe your home wifi has died again… or maybe you went for a nice long walk in the countryside and realised too late that the countryside is actually quite big, most of it doesn’t have terribly good mobile phone signal, and Google Maps doesn’t work if you don’t have signal. In any case, you’ve had to figure out how to survive Without The Internet for a spell. We know it is theoretically possible to survive Without The Internet because apparently human beings did it for, well, most of history… but hey, just because you’ve got no 4G is no reason to go back to the cretaceous period… or is it?

You see, the “No Internet” screen built into Google Chrome, and in most other browsers that use the Chromium browser engine, has a little picture of a dinosaur as a sort of in-joke about life without internet being prehistoric. But you might not have known that there’s actually an entire game hidden in there… try it! You could disconnect from wifi, but you’re reading my blog, so that would be a really bad idea. Instead, try browsing to chrome://dino/, press Space (or tap the screen if you’re on a touchscreen)… and away you go!

Run merrily through the desert, jump over the cacti, avoid the pterodactyls… like Space Invaders and Tetris, the gameplay is incredibly simple but remarkably well-balanced and it’s surprisingly easy to get hooked on it. Especially if you’re stuck on a train that’s in a tube tunnel waiting for a vacant platform at Liverpool Street (y’all do remember trains, right?)

Check it out at chrome://dino/, or just wait for the internet to go down. It’s on most Chrome-based browsers, and it’s also built into Chrome for iOS, even though Chrome on iOS devices is actually a Chrome wrapper around the Safari rendering engine. Have fun, and watch out for the absolutely wonderful expression of stunned surprise on the T-rex’s face when he gets hit right in the unfortunates by a low-flying pterodactyl. It’s delightful.

And to think people say there’s no character development in computer games.

Dylan's Advent of Cool Nerd Things Day 12: Open Hardware Monitor

2020-12-12T12:00:00+00:00

Until this year, my main computer was a 2014 Macbook Pro – in fact, for much of that time, it was the only computer I owned. I still use it for almost all my music production work, but when 2020 kicked in and it became apparent that there wasn’t likely to be a whole lot of travel going on, I invested in building a proper powerhouse PC workstation. I wanted something that struck a very delicate balance between processing power and fan noise - it’s pretty easy to get modern PCs to run ridiculously fast if you can just stop them overheating, but finding something that’s fast and quiet is a bit trickier.

Enter Open Hardware Monitor. It’s a free open source application that’ll read just about every hardware sensor on any modern PC motherboard - CPU voltages, memory utilisation, various load metrics for SSD and NVRAM storage devices, and, most usefully for me, fan speeds and temperatures. I keep Open Hardware Monitor running pretty much all the time, and if one of my apps starts grinding or becomes unresponsive, a quick glance at OHM shows me whether something’s overheating or not. The first couple of weeks running in my new system, I’d often have to tweak fan profiles and settings to get everything nicely balanced; these days it’s all settled in quite nicely but it’s still reassuring to look over at OHM and see that everything’s under control.

Oh, and it’s a .NET WinForms app. The BEST application development framework in the history of software development, ever, bar none. Remember when you could build apps without having a meeting to decide how to hire the design agency who would tell you what colour to make the buttons? A framework from a day when computers were beige, buttons were grey, and rounded corners were for soft play areas. And, hey, it’s just been ported to .NET Core and will be part of the .NET 5 runtime, so who’s laughing now, hey? Did somebody say WPF?

You can get Open Hardware Monitor from https://openhardwaremonitor.org/ - and if you fancy a bit of old-school WinForms hacking, the source code is on GitHub and released under the Mozilla Public License 2.0, so grab the code and see how we used to build apps back in the good old days. And if anyone wants to add “dark mode” to it, I’ll buy them a beer… 🍺😃

Dylan's Advent of Cool Nerd Things Day 11: Audacity

2020-12-11T12:00:00+00:00

Audacity is a free program for recording and editing audio files - sound recordings, samples, music. More than an audio player, but simpler than a full-blown digital audio workstation like ProTools or Logic Pro, it’s another one of those tools that I find myself reaching for way more often than you might think.

If you’ve ever wanted to edit a sound clip to include it in a presentation, record a voiceover, or even just record your own microphone to see what you sound like before that Zoom job interview you’ve got coming up, Audacity is ideal. It’ll read and write just about any audio format you can think of. It also includes dozens of professional-quality effects, filters and tone generators - so if you want to know what you’d sound like as a Dalek, Audacity can help.

Check it out at https://www.audacityteam.org/

Dylan's Advent of Cool Nerd Things Day 10: Fira Code

2020-12-10T12:00:00+00:00

Today’s Cool Nerdvent Thing isn’t a tool, or a utility, or a website, or a hidden Windows feature… it’s a font. But not just any font – this font is Fira Code, and it is absolutely the best font ever created for computer programming.

Modern computer font systems like TrueType and OpenType support a feature called ligatures. Ligatures date all the way back to the days of mechanical typesetting and movable type, when printers (as in, people who run printing presses, as opposed to, say, HP Laserjets) would pick a specific glyph to represent common pairs of characters:

You can get lots of fonts that include ligatures for common letter sequences like those above, but Fira Code takes things to a whole new level by using ligatures to replace common programming character sequences with symbols. It’s not the only programming font that uses ligatures – there’s a great article on Better Web Type about monospaced fonts with coding ligatures – but it’s one of the best, and it’s definitely my personal favourite.

Here’s a snippet of F# code using the regular Consolas typeface, and the same code shown in Fira Code:

This is exactly the same source code – it’s a regular ASCII text file; the only thing that’s changed is the font. At a glance, the ligatures might look like an eye-catching gimmick, but I find them really, really useful for working on code that has lots of gnarly operators - if you’ve ever had a JavaScript bug caused by double-equals (==’) vs triple-equals (===), you’ll appreciate how visually distinct Fira Code makes these two character sequences.

Underneath the ligatures, though, there’s a really solid, readable monospaced font, available in various weights for different kinds of editors and displays. Fira Code is released under the SIL Open Font License 1.1, and it’s available as a TrueType TTF file that’ll work on most operating systems, most text editors and programming tools, and it’s available in Open Font WOFF format for use on your website as well.

Fira Code is created by Niki Tonsky, and it’s available at https://github.com/tonsky/FiraCode.

Dylan's Advent of Cool Nerd Things Day 9: Microsoft PowerToys

2020-12-09T12:00:00+00:00

Way back in the days of Windows 95, Microsoft released a set of freeware utilities called the PowerToys for Windows 95. It included some things that were genuinely useful – like being able to right-click any folder and open a command prompt in that folder – and some stuff that was, um, less useful, like the FindX tool that added a drag’n’drop capability nobody wanted to a Find menu nobody ever used. Anyone remember FindX? Anyone…? Anyone…?

Fast-forward a few decades, and the PowerToys are back – this time for Windows 10. They’re free-as-in-beer, they’re free-as-in-speech (MIT license), and they’re really rather good. There’s two that particularly stand out for me. One is the Windows key shortcut guide – hold down the Windows key for more than a second and an onscreen guide pops up telling you all the available shortcuts:

The other is the Image Resizer, a shell extension that offers bulk image resizing via a right-click context menu, which is incredibly useful when you’ve got a bunch of 20 megapixel images off your DSLR camera and you want to email copies to somebody without breaking the internet.

Alongside those, there are a bunch of tools for video conferencing, batch renaming files, and organising your Windows desktop – I’ve heard that Fancy Zones is fantastic if you’ve got one of those ultra-super-mega-widescreen wraparound displays. They’re definitely worth a look if you’re a Windows power user, and the PowerToys team is adding new experimental tools and utilities all the time so check back often to see what’s new.

Check them out at https://github.com/microsoft/PowerToys, or go straight to the releases page for the latest installer.

Dylan's Advent of Cool Nerd Things Day 8: Instant Eyedropper

2020-12-08T12:00:00+00:00

This one’s a tool I’ve used for years and years, and it’s absolutely wonderful. You know the eyedropper tool in Photoshop? It’s basically that, but it will pick up the colour of any pixel on your desktop and copy the HTML hex colour code to the clipboard.

That’s pretty much it. You can configure the mode; “click” mode is one click to activate the tool, and again to select a colour and copy it, and if that’s just too many clicks, put it in “drag” mode and then just drag the mouse from the tool onto any pixel on the screen. You can also tweak the exact HTML format to get your hex digits in your preferred format – but, really, it’s just a wonderfully simple, useful, unobtrusive tool. It sits in your system tray until you need it, it does exactly what it’s supposed to, and then it gets out the way so you can get on with whatever you were doing.

It’s free, open source, runs on Windows XP through Windows 10 (and I know, because I’ve run it on every single version of Windows from XP to 10) – check it out at http://instant-eyedropper.com/, and if you find it useful, why not throw $5 their way to support the project?

Dylan's Advent of Cool Nerd Things Day 7: ngrok

2020-12-07T12:00:00+00:00

Most of them time, when I’m building web apps and websites, I follow a fairly common workflow: get stuff working locally, test it locally, and then push it out on the Actual Internet somewhere where people can see it. Most of the time, this works pretty well, although there are definitely occasional cases of “well, it worked on my machine!” when code that was working perfectly on localhost doesn’t do what you expect when it goes live.

But there are a handful of workflows where testing on localhost isn’t an option because something else, out there on the wild wild web, needs to interact with your code. Maybe you’re building something like an OAuth2 flow so users can sign in to your apps using their social media accounts. Maybe you’re using a cloud-based CRM or a deployment system that’s using webhooks to pass notifications back to your own code, and you need a way of testing that the webhooks are working. For these kinds of applications, localhost isn’t enough - you need a way that somebody else’s code, running out there on the web, can talk to the development code that you’re running on your machine right now.

Now, you could set up a certificate, open a port on your home or office firewall, register a domain name, set up some DNS host records… in fact, I’ve posted recently about how I’ve done exactly this. But it’s a lot of effort and it’s easy to overlook something and end up leaving something exposed on the internet that shouldn’t be there.

Alternatively, you can use ngrok. Ngrok is brilliant. It’s a tiny executable. You download it, you run it, you tell it what port to listen on, and it creates a secure, temporary tunnel from the internet to that port on your local machine. The tunnel stays open for a long as ngrok is running – and when you stop ngrok, the tunnel goes away.

There’s a “free forever” plan that’ll give you a single process supporting up to 4 tunnels on randomly assigned URLs; if you upgrade to one of their paid plans you can get reserved domains, reserved TCP addresses, SSO, and higher connection limits. Like a lot of the tools on this list, I used the free version for a while and then paid for a pro license because I realised it would save me so much time it would easily offset the cost of licensing it; it’s one of those tools that once you get the hang of it, you’ll find it’s useful for all kinds of things. I’ve used it to test “sign in with Twitter”-style OAuth2 flows; I’ve used it to debug webhooks from Paypal and from Microsoft Dynamics 365; I’ve even used it to share work-in-progress code with folks over a Zoom call or something so they can check it on their own browsers & mobile devices.

Check out ngrok at https://ngrok.com/

Dylan's Advent of Cool Nerd Things Day 6: Franz

2020-12-06T12:00:00+00:00

Have you ever had that thing where you wake up in the morning, and you think you might have seen a message from somebody on your phone at 5am, but now you can’t see it? And you don’t know whether you dismissed the notification or dreamed the whole thing? And you check email, and WhatsApp, and Telegram, and Slack, and don’t find it, and so you assume you must have dreamed it and then a week later you find it in your LinkedIn inbox?

It’s kinda nice living in a world where it’s so easy to stay connected to people, but, well, gosh, things have gotten a bit out of hand. Right now I’m in 34 Slack workspaces, 5 Discord servers, 4 WhatsApp group chats, plus Twitter DMs, Facebook Messenger, Skype, LinkedIn messages, email… and when you’re an independent consultant like I am, you need to work with whatever channel your clients are using.

That’s where Franz comes in. It’s a Windows desktop app that pulls together basically ALL THE THINGS and gathers them into one place:

The free tier lets you add up to three services; paid subs give you more connected apps, plus cool things like the Franz Todo List. I’ve been running it for a couple of weeks, and recently activated the free trial to see how it copes with lots of connected apps - but so far I’ve been really, really impressed.

I’ve got to admit, there’s a point in the setup where I stopped and thought “hang on… I downloaded this random app from the internet and now I’m giving it credentials to send messages, as me, on basically every platform I use” - which is a little scary. But behind the scenes, I think it’s just pulling down the web interface versions of all those different services and wrapping them into a single desktop app with separate views for each service. Don’t quote me on that, but it would explain how they’ve managed to integrate things like WhatsApp that don’t really publish any kind of API.

Oh, and their SEO strapline made me giggle:

Franz* is a free messaging app /former emperor of Austria, that combines chat & messaging services into one application.

Check it out at https://meetfranz.com/

Dylan's Advent of Cool Nerd Things Day 5: WClock

2020-12-05T12:00:00+00:00

A few years back, I had the singular experience of flying from Novosibirsk to London via Moscow, on the weekend that the clocks change. Novosibirsk is 4 hours ahead of Moscow and it’s a 4-hour flight - so you get on a plane at 10pm and land four hours later, at 10pm, and then somewhere in the air between Sheremetyevo and Gatwick the UK jumps around by an hour and you land sometime in the morning completely discombobulated.

Because, yeah, timezones. Just one more way in which 2020 has been an endless amount of fun - folks who used to spend half the year in airport lounges and hotels are now spending at at home, trying to keep up with all the events and meetings and people they would normally, and doing it all remote means lots of fun trying to schedule meetings and work out whether that conference you’re going to has published their schedule on your time, their time, or UTC.

Enter WClock. It’s a desktop clock for Windows computers. It supports multiple timezones, it’s extremely customisable, it’s “always on top” - so whenever I’ve been running a training course or speaking at a conference where we’ve got folks from multiple timezones joining, I’ll take a moment to put all their clocks into my WClock so I can remember who’s when.

My regular one currently looks like this:

Well, what, did you expect? High-def raytraced graphics and lots of lens flare? It’s a clock!

OK, here it is with some options:

It’s wonderfully old school, it’s simple, it works, and I’ve found it really, really useful. Check it out: https://www.di-mgt.com.au/wclock/download.html

Dylan's Advent of Cool Nerd Things Day 4: X-Box Game Bar

2020-12-04T12:00:00+00:00

You know when you stumble across something that makes you go “wow, how did I have no idea this was a thing?” Well, I’ve run some version of Microsoft Windows almost every day since Windows 3.1, and until a few months ago I had no idea the X-Box Game Bar even existed.

If you’re on Windows, try it. Press Win-G - there’s a whole bunch of interesting things hidden in there:

The one that stands out for me is the Capture tool - one-click screen recording, which is really useful if you need to record a demo or reproduction steps for a bug report and don’t have anything like Camtasia available. There’s also audio volume controls, CPU/GPU performance metrics, and a chat system that’ll connect to Steam, Reddit, Discord, Facebook, Twitch and various other chat services. And if you’re running Windows 10, you have all this installed already. Just press Win-G. Pretty cool, huh?

Dylan's Advent of Cool Nerd Things Day 3: Reincubate Camo

2020-12-03T12:00:00+00:00

2020 was the year that webcams suddenly became a big deal. For folks who’d already been working remote for a while, it was no surprise how important camera and audio quality are for video meetings - Scott Hanselman wrote a great post about this back in August last year - but with the entire world suddenly working from home, webcams sold out pretty much everywhere back in March and lots of folks have been working remote using their laptop’s built-in camera.

Of course, most of those folks also have a smartphone, and the camera in most modern smartphones is truly astonishing, which raises the question: can I use the camera in my phone as a webcam? There are several apps out there that claim to do this, and I’m pretty sure I’ve tried all of them, but the first one that’s actually worked for me is Camo by Reincubate.

I’ve tried it with an iPhone 6S and an iPhone SE, on a 2014 Macbook Pro and a couple of different Windows machines, and been really, really impressed. It just works. Run the installer, install the app, connect your phone using a USB cable, and it works. The image quality is fantastic; the free version gives you control over the camera’s focus and resolution, and if you pay £34.99 for the Pro version, you get full control over your phone camera’s exposure, white balance and various other settings.

Plus - did I mention the image quality is good? Here is in Zoom, side-by-side with my Logitech C920 HD webcam:

It’s on macOS and Windows, but it’s iPhone only (sorry, Android fans.) The consumer version of Camo for Windows is still in beta - there’s a signup form on their site; I’ve been running it on Windows for a month or so without any problems at all, including running it alongside four other cameras in a really elaborate multi-cam streaming setup.

Oh, and apparently there’s a 40% off until December 4th if you sign up with the code CYBERCAMO.

Check it out: https://reincubate.com/camo/

Dylan's Advent of Cool Nerd Things Day 2: WinDirStat

2020-12-02T12:00:00+00:00

“But I had LOADS of disk space… where’s it all gone?”

We’ve all been there, right? Shiny new computer with endless gigabytes of free disk space… and then suddenly one day you’ve got 128Kb free and you’re like “No way…”

Well, you can spend hours and hours opening folders, poking around, trying to work out where it all went – or you can use WinDirStat. (If you’re on macOS, DiskInventoryX does pretty much the same thing; on Linux there’s QDirStat.

Point it a drive or a folder, give it a few minutes to figure everything out, and boom – there’s a map of all the files on your hard drive. Bigger rectangles are bigger files, file types are colour-coded, and there’s loads of useful metadata about them all too. Here’s my D: drive:

Yep, that’s a 333Gb Dropbox with 120Gb of musical parodies in it – and yes, I have over 50Gb of Powerpoint presentations in my Dropbox. Good thing storage is so cheap, right?

WinDirStat is free (it’s released under a GPLv2 license), and you can find it at https://windirstat.net/index.html.

Dylan's Advent of Cool Nerd Things Day 1: Sizer

2020-12-01T12:00:00+00:00

I’m a stickler for detail. If I’m taking screenshots to use in a blog post or a PowerPoint slide, I like to get the window I’m capturing to be exactly the right size and shape before I snap it. This used to involve a lot of very, very patient dragging – and then I found Sizer. It’s a tiny Windows app that runs in the background until you right-click a window frame:

It’s got loads of cool extra features as well, like snap to grid, custom sizes, and it’ll move windows as well as resizing them. Check out the user guide to see what else it can do.

Sizer is online at http://www.brianapps.net/sizer4/ – check it out.

Spark - a tiny standalone HTTP server

2020-11-24T12:00:00+00:00

TL;DR: Spark (https://github.com/rif/spark) is a tiny standalone web server, written in Go, with binaries available for most platforms. And it’s very cool.

When it comes to web dev, I’m kinda old school. I’ve been building web apps long enough that I remember when you could build an entire web site just by pointing your browser at file:///c:/websites/mysite/index.htm, hack it around until it worked, then upload the whole folder via FTP and boom, job done.

These days, you can build some pretty amazing things using HTML, JavaScript and CSS; everything runs in the browser and all you need on the server side is a way of sharing static files over HTTP. But for all sorts of reasons - mainly to with security around things like AJAX requests - you’re gonna need an actual web server running on your dev machine; most of the cool stuff that the modern web does just doesn’t work over file:/// URLs. So I’ve been looking around for a while for a really fast, lightweight HTTP server.

For a while, I was using Python for this - python 2.x has a builtin module called SimpleHTTPServer that will share the current directory over HTTP:

D:\projects\github\>python -m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80 ...

That works pretty well, but it’s a bit verbose. Plus, the module syntax has changed in the Python 3 - it’s http.server instead of SimpleHTTPServer - and, obviously, it won’t work without Python installed.

Enter Spark. It’s a tiny standalone web server, written in Go by Radu Ioan Fericean, with binaries available for Windows, macOS, and Linux. Add it to your system path, and then you can just type spark in any directory and it’ll share it over HTTP on port 8080.

D:\Projects\github>spark
2020/11/24 11:19:49 Warning: serving files without any filter!
2020/11/24 11:19:49 Serving . on 0.0.0.0:8080/...

D:\Projects\github>spark -port 80
2020/11/24 11:19:58 Warning: serving files without any filter!
2020/11/24 11:19:58 Serving . on 0.0.0.0:80/...

It’ll serve static content, from a file or provided on the command line:

D:\Projects\github>spark "<h1>Hello World!</h1>"
2020/11/24 11:22:17 Serving <h1>Hello World!</h1> on 0.0.0.0:8080/...

D:\Projects\github>spark offline.html
2020/11/24 11:25:52 Serving offline.html on 0.0.0.0:8080/...

It supports SSL certificates, and some very rudimentary security features to exclude specific paths when sharing a directory. It’s tiny, it’s fast, and it’s incredibly awesome.

In fact, the only problem with Spark is the name… when I went to put this post together, all I could remember about it is that it was a web server called Spark. It took me a good 10 minutes to find it again; it turns out there are quite a lot of web server related things called spark:

Yeah, I know, naming things is hard. But hey, one of the reasons I write posts like this is so that it really helps me remember stuff for next time. And if it doesn’t, I can always google my own blog, right?

Using HTTPS with Kestrel in .NET Core

2020-11-18T12:00:00+00:00

During 2020, I’ve been running a bunch of online workshops about building distributed systems with .NET Core. Over the course of the day, we build a REST API, hook it up to some microservice-style components using RabbitMQ and EasyNetQ, use gRPC to connect to a pricing engine, and then send the whole lot back into your web browser in real time using SignalR.

As it’s a hands-on workshop where attendees are writing their own clients and sending requests to my demo code, I need to run a real, public HTTP server as part of this - and because we’re hacking code as we go, I’m actually mapping an incoming port to my workstation here, which makes things a little interesting. If you’re running Kestrel in production, you’d normally stick it behind something like nginx, and terminate any HTTPS/SSL connections at the load balancer - but here I’m running it as a standalone .NET app, so I needed a way to use a real, properly signed SSL certificate for a local standalone Kestrel instance.

Here’s how I set it all up.

I’m using a free certificate from ZeroSSL - I’m hosting my example code here on workshop.ursatile.com, so I signed up with ZeroSSL and registered a free 90-day certificate for that hostname. When you click on “Install” here, you get the option to choose a Server Type - but there’s no mention of .NET Core, Kestrel or IIS in that list, so I went for “Default Format”:

This will download a ZIP file containing three files - certificate.crt, ca_bundle.crt and private.key.

That’s the certificate itself, the chain of certificates used to verify it, and the RSA private key that was used to sign the certificate. In order to use the cert with Kestrel, though, we need to convert it from a CRT format into something called a PFX file.

To convert it, I’m using OpenSSL and following the instructions documented here:

D:\workshop.ursatile.com>openssl pkcs12 -export -out certificate.pfx -inkey private.key -in certificate.crt

It’ll prompt you for an export password - if you’re using this for anything even remotely sensitive, it’s a good idea to specify a secure password here.

If you leave the export password blank, anybody who gets hold of the .PFX file can impersonate you
Do not check your PFX file into Github or publish it online anywhere

Remember - there’s no point securing your PFX file with a secure password if you then hardcode that password into your Program.cs and check it into Github along with the .pfx file itself.

Once you’ve got your certificate exported as certificate.pfx, we need to tell the Kestrel web server to use it.

If you’ve created your ASP.NET Core web app the usual way, you’ll find a method in Program.cs called CreateHostBuilder - this is where we can specify the options passed to the Kestrel server when it’s started up. We’ll need to add a call to the webBuilder.ConfigureKestrel() method like this:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder => {
            
            webBuilder.ConfigureKestrel(options => {               
                var port = 5001;
                var pfxFilePath = @"D:\workshop.ursatile.com\certificate.pfx";
                // The password you specified when exporting the PFX file using OpenSSL.
                // This would normally be stored in configuration or an environment variable;
                // I've hard-coded it here just to make it easier to see what's going on.
                var pfxPassword = "green cairo angle piano"; 

                options.Listen(IPAddress.Any, port, listenOptions => {
                    // Enable support for HTTP1 and HTTP2 (required if you want to host gRPC endpoints)
                    listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
                    // Configure Kestrel to use a certificate from a local .PFX file for hosting HTTPS
                    listenOptions.UseHttps(pfxFilePath, pfxPassword);
                });
            });

            webBuilder.UseStartup<Startup>();
        });

To check it’s all working, fire it up with dotnet run and then point a browser at http://localhost:5001 - you should see an error message just like this one:

but if you click “Advanced” and read the small print:

That’s actually exactly what we want. To run code locally using your new certificate, hack your hosts file and add a line to the end pointing workshop.ursatile.com or whatever domain you’re using at 127.0.0.1; if you want the rest of the world to play along too, you’ll need to register a DNS CNAME record pointing at your IP address and make sure that port 5001 (or whatever you used) is mapped to your workstation - oh, and remember to to disable the Windows firewall for that port.

But… is it safe?

Well, like most things in IT - it depends. If you do this, you’re effectively running a public web server in your house - and, yes, it’s definitely better doing this with a proper HTTPS certificate than just leaving it wide open. But it’s still not a great idea in terms of security.

When I do this for workshops and training, I open the ports in the morning, run it while I’m actually working with the group, then at the end of the day I shut the whole thing down and disable the port mappings - so if anybody did try to do anything malicious with it, I’d probably notice straight away, especially since I’m normally watching every single HTTP request because I’m curious to see how my workshop attendees are getting on. So as a teaching tool, I find it incredibly useful - but I’d never run an actual website from a development machine like this.

Whatever Happened to "Don't Be Evil"?

2020-09-01T12:00:00+00:00

How to make software be Not Evil:

Be transparent - show your user exactly what you’re doing.
Don’t use productivity tools as a marketing channel for your other business activities.
Don’t hijack your users’ day-to-day business activities to promote your own agenda.
Don’t email things on behalf of your users that they haven’t seen first.
Don’t do things that could make your users look foolish in front of customers and colleagues.

Not that hard, right? Like, that’s all stuff you’d have to do on purpose, by design - we’re not talking about bugs here, we’re talking about don’t deliberately put things into your software that will upset your users.

Well, I found a feature in Google Calendar today that scores zero out of five on this scale. If you create a new appointment in GCal, add a guest, and click Save, it will:

Add a Google Meet meeting URL to your appointment. Unprompted.
Do this in the background, in the shaded area behind a modal dialog box
Include this URL in the invitations that’ll get sent to your meeting attendees.

Go on. Look.

(yes, I know the video is on YouTube. Yes, I know YouTube is owned by Google. No, I don’t believe for one second that Google is going to change anything because I wrote a blog post. But maybe reading this might inspire one of you to Not Be Evil when you’re doing your own stuff, and perhaps that’s enough.)

You want to have a meeting on Webex or Zoom or Jitsi? Go for it. But Google’s gonna add a Google Meet URL as well.

Of course you can switch this off. There’s a settings flag for it. But this “feature” was rolled out at some point this year, with users opted-in by default - I certainly never switched it on, because I don’t use Google Meet and I positively do not want a second URL added to the invitations to all my online meetings and calls. I generally don’t like software doing things I haven’t asked for, and I really don’t like software doing things I haven’t asked for and them sending it in an email straight away without giving me a chance to see it first.

But hey. “Don’t Be Evil” was a long time ago. I guess things have moved on a bit since then.

Where's the YouTube Video?

2020-08-26T12:00:00+00:00

I’ve spoken at a few meetups recently where people have asked later where the YouTube video is. Well, folks, here’s the deal. I’m asking for the time being that community events and meetups don’t publish recorded videos of talks that I’m giving remotely - and here’s why.

First: I want people to actually attend these events, live - and so do the organisers. We want an audience, we want questions, we want people hanging around afterwards for a drink and a Zoom chat. The social contact is part of the experience. You don’t get that watching a video afterwards.

Second: as much as I’m enjoying being part of these online events, I’ll be the first to admit that they are very different experience to speaking at a live, in-person event. Even if it’s the same talk; the venue, audience, camera angles, all that comes through in a recorded video. I’ve always been happy having a camera in the room when I’m speaking and sharing the video afterwards, because it’s obvious that watching the video afterwards is like watching sports on TV. It’s obvious that it’s a recording of a real thing that actually happened, and that the folks who were in the room at the time had a very different experience to the one you’re having watching it on YouTube - and, let’s face it, you’re probably watching it at 1.5x normal speed on your iPad while you’re doing emails or writing some JavaScript. Definitely not the same experience.

But with online events, the boundary between “live” and “recorded” gets a bit.. fuzzy. When I speak at remote events, it’s always me, in the same room, doing the same talk, to the same camera. Sure, I change things around to suit the event, wear a different shirt, add slides, take questions from the audience, those kind of things - but fundamentally, it’s the same thing. And I don’t see a huge amount of value in having five different versions of the same talk on five different YouTube channels when all of them are just me doing the same talk to the same camera in the same room. Audience Q&A and live chat don’t come through terribly well on these recordings, either.

Finally: if I’m going to stand in my home studio for an hour, mic’ed up, lights on, talking to a high-definition camera, it seems a little silly that the recording of that which ends up getting published is an unedited single take recorded over the wrong end of a YouTube stream, y’know?

If you’re paying me to speak? Different story. You give me money, you get to make the rules. That’s different. We’re all trying to figure out a whole new revenue model for events and conferences right now, and if you have an audience who want to pay for my talks, I’m more than happy to work with you on producing something that works for everybody.

But for free meetups and community events - I’ll join you live, I’ll give you the best talk I possibly can, I’ll stick around and answer questions and chat… but after that, it’s over. Missed it? That’s OK. There will be another one - and hey, it’s not like the travel will be a problem. 😉

How I Write Timed Talks

2020-07-09T12:00:00+00:00

This is one of those blog posts that started life as a tweet, and then a tweet thread, and quickly grew to a point where it actually makes more sense as a post. It started out with this:

Folks who give lots of talks, and create new *timed* talks from scratch, do you follow a specific process/use a template etc?

Trying to learn how to get better at this. 🙃
— C:\hristina 👩🏽‍💻 (@divinetechygirl) July 9, 2020

So, here’s how I do it.

First: figure out how fast you talk. In my case, I took some videos of a couple of talks I’ve given that went well, and transcribed them - literally word for word, number for number. Every single word. Then I measured the word and character counts compared to the length of the videos. I talk at almost exactly 1,000 characters per minute - which works out around 178 words per minute, give or take.

That’s useful, because it means I can easily turn a time limit into a word limit. A 5-minute lightning talk needs about 5,000 characters worth of material - just under 1,000 words. A one-hour keynote talk needs roughly 60,000 characters.

Second: learn how to write the way you speak. This can take a while, but it’s a really useful skill to develop. When I write talks, I write them exactly the way I talk when I’m talking - jokes, contractions, I spell out numbers long form (262,144 is only six characters on a word count, but “two hundred and sixty two thousand, one hundred and forty four” takes about four seconds to say out loud).

I’m not writing a script - what I actually say on the day probably correlates about 50% with what I wrote - but they include all the important bits. The key statistics and figures, the important points (and how I want to phrase them), the punchlines to the jokes. Think of it like the score for a jazz standard - the intro, the melody, the refrain and the ending are all pretty clearly mapped out, but they leave structured spaces for improvisation.

Sometimes, at this point, I just start writing. Not necessarily at the beginning - I’ll often start writing in the middle, I’ll put in placeholders and headings, I’ll move things around. If I know I’m using prerecorded video clips during the talk, I’ll check the durations of these and adjust the word limit accordingly. Other times, I’ll map out the talk into chunks:

Intro: 5 minutes (5,000 characters / 950 words)
History of radio, signal processing: 15 minutes (15,000 characters / 2,880 words)
Digital cameras & JPEG compression: 15 minutes (15,000 characters / 2,880 words)
Unicode, text encoding: 10 minutes (10,000 characters / 1,920 words)
Conclusions and wrap up: 5 minutes (5,000 characters / 950 words)

I’ll keep writing and editing and tweaking until I have something that’s about the right length. Then I’ll record myself reading it out loud. This is normally where a bunch of things jump out that sounded good on paper… so I’ll tweak and edit some more.

Then I’ll go through the whole thing and identify the points where I need a slide. Sometimes I’ll talk for 2-3 minutes around a single image; sometimes I’ll literally have a slide or animation for each word in a sentence. Sometimes a line or two becomes the storyboard for a piece of animation I’ll create.

Then I’ll fire up an empty Powerpoint presentation, and for each chunk of prose that accompanies a slide, I’ll paste that chunk of text onto the Notes area on the slide and put in a one-line placeholder (“PICTURE OF ELEPHANT HERE”, “ANIMATION ABOUT 16QAM HERE”) as the slide heading. That’ll let me start using Powerpoint’s outline view to navigate around. Finally, I’ll go through each slide and add the relevant images, text, titles, video clips - whatever’s needed to support that part of the talk.

For each slide, I’ll also add a few bullet points in the speaker notes summarizing what I need to say - and what I need to end with for the transition to the next slide to make sense.

Then I’ll do a “dress rehearsal” - run through the whole thing, check the timings, transitions, animations, make sure my own speaker notes make sense - and I’m good to go.

Using <a href="https://agendadefender.app/>AgendaDefender.app</a> to keep track of time during a talk

When I’m actually giving a talk, I’ll use a tool I built called Agenda Defender to stay on track with timing - I’ll put in the actual start times for each section of the talk, and Agenda Defender draws live animated progress bars for each section of the talk so I can tell whether I need to speed up a bit, or have time for a bit of audience Q&A, maybe throw in a few jokes or stories that weren’t in the original talk outline.

That’s pretty much it. A couple of things to add:

The first time I deliver a new talk, I normally don’t refer to the notes at all - it’s all fresh in my mind. But the notes are invaluable if you’re invited to give the same talk again six months later and need to refresh your memory.
Breaking down the talk structure into timed sections makes it almost “modular” - if you need to adapt a 45-minute talk to fit a 30-minute time slot, sometimes there’s literally a chunk you can just cut out and tweak the rest a bit to make it flow across the gap.
When I first started out, I used the 2-minutes-per-slide approach - for a 60 minute talk, I need to make 30 slides, then for each slide, come up with 2 minutes of content for it. I worked like that for a year or two and absolutely hated it.
A one-hour talk is about 10,000 words. That’s the same length as an undergraduate dissertation at many UK universities. Preparing talks this way is a lot of work - I normally reckon on about 30-50 hours of prep time from the first rough outline to the having the talk “stage ready.”

And for the sake of comparison, this blog post is 1,156 words - it’d make a pretty good 5-minute lightning talk with a bit of trimming - and it’s taken me just over 45 minutes to write it. Maybe I should just have replied on Twitter after all… :)

Workshop: Introduction to Distributed Systems with .NET

2020-07-07T12:00:00+00:00

Next Thursday, July 16th, I’m running a workshop with the folks from Fusion Workshops in Birmingham, all about building distributed systems using .NET Core. It’ll be completely online, hands-on, delivered using Zoom and various online tools - plus a lot of live coding, hooking our code into various cloud services and watching things running live over the internet. I’ve run this workshop with a couple of different groups and the online format actually works brilliantly - plus it’s way more impressive running a distributed application when you’re sending messages a few hundred miles between cities, instead of between two laptops in a classroom.

Tickets are on sale now: https://www.eventbrite.com/e/distributed-systems-net-with-dylan-beattie-tickets-107811475024

What’s a distributed system?

Good question. Fundamentally, it’s a software system where you have multiple components that communicate across a network. After that, it can get very complicated, very quickly… before long you’re reading about microservices and REST APIs, GraphQL, gRPC, message queues, pub-sub, “backends for frontends” - there’s a lot of good ideas, and some very powerful patterns out there, but it can all be a bit overwhelming if you’re not sure where to start.

What’s in the workshop?

This is very much a hands-on workshop - lots of code, lots of demos, and by the end of the day you’ll have working examples on your own PC of most of the things we’re looking at. Using a (very simplified!) used car sales platform as our example, we’ll kick off by building a simple REST API (and yes, it will actually be RESTful - with HATEOAS and everything!) that dealers can use to add new cars to our platform. Then we’ll plug that into a set of message queues, using EasyNetQ, and build some microservices that receive messages from those queues. We’ll plug in some gRPC endpoints to handle price calculation, and finally hook the whole lot into a SignalR hub to display live notifications in a web browser.

Along the way, we’ll discuss the pros and cons of each of these models, look at why and when you’d used them in production, and where you can go to find out more about running them on production systems.

The best part is that, because we’re all remote, the code we write during the day will be an actual real-world distributed system - we’ll be bouncing messages and HTTP requests around the country, triggering workflows on each other’s laptops, seeing messages from other participants being filtered through our own code, and get a chance to see how things like latency and bandwidth affect the reliability and performance of real-world distributed systems.

You’re not going to learn everything in a day. We’re not going to talk about security, monitoring, logging, deployment… the idea of this course is to give you an overview of how everything fits together, and a solid foundation to go away and learn more (or take another course!) before putting these ideas into production.

Who’s it aimed at?

Developers who have done at least some .NET, who want to understand more about the different patterns and architectural styles used to create distributed applications, and the frameworks that exist in .NET Core to help you build those kinds of applications.

The course assumes some familiarity with .NET - if you’ve built websites with ASP.NET MVC or done any kind of .NET development, you should be fine.

Using Canon DSLR Cameras as Webcams

2020-07-05T12:00:00+00:00

I love Canon DSLR cameras; my main DSLR at the moment is an EOS 700D (which is sold as the Rebel T5i outside the UK, for reasons I will never, ever understand), which for me hits the perfect sweet spot between affordability and quality. And, with the whole lockdown thing going on, I’ve been spending a lot of time looking for ways to use gear I already own to produce better live streams and studio recordings.

If you’ve read Scott Hanselman’s post from August last year about remote working, you’ll know that the best webcams aren’t actually webcams at all - they’re proper cameras, running through some kind of HDMI > USB converter like the Elgato Camlink. I’ve been trying to pick up one of these for a while, but they’ve been out of stock everywhere since March, so instead I managed to track down an Elgato Game Capture HD60 S, which does kinda the same thing.

Well, turns out my beloved Canon EOS 700D doesn’t quite work as a webcam. Let’s clarify that. Out of the box, it doesn’t work at all - there’s no way to produce what’s called “clean HDMI” - i.e. a video stream with no overlays, menus, focus grids, or anything.

But, it turns out there’s an open source firmware upgrade for Canon DSLR cameras called Magic Lantern. Now, this is some seriously gnarly stuff - if you’re not happy downloading binaries from a website and then running them through the firmware upgrade routine on your DSLR camera, then it is probably not for you, and it is absolutely possible to brick your camera by fooling around with this kind of thing. But hey, nobody ever found fortune and glory by following the terms and conditions, right?

Magic Lantern actually worked really well for me. It allows me to switch off all the overlays and focus grids to get a clean HDMI signal out of the EOS 700D - and with a 50mm f1.8 lens, it looks fantastic. But it’s not perfect. I used it for a live streamed gig last night, and it worked beautifully for about the first 45 minutes… then the shutter cycled (no idea why), and when it came back after about a second, there were some weird letterboxing artefacts around the video frame. Finally, after about 75 minutes of streaming, it stopped autofocusing completely. All temporary - turning it all off and on again got it all back to normal - but not ideal for a live broadcast.

Working beautifully...

...then some weird letterboxing...

...and finally the focus grid appeared in shot, and wouldn't go away. (And then the autofocus stopped working.)

Of course, having seen just how good it looks when it’s working, I’m now in the market for something that looks that good but doesn’t suffer from the same reliability issues. Just so we’re perfectly clear: this is absolutely not any kind of problem with the EOS 700D. I’m trying to use it for things it was never, ever designed to do, and the fact it worked at all is pretty remarkable!

So, there’s two options. One is to get a DSLR that is known to produce good clean HDMI for indefinite periods - there’s a list of these over on Elgato’s website.

The other is that a few months ago, Canon’s engineers released a beta of a USB utility that would allow you to use certain Canon DSLR and compact cameras as webcams directly, via the camera’s onboard USB interface. This is officially only supported in the United States, which isn’t a big deal - except that, for some ridiculous reason, many Canon DSLR cameras are sold under different names in the UK than they are in the US, and the list of supported models refers only to the US model names.

So I’ve spent a couple of hours pulling together the list of supported cameras, adding the UK model names, cross-referencing it with the Elgato camera checklist, and, just for good measure, adding in the Canon recommended retail price and a link to Amazon for those cameras that are currently available.

Here’s what I found:

Camera Model (US)	Camera Type	UK Model Name	RRP	Amazon	Elgato CamLink / clean HDMI?
EOS-1D X Mark II	DSLR	same as US	£5,429		Yes
EOS-1D X Mark III	DSLR	same as US	£6,499		Yes
EOS 5D Mark IV	DSLR	same as US	£2,799	£2,799	Yes
EOS 5DS	DSLR	same as US	£2,499		Not listed
EOS 5DS R	DSLR	same as US	£2,699		Not listed
EOS 6D Mark II	DSLR	same as US	£1,349	£1,349	No
EOS 7D Mark II	DSLR	same as US	£1,429		Yes (manual focus only)
EOS 77D	DSLR	same as US	discontinued		Yes (manual focus only)
EOS 80D	DSLR	same as US	discontinued		Yes (manual focus only)
EOS 90D	DSLR	EOS 90D	£1,209		Yes
EOS Rebel SL2	DSLR	EOS 200D	£369.99		Yes (manual focus only)
EOS Rebel SL3	DSLR	EOS 250D	£529.99	£539.00	Yes
EOS Rebel T6	DSLR	EOS 1300D	discontinued		not listed
EOS Rebel T6i	DSLR	EOS 750D	unknown		Yes (community verified)
EOS Rebel T7	DSLR	EOS 2000D	£329.99	£299.00	not listed
EOS Rebel T7i	DSLR	EOS 800D	£739.99		Yes
EOS Rebel T100	DSLR	EOS 4000D	£279.99
EOS M6 Mark II	Mirrorless	same	£869.99
EOS M50	Mirrorless	same	£539.99
EOS M200	Mirrorless	EOS M200	£549.99
EOS R	Mirrorless	EOS R	£1,879.99
EOS RP	Mirrorless	EOS RP	£1,219.99		Yes
PowerShot G5X Mark II	Compact	PowerShot G5X Mark II	£869.99		Yes
PowerShot G7X Mark III	Compact	PowerShot G7X Mark III	£719.99		Yes
PowerShot SX70 HS	Compact	PowerShot SX70 HS	£539.99

Camera Model (US)

Camera Type

UK Model Name

RRP

Amazon

Elgato CamLink / clean HDMI?

EOS-1D X Mark II

DSLR

same as US

£5,429

Yes

EOS-1D X Mark III

DSLR

same as US

£6,499

Yes

EOS 5D Mark IV

DSLR

same as US

£2,799

Yes

EOS 5DS

DSLR

same as US

£2,499

Not listed

EOS 5DS R

DSLR

same as US

£2,699

Not listed

DSLR

£1,349

DSLR

£1,429

Yes (manual focus only)

EOS 77D

DSLR

same as US

discontinued

Yes (manual focus only)

EOS 80D

DSLR

same as US

discontinued

Yes (manual focus only)

EOS 90D

DSLR

EOS 90D

£1,209

Yes

EOS Rebel SL2

DSLR

EOS 200D

£369.99

Yes (manual focus only)

EOS Rebel SL3

DSLR

EOS 250D

£529.99

£539.00

Yes

EOS Rebel T6

DSLR

EOS 1300D

discontinued

not listed

EOS Rebel T6i

DSLR

EOS 750D

unknown

Yes (community verified)

EOS Rebel T7

DSLR

EOS 2000D

£329.99

£299.00

not listed

EOS Rebel T7i

DSLR

EOS 800D

£739.99

Yes

DSLR

£279.99

Mirrorless

£869.99

Mirrorless

£539.99

Mirrorless

£549.99

Mirrorless

EOS R

£1,879.99

EOS RP

Mirrorless

EOS RP

£1,219.99

Yes

PowerShot G5X Mark II

Compact

PowerShot G5X Mark II

£869.99

Yes

PowerShot G7X Mark III

Compact

PowerShot G7X Mark III

£719.99

Yes

PowerShot SX70 HS

Compact

PowerShot SX70 HS

£539.99

I was hoping to find a new camera body for around £300 that will work with my existing lenses, supports the new Canon EOS Utility software, and is verified for use with the Elgato HDMI capture devices; there’s no clear winner, but the EOS 2000D and EOS 4000D aren’t listed either way by Elgato so I will do some digging, see what I can find out. I’ll keep you posted. ;)

Oh, and if anybody has any idea at all why Canon decided to give their cameras different names in the UK and the US, I’d love to know…

Stay In, Rock Out: I'm Playing Live Online!

2020-07-02T12:00:00+00:00

On Saturday, pubs in England are allowed to reopen. I love pubs. I love pubs more than you can imagine. But there is no way in a million years I am going anywhere near a pub on a Saturday in the middle of summer, when they’ve all been closed for three months, and they’re only opening now because the government think there’s probably enough hospital beds to deal with the aftermath.

So instead of going to the pub on Saturday, I’m going to play a live rock show, from my house, and stream the whole thing live online. An hour or so of classic rock covers - AC/DC, Bon Jovi, Queen, Muse, Pink Floyd, Guns n’ Roses, and maybe a few surprises. I’ll be using a whole bunch of technology to put together something a bit special - if it works, it’ll be awesome… if it doesn’t work, it’ll be probably be hilarious. Either way it’d be cool if you want to tune in.

Here’s the invitation so you can add it to your calendar:

dylan-beattie-stay-in-rock-out.ics

It all kicks off at 19:00 UK time this Saturday, July 4th. I’ll be streaming live on https://twitch.tv/dylanbeattie, and on facebook.com/dylanbeattie and youtube.com/dylanbeattie as well - and then firing up a Zoom for the after-party.

Get some beers in, crank up the volume, and let’s have some fun. :)

Locking Down & Levelling Up, Part 1: New Computer Time

2020-05-28T12:00:00+00:00

One of the nice things about this whole weird lockdown situation is that for the first time in a long, long while, I know exactly what my travel plans are going to be for the next few months: none. Zip. Zilch. Nada. The likelihood of being asked to throw some things in a bag and jump on a plane tomorrow is… zero. Which kinda sucks, because I love travelling, but, hey – here I am. I’m at home, it looks like I’m gonna be here for a long while, and that means I have time, and incentive, to turn my home office into an ideal workspace.

Now, I have a bit of a head start here, because I do actually have a dedicated room in my house for “work”. Which means it was already a combination of office, recording studio, library and very occasional spare room, but when the lockdown hit and so many conferences and events started going online, I really wanted a proper setup for livestreaming and recording video, complete with greenscreens, studio lightning, all that kind of stuff. (Yes, buying hardware is a coping strategy for me. This is OK.)

Plus, with everything going online, I kept getting invited to join Slack workspaces. Compiling the .NET Framework from source while running Adobe Premiere and Far Cry simultaneously and livestreaming the whole lot to Twitch in 1080p in real time is one thing, but being in 15 Slack workspaces at the same time requires some serious processing power…

It’s been a long while since I’ve had a proper workstation. Like most nerds of a certain age, I spent the 1990s and 2000s constantly tinkering with big PCs, adding RAM, upgrading CPUs and graphics cards - y’all remember 3Dfx and the Voodoo chipset? Some time back in 2015 the graphics card in my last one failed. I just copied everything across to my laptop and used that as my main machine until I got it fixed… and then never got it fixed. So step 1 of this process was to get a proper computer. I wanted to switch back to running Windows 10 as my main day-to-day operating system (and keep macOS on the laptop for music stuff); I wanted something that could drive a stupid number of external monitors and peripherals, and I wanted something that was blazing fast and whisper quiet.

(Did I mention the bit about how buying hardware is a coping strategy? Cool. Glad we’ve cleared that up.)

Enter the lovely people at QuietPC. A few hours poking around their website, and one very, very helpful phone call, and I had my ideal machine spec’ed out. After having a few too many recordings ruined by the sound of the fans on my Macbook firing up halfway through a take, I wanted something completely fanless. I got pretty close: the system I ended up with has a passive-cooled power supply and CPU cooler, I ended up adding a single 140mm BeQuiet case fan, and there’s a fan on the GPU that only starts up when you drive it hard. Most of the time it’s completely silent.

Here’s the spec I went with:

Case: Streacom DA2 Black Full Aluminium Compact ITX Chassis
PSU: Silverstone Nightjar 450W Fanless Modular SFX Power Supply, NJ450-SXL
CPU: 9th Gen Core i7 9700 3.0GHz 8C/8T 65W 12MB Coffee Lake CPU
RAM: Vengeance LPX 32GB (2x16GB) DDR4 2666MHz Memory
Motherboard: ROG STRIX B360-I GAMING LGA1151 Mini-ITX Motherboard
CPU Cooler: CR-80EH Copper IcePipe 80W Fanless CPU Cooler
HDD: 2 x Samsung 970 EVO PLUS 500GB M.2 NVMe SSD, MZ-V7S500BW
Graphics card: ASUS GeForce GTX 1660 Ti DUAL OC Edition 6GB EVO Graphics Card

I need to take a moment here to mention how awesome the folks at QuietPC are, because when I first placed the order, I picked a different CPU cooler – and they actually rang me up, explained that the cooler I’d ordered wouldn’t fit in the case I’d ordered, offered an alternative, and sorted the whole thing out. That’s some absolutely first-class customer service. And sure enough, a few days later, a large pile of exciting cardboard boxes arrived.

A large pile of exciting cardboard boxes.

Building computers is great fun. It’s like the best Lego ever, and this one was no exception. Getting everything to fit on this system was a tiny, tiny bit fiddly… the Streacom DA2 chassis is a wonderful piece of engineering, based around an open chassis with mounting rails you can move to wherever you need them, but there’s really not a whole lot of space inside. It all fit in the end – mainly thanks to the M.2 hard drives, which aren’t really drives at all, they’re SSD chips on their own circuit boards that mount directly onto the motherboard. I also had no idea just how big the fanless CPU cooler would actually be.

The CR-80EH Copper IcePipe 80W Fanless CPU Cooler is... not small

I got it all put together, fired it up, installed Windows 10, everything ran beautifully - and absolutely silent. Like, ZERO MOVING PARTS silent. This worked great until I got Visual Studio and Adobe AfterEffects installed and starting putting it through its paces, at which point it crashed hard… turns out that a completely fanless PC is ever so slightly prone to overheating.

Oops.

So I compromised, and ordered one very big, very quiet fan – it doesn’t take a whole lot of airflow to keep things cool, and there’s an ASUS utility called Fan Xpert that I used to create a fan profile, so it’ll monitor the CPU temperature and automatically spin up the fan when required.

Creating a fan control profile using Asus Fan Xpert

That seems to have done the trick – it’s been running stable for 6 weeks now; I’ve been using the rather good Open Hardware Monitor (which, by the way, is written in C# and .NET!) to keep track of all the various temperatures and things, and haven’t had any more crashes or CPU temperature errors.

Temperature readings over a typical 24 hour cycle. The CPU is the blue one.

In the next post, I’ll talk you through how – and why – I’m using it to drive six monitors, four cameras and four microphones, and how I ended up building a bespoke computer desk to put it all on.

Running Jekyll on WSL2

2020-05-19T12:00:00+00:00

Back in August last year, I migrated this site, including 12 years worth of blog posts, from Blogger to Jekyll, primary so that I could host it on Github Pages. At the time, my main system was a Macbook Pro running macOS, I was doing a lot of work with Ruby and had a full Ruby dev environment set up, so running Jekyll locally was trivial.

I’ve recently switched back to Windows as my main OS, and Windows is still very much a second-class citizen in the wonderful world of Ruby web development. It’s a lot better than it used to be, and almost everything now works out of the box, but because the vast majority of Ruby devs (and sites) are running on macOS or Linux, Windows still isn’t quite as slick, or as fast, when it comes to running tools like Jekyll.

But this is 2020, and that doesn’t matter, because you can run Linux on Windows! Thanks to the Windows Subsystem for Linux (WSL), you can boot a full Linux kernel environment on your Windows PC and run native Linux apps and toolchains. I’ve been running Jekyll under WSL 1 for a long while to run my sites locally, and it works great, but it’s a little slow; spinning up a local copy of this site would take around 30 seconds, compared to 10–12 seconds on my old Macbook Pro.

I’d heard good things about performance improvements in WSL2, so last night I took the plunge, enabled the Windows Insider Program on my main PC, and ran a rather hefty Windows Update.

Now, by way of establishing a baseline, here’s running bundle exec jekyll serve on WSL 1. This is what I’ve been using to run my site locally and test things for the past few months.

30 seconds to build the site from scratch. Not great, not terrible.

This is on a the Ubuntu-20.04 LTS Linux kernel that’s available from the Windows Store. Having upgraded Windows 10 to the 2004 build, I could now upgrade this in-situ to use WSL2 instead of WSL1:

C:\Users\dylan>wsl -l -v
  NAME            STATE           VERSION
* Ubuntu-20.04    Running         1

C:\Users\dylan>wsl --set-version Ubuntu-20.04 2
Conversion in progress, this may take a few minutes...
For information on key differences with WSL 2 please visit https://aka.ms/wsl2
Conversion complete.

C:\Users\dylan>

Ooh. Shiny. OK, let’s see what this thing can do:

155 seconds. Gosh. That’s… not quite what I was expecting, and certainly not something I’d claim was a “performance improvement”. So I did a little digging, and found this article from scotch.io, which includes a “Gotchas with WSL 2” section that specifically says:

To take advantage of all the new speed improvements in WSL 2, our files will need to be moved into the Linux filesystem.

Until now, I’d just kept everything on my Windows D: drive and used /mnt/d/ under WSL to run everything. Apparently that’s not a good idea any more… so I did a fresh git clone of my blog code into ~/dylan/github/dylanbeattie.net/ under WSL2 and ran it again:

TWO POINT ONE TWO NINE SECONDS! HOLY BATFISH CATMAN! THAT’S FAST!

For the sake of comparison, running the same thing natively on Windows, using the Windows distribution of Ruby 2.6, takes about 21 seconds:

So, running Jekyll on the Linux native filesystem is an order of magnitude faster than the same thing running natively on Windows, and around two orders of magnitude faster than running the same thing on WSL with a mounted Windows filesystem. That’s quite an astonishing difference.

Of course, this means that all my code now has to reside on the Linux filesystem, so it isn’t quite so easy to get to it from Windows any more. But the Linux filesystem is shared via a network path, so I was able to go:

C:\Users\dylan>net use u: \\wsl$\Ubuntu-20.04\
C:\Users\dylan>u:
U:\home\dylan>cd github\dylanbeattie.net\
U:\home\dylan\github\dylanbeattie.net>

And having got this far, I couldn’t resist running two more tests just to round the whole thing out. First, the same thing but on a full Ubuntu 20.04 desktop VM running under HyperV:

That’s 2.45 seconds – marginally slower than the WSL2 version with the native FS. And finally, just for the hell of it, here’s the Windows native port of Ruby and Jekyll, against the repo checked out onto the Linux native FS, mounted as a network drive in Windows:

618.84 seconds… just a tiny bit slower than the WSL2 native Linux result, wouldn’t you agree?

Here’s how the six different scenarios end up:

Conclusions:

I was genuinely surprised at the orders of magnitude of difference seen here. I expected a factor of maybe 2-3 times – I did not expect the slowest (albeit rather silly) option to be nearly 300 times slower than the fastest. But clearly whatever the WSL team has done in WSL2 has made a substantial difference when it comes to performance – as long as you’re working with a native Linux filesystem.

If your base operating system is Windows, running WSL2 with the Linux filesystem is actually faster than any of the other options – including running Linux in a full VM.
If you need Windows/Linux interop across the same filesystem, you’re probably better off sticking with WSL1. The Linux kernel in the WSL2 system is significantly slower when dealing with files stored on the Windows filesystem than it was in WSL1.
For simple Jekyll sites, using the Windows native versions of Ruby and Jekyll is probably fine.
If performance is a big deal, go for WSL2 and clone your repo onto the Linux FS

Usual caveats apply, that one guy running one Jekyll site and writing a blog post about it is not a rigorous scientific analysis. I did this because I wanted to work out what solution worked best for my specific scenario. Your mileage, as the saying goes, may vary. But it’s definitely worth trying out a few different options and seeing what works best for your particular workload - and hey, when you do, publish the results. I’m curious to see just how good WSL2 is across different scenarios, but early indicators are it’s very, very good indeed.

Piedroit: A Raspberry Pi-powered USB Footpedal (Part 2)

2020-05-18T12:00:00+00:00

In part 1 of this post, I described the software and config required to use a Raspberry Pi Zero W to create a bank of footswitches that the host computer thinks is a USB keyboard. So far, so good: a little Python hacking, some Linux modules, a bit of GPIO and USB HID programming. All nice and friendly and revision-controlled.

That’s only half the problem, though. The main reason I’m building this thing is that I want a way to control the computer by stamping on things. When I’m playing the guitar and doing live shows, on stage or streaming via Twitch or YouTube, I want to be able to switch shots and control backing tracks by pressing switches with my feet. Now, I’m not exactly sure what would happen if 102kg of slightly excitable developer stood on a Raspberry Pi Zero W – or a normal computer keyboard, for that matter – but I suspect if it happened regularly, the devices in question might start complaining a bit.

So having solved the logical elements of this puzzle, let’s figure out the physical bits. How do you connect a Pi Zero to a bank of footswitches that can cope with a big hairy guy pressing them with a pair of size 10 boots?

Well, as with many problems in life, the answer is… metal. In this case, 0.6mm bright mild steel sheet, because that’s what I could get hold of. I don’t have access to a proper machine shop or any welding gear (#lifegoals), so I had to come up with a design that I could make using the tools and materials I had available, but which would protect the Pi and other bits of delicate circuitry within.

The switches I’m using for this project are a brand called Mintice, available from Amazon for £11.79 for a pack of 5. They’re pretty easy to work with, other than requiring you to drill a 12mm diameter hole… making holes in sheet steel is easy, especially if you have a drill press… I don’t have a drill press. I’m working with a cordless hand drill here, so making sure the holes end up exactly where they’re supposed to be is a bit tricky.

The Cardboard Prototype

Just like software, you should always plan to build one version to throw away. Unlike in software, when it comes to metalwork it’s normally really obvious which one’s the prototype… it’s the one made out of ~~JavaScript~~ cardboard. The prototype I’m building here is to answer a couple of key questions:

Will all the components actually fit?
How much sheet stock will I need, and what shapes will I need to cut?
Is there enough space between the switches that I won’t accidentally press two switches at once?

Sketching layouts and checking clearance for the components

Cardboard prototype for the top part of the housing

Testing the footswitches in the cardboard housing mockup

The Pi Zero board and the footswitches installed in the prototype housing to check clearance

Time to Get Metal

The cardboard prototype worked. The components all fit, nothing’s fouling anything else, and it’s small enough to make from a single sheet of the 250mm x 400mm steel sheet stock I had available, but big enough to install five switches on the unit with enough distance between them that I’m not going to hit two switches at once.

Now I just need to make the same thing again, except in steel instead of cardboard. That’s not actually as daunting as it sounds. I really enjoy working with steel. Cardboard’s easy to cut and fold, sure, but it also frays and has a tendency to unfold. Steel takes time (and the proper tools) to cut and shape, but once you fold it, it stays folded.

First step was to cut a piece of steel for the top of the housing, and then drill the five mounting holes for the switches:

Mounting holes for the foot switches

Next step was to bend the top part of the housing. There’s dozens of videos on YouTube about how to bend sheet metal – some show you how to use a machine called a metal brake; some show you how to make a metal brake if you don’t have one available, and some show you how to bend metal without a brake. This video from Cosador is great if you want an overview of a few different techniques. I ended up using hand bends and hammered bends for most of my project, using a couple of lengths of cold-rolled steel angle and lots of clamps to hold everything straight. It came out OK – not perfect, but good enough for what I need.

Improvised metal brake made by clamping angle and square section steel onto my kitchen table.

The main bends for the top part of the casing done.

Test fitting the foot switches into the bent metal casing.

For the bottom half of the casing, I needed to bend up the edges of the case, but also to cut notches for the HDMI and USB ports on the Pi Zero.

Rough-cutting the bottom casing panel using a jigsaw

Work in progress filing out the notches for the Pi IO connectors

Testing fit of the Pi Zero IO ports

The Pi board itself was mounted on a small piece of 6mm MDF – you can see the original sheet in this photo, that I’m using to hold the Pi in place to check the fit of the IO ports.

One more thing I did was to use a needle file to enlarge the mounting holes on the Pi so I could use 3mm screws. Out of the box, the Pi uses 2.5mm mounting screws. I don’t have any of those lying around – but I have lots of 3mm screws and bolts, so slightly enlarging the holes gave me way more options when it came to mounting the board.

Wiring it up

To wire the switches to the Pi’s GPIO ports, I cut one end off a 40-pin ribbon cable, giving me a wiring harness with about 15cm of wire on each pin. The fun part here was figuring out which wire connects to which pin. I used a Raspberry Pi breakout board for this part, and it took about an hour of testing connections with a multimeter and labelling each wire as I identified which pin it connected to. Fiddly, but not actually difficult.

Tracing the individual cables in the 40-pin ribbon cable back to the source pins

For this project, I wired GPIO pins 04-08 to switches 1-5. At this stage I also wired in two extra breakout ports – one’s a mono 6.5mm jack socket, the other is a stereo 6.5mm jack socket. I wired these into GPIO pins 9 and 10+11, so I could hook up three extra external footswitches to give me another three inputs on the board.

Here’s both parts of the casing, with all five switches and the two 6.5mm breakout ports all wired in and the ribbon cable seated on the Pi’s GPIO connector. You can see the 6 holes I drilled in the case here – the original plan was to tap M3 threads into the top half of the casing and assemble the case using M3 bolts. Turns out 0.6mm mild steel isn’t actually hard enough to hold a thread… two of them came out OK, one of them I stripped completely and ended up bodging by supergluing a nut in place above the hole, the other three I’m just going to ignore for now. You can also see the Pi board here mounted on a piece of 6mm MDF, which is attached to the steel using 3M Command Strips – again, that’s what I had easily to hand. They’re normally used to hold pictures on the wall, but they adhere to steel and to MDF pretty well and they’re easily removable if necessary.

The Piedroit wired up and ready for final assembly

And – that’s it! Put the whole thing together, boot it up, and… it works. Windows detects each footswitch as a key on a USB keyboard, and now I can control WinAmp, OBS, and any other application that supports global hotkeys by pressing footswitches.

Ideas for vNext

This was a great project. Frustrating in places, especially the Bluetooth dead end, but I had a lot of fun putting it together. At a time when so much of what we’re all doing – work, social life, entertainment – is all taking place on screen, it’s also a nice change to build something physical; something you can actually pick up and touch and handle.

But, of course, I have loads of ideas for building a better version.

The circuitry. The Pi is a great little computer, but… do I really need 512Mb of RAM in a footswitch? I’m also still keen to see what I can do with Bluetooth, so I suspect the next incarnation of the project will use either something like an Arduino or a Teensy, or I’ll cannibalise a Bluetooth keyboard and hack the circuitry from that into a footpedal controller.
Status LEDs. There’s currently no way of telling whether the board has actually booted up properly. It wouldn’t be that hard to wire up a couple of LEDs to the GPIO pins – something as simple as a red LED for power and a green LED that lights up once the USB keyboard event loop is running would make the device a lot friendlier.

And, of course, there’s all the “it would be cool if…” – reprogramming key mappings, macros, modifier keys… once you get into these kinds of projects, the hardest part is knowing when to stop.

But this one is done. I’ll be using it for a bunch of events I’m doing this week, including HalfStack Online on Friday, and I’m sure you’ll be able to catch it in action on a YouTube or Twitch stream near you before too long!

Piedroit: A Raspberry Pi-powered USB Footpedal (Part 1)

2020-05-17T12:00:00+00:00

I often find myself doing daft things with computers and playing the guitar at the same time. I use laptops on stage to run videos and backing tracks, I record my own music, and, most recently, I’ve been livestreaming music over YouTube and Twitch for some of the online conferences I’ve spoken at.

Since playing the guitar tends to require both hands, many guitar amps and effects include some kind of footswitch controller, so you can switch between different sounds and effects without taking your hands off the guitar. For a while, I’ve been after something similar that’ll let me control a computer remotely using foot switches. I’ve used various kinds of foot controllers in the past, but none of them has ever done quite what I wanted. The two that have stuck around the longest are the AirTurn PEDPro and the iRig Blueboard.

The PEDPro runs on Bluetooth and works great as a hands-free Powerpoint clicker. It connects to Windows or macOS as a Bluetooth keyboard, but it only has two keys, which are (normally) hardwired to be Left and Right. This works great for controlling Powerpoint, but it’s a bit limited.

The iRig Blueboard has four footswitches, and works very nicely with iOS and macOS, but it connects to the host system as some kind of proprietary Bluetooth MIDI device, and it doesn’t work on Windows.

So I figured building my own pedalboard might be a fun lockdown project. At a bare minimum, I wanted to be able to switch between scenes in OBS and control some sort of media player at the same time, so I can stop/start videos and backing tracks and switch between different camera angles when doing live streams.

There’s two approaches I could have taken:

Network-based: the footswitch device connects to wifi and sends signals to a host PC over the network. I write some bespoke software that runs on the host PC, which either translates those signals into emulated keystrokes, or controls the target applications directly.
Device emulation: the footswitch connects directly to the host PC, via USB or Bluetooth, and shows up as a keyboard, joystick or some other kind of human interface device (HID).

I went for device emulation, mainly because the host PC is already going to be pretty busy handling multiple cameras, OBS, media playback, greenscreen effects, and live network streaming, and having my footswitch just show up as a keyboard seemed a lot more straightforward. Plus, I’ve done a whole bunch of network programming and client/server stuff, but I’ve never built something that emulated a physical keyboard before and it sounded like fun.

The Raspberry Pi Zero W

For this version of the project, I used the Raspberry Pi Zero W. Lots of people asked why I didn’t use an Arduino or some other device: well, I used a Pi because I like them. They’re wonderful little devices, they’re lots of fun to work with, and I’ve done some interesting projects with them before so I’m not starting from scratch here.

One very interesting idea which I’ll probably return to in a future project would be to cannibalise the controller from a Bluetooth keyboard and use that instead of building my own device. More on that in a moment.

Banana! (Raspberry Pi Zero W for scale)

The Pi Zero is a tiny Linux computer, not much bigger than a stick of gum. It’s got a 1GHz single-core CPU, 512Mb RAM, a mini HDMI port, two USB ports – one for power, one for peripherals. Most important of all for this project, it has a bank of general-purpose input/output (GPIO) pins that you can wire up to external switches, LEDs, sensors, all kinds of things, and then write some fairly simple code to interface with them. Which is brilliant if you want to connect a bunch of mechanical footswitches to a tiny Linux computer and don’t really know what you’re doing.

The other nice thing about the Pi Zero W is that it’s got exactly the same hardware and programming interface as its big brother, the Raspberry Pi 4 Model B – so if you get into serious code-wrangling, you can use the more powerful Pi 4B as your dev platform, get your code working, then pop the MicroSD card out, put it into the Pi Zero and boot the exact same code on the smaller device.

Plan A: Bluetooth

My first approach with this project was to get the Pi Zero to connect to the host PC over Bluetooth, and emulate a Bluetooth keyboard. I spent a rather fun, if occasionally frustrating, weekend playing around with this. You can read the tweet-by-tweet accounts from day 1 and day 2 – but to cut a long story short, I couldn’t get it to work well enough for what I wanted.

I got to the point where I could boot the Pi, start the various Python scripts, then go onto my Windows machine and add a new Bluetooth device – and it worked. The device would show up as paired, then connected, and it would actually send keystrokes to the host PC:

I have hardware switches connected to the Pi Zero GPIO pins. Pressing a switch sends a a keystroke to the Windows PC. The PC thinks it’s talking to a Bluetooth keyboard. The whole thing is as hacky as all hell - but it works. :) pic.twitter.com/VQp5jHjG8a
— Dylan Beattie 🇪🇺 @ 🏡🔑🔽 (@dylanbeattie) May 3, 2020

There was a problem, though. When I stopped the Python script that was running the Bluetooth service on the Pi, the device would go from connected to paired – and I could not find any way to get it to reconnect without removing and re-adding it. Which made the whole thing a bit impractical, particularly if I was going to try using it in any kind of live performance situation.

Any folks out there who know the intimate details of the bluez Bluetooth stack? I need to force a Linux device to connect to a host that’s already paired... you can see exactly what I mean in the video here. Any help much, much appreciated. pic.twitter.com/7YOYV59NfE
— Dylan Beattie 🇪🇺 @ 🏡🔑🔽 (@dylanbeattie) May 9, 2020

In this video, you can see that with my Logitech K380 Bluetooth keyboard, this works perfectly – and part of me wishes I’d just cannibalised a K380, taken out the controller chips and switches, and wired my own footswitches into it. That would have made the hardware side of this a lot more straightforward… but hey, we live and learn, right?

Plan B: Wired USB

In tech projects, it’s way too easy to get fixated on your current approach and lose sight of what you’re actually trying to accomplish. It was this tweet from Jonty – and particularly the phrase “you’re wayyyy down this rabbit hole” – that got me to take a metaphorical step back and remember that Bluetooth is not actually a requirement here. The goal is to be able to control OBS and WinAmp, on a desktop PC, using my feet. I’m trying to control a stationary computer from a stationary footswitch, so having a wire connecting the two is not a problem. It turns out the Pi Zero will also quite happily emulate any number of USB gadgets, from virtual keyboards to joysticks and network interfaces.

So, I recast the project a bit. The milestone for this part was to get to a point where:

The Pi Zero is connected to a Windows 10 PC via USB
The Windows PC thinks the Pi Zero is a keyboard
I can run some Python code on the Pi Zero to send arbitrary keystrokes (including modifier keys) to the PC

This was actually pretty straightforward, thanks to a really great article “Composite USB Gadgets on the Raspberry Pi Zero” over at iSticktoit.net. I did have one slight stumbling block: when I first started working on this, I originally found this tutorial instead. That’s almost exactly the same (the code samples are identical) except the Random Nerd Tutorials article says quite clearly that you can run the whole thing off a single micro USB port:

Now, a little later in the article, our Random Nerd author does actually say:

To say this does not match my own experience would be putting it mildly. The first few times I tried booting the Pi drawing power through this USB port, it caused all sorts of chaos, including completely shutting down the entire USB bus on my Windows PC more than once, leaving me with no mouse and no keyboard. That was fun. It is possible – I got a working setup a few times using only a single USB cable – but I wouldn’t recommend it. After a few rounds of this, I tried booting the Pi using standalone USB power, waiting until it was up and running, and then connecting the second USB port to the host PC – and it worked flawlessly. You have no idea how excited I got when I saw this little pop-up in the corner of my Windows 10 display:

TL;DR: Boot the Pi using dedicated USB power. Once it is up and running, connect the peripheral USB port to the host PC, and everything works fine.

The Code

The software side of this is really in two parts. First, there’s the configuration you’ll need to run to persuade the Pi to start pretending it’s a USB keyboard. Then there’s a bit of Python code that will detect when a circuit is closed across a pair of GPIO pins, and send a particular keystroke over the USB interface when this happens. This is wonderfully, beautifully simple on the Pi Zero, thanks to the RPi.GPIO library that makes it really easy to write Python code that talks to the GPIO pins. All the code for my implementation is available on Github, along with instructions about how to get it set up and running:

https://github.com/dylanbeattie/piedroit

The important part is here – piedroit.py. This is the code that detects events on the GPIO pins, translates them into the specific data structure required by the USB keyboard interface, and sends those events to the /dev/hidg0 device that’ll pass them to the host PC:

#!/usr/bin/env python
# piedroit: translate GPIO pin inputs into USB keystrokes
# https://github.com/dylanbeattie/piedroit

import RPi.GPIO as GPIO
import modifier_keys

NULL_CHAR = chr(0)

def send_data_to_usb(data):
  with open('/dev/hidg0', 'rb+') as fd:
    fd.write(data.encode())

def get_key_code(gpio_pin):
  # https://www.usb.org/sites/default/files/documents/hut1_12v2.pdf
  # Physical switches are wired to GPIO pins 4-11
  # I want footswitch #1 (GPIO pin 04) to send F1, #2 > GPIO02 > F2, etc.
  # F1 has USB key code 58 (0x3A), F2 is 59, etc. 
  # So we can get the key code we need by adding 54 to the GPIO pin number.
  return(gpio_pin + 54)

def send_key_down(key_code):
  # USB key events are an 8-byte struct containing:
  # – a one-byte bitfield of modifier keys 
  # – a null byte
  # – Up to six key codes. (USB allows you to press up to six keys simultaneously)
  #
  # This code will always send Ctrl+{key}
  modifiers = modifier_keys.LEFT_CTRL
  key_data = chr(modifiers) + NULL_CHAR + chr(key_code) + NULL_CHAR*5
  send_data_to_usb(key_data)

def release_all_keys():
  send_data_to_usb(NULL_CHAR*8)

def send_key_for_gpio_pin(gpio_pin):
  key_code = get_key_code(gpio_pin)
  send_key_down(key_code)

# This code will support GPIO pins 4-21, although 
# only pins 4-11 are actually connected in my device
FIRST_GPIO_PIN = 4
FINAL_GPIO_PIN = 21
ALL_PINS = range(FIRST_GPIO_PIN,FINAL_GPIO_PIN)

# Tell the Pi which numbering mode we're using to talk to the GPIO pins
# https://raspberrypi.stackexchange.com/questions/12966/what-is-the-difference-between-board-and-bcm-for-gpio-pin-numbering
GPIO.setmode(GPIO.BCM)

for pin in ALL_PINS:
  GPIO.setup(pin, GPIO.IN, pull_up_down=GPIO.PUD_UP)

# Remember closing a switch connects the GPIO pin to ground (GND)
# so GPIO.input(pin) will return 0 ("grounded") or 1 ("not grounded")
SWITCH_CLOSED = 0

# Now we just sit in an infinite loop, reading all the GPIO pin states 
# every time we loop, and watching to see if any pin state has changed.
previous_states = [1] * FINAL_GPIO_PIN

while True:
  for pin in ALL_PINS:
    state = GPIO.input(pin)
    previous_state = previous_states[pin]
    if (state != previous_state):
      if (state == SWITCH_CLOSED):
        print("GPIO PIN {} now CLOSED".format(pin))
        send_key_for_gpio_pin(pin)
      else:
        print("GPIO PIN {} now OPEN".format(pin))
        release_all_keys()

      previous_states[pin] = state
 
# This is good practice, but since we're going to sit in our loop
# until we kill the process or disconnect the power, we'll never
# actually get here. But in this scenario, that's OK.
for pin in ALL_PINS:
  GPIO.cleanup(pin)

The Case

Finally, I needed a box to put it all in – a box that wouldn’t get upset if 102kg of Dylan stood on it wearing cowboy boots. Check out part 2 of this post which involves a lot less code – and a lot more sheet steel.

🤘🏻

Microphone Tips for Remote Presentations

2020-05-12T12:00:00+00:00

In my last post, I shared some tips for giving online presentations. In this talk, we’re going to completely nerd out about microphones, and how to get the best possible results out of the equipment you already have.

drestwn via Flicker / CC BY 2.0

Audio quality is essential to connecting with your audience when you’re presenting remotely. A conference is about speakers giving talks. Not looks, not stands, not dances – talks. We are speakers, we are there to talk, and our audiences are there to listen. If your audience can’t hear you, you’ve lost them, no matter how beautiful your slides or or how many hours you’ve spent practising… and when you’re presenting online, your microphone is the Jesus nut – the single point of failure that can bring the whole thing crashing down.

You can easily spend a fortune on specialist microphones, but if you don’t know what you’re doing, it won’t make any difference – and if you do know what you’re doing, you can get great results from relatively inexpensive kit.

Your ears never blink

Humans are good at dealing with visual clutter. We can focus our eyes, look at the things we care about and ignore unnecessary background detail - check out this video for a remarkable demonstration of this.

Audio isn’t like that. Your ears never blink. We hear every detail, every keystroke and passing car and buzzing fluorescent lamp - and so when we’re doing presentations, it’s absolutely vital to understand how much background noise and interference your mic is picking up.

Before we go any further, take a look at this video. I recorded myself using seven different microphones at the same time, so you can hear for yourself how the different microphones sound, and how they cope with different kinds of interference and background noise.

Microphone placement

You want your mic as close to your mouth as possible; 5 cm (2”) to 25 cm (10”) is ideal. You also want to avoid moving around too much relative to the microphone.

Headset mics are great, because they keep the microphone in the same position relative to your mouth – you can turn your head, walk around, and they still sound good. Lavalier mics – the ones that clip into your clothes – and phone headsets also work pretty well, and you can stand up and walk around, as long as you remember not to turn your head. If you’re using a fixed mic on a stand, you’ll need to sit very, very still to get the best results from it.

Background noise and isolation

You’ll notice in the video that the built-in mic in the Surface Pro sounds really good, right up until I start typing. It’s actually a really good mic, but because it’s built in to the device, it’s in the worst possible position – it’s on the desk, quite a long way from my mouth, right next to my keyboard, and it’ll pick up every bit of vibration and background noise.

This isn’t just a problem with built-in mics – I’ve seen folks spend quite a lot of money on specialist USB microphones like the Blue Yeti, and then sit them on the desk right next to their mechanical keyboard and wonder why they’re not getting a good sound.

If your mic is attached to your clothing, or you’re using a phone headset, watch out for clothing rustle – check out this article from Rode about lavalier mic placement, which has some excellent tips.

Connection

If you’re going shopping, you’ll need to consider how your mic connects to your computer. USB headsets will just plug straight in, and the headphone jack in most modern laptops is compatible with the TRRS plugs used on phone headsets.

If you’re going for a standalone mic or a wireless system, you’ll need an audio interface to connect it to your computer. I highly recommend the Yamaha AG03 and AG06, for this – they’re great quality, incredibly versatile and easy to use.

Yamaha AG06 and AG03 portable USB mixing consoles

Listen to yourself

The golden rule with any microphone, though is: listen to yourself. Install some audio recording software – Audacity is free, open-source and cross-platform – make some recordings, listen back to yourself (and make sure to listen through headphones.) Record 30 seconds of silence and listen back to it. What do you hear? Traffic noise? Dishwasher? Mouse clicks? If you can hear it, your audience will hear it. You might not able to eliminate background noise entirely, but figure out what you can to reduce it – close windows, close doors, take off your shoes, switch off the aircon. Make yourself a pre-flight checklist.

Now record 30 seconds of you speaking. at the same volume you’d use when giving a talk, and listen back to that. Try reading these ten sentences out loud, then listen back and see what you sound like:

The dark pot hung in the front closet.
Carry the pail to the wall and spill it there.
The train brought our hero to the big town.
We are sure that one war is enough.
Grey paint stretched for miles around.
The rude laugh filled the empty room.
High seats are best for football fans.
Tea served from the brown jug is tasty.
A dash of pepper spoils beef stew.
A zestful food is the hot-cross bun.

This is one of 72 lists of stock phrases known as the “Harvard Sentences”:

“a collection of phonetically balanced sentences that measure a large range of different qualities in the human voice. These were originally published in 1969 as the Institute of Electrical and Electronics Engineers recommended practice for speech quality measurements.”

– you can find the whole list online here.

If you hear loud pop or thud on words like ‘pepper’ and ‘bubble’, that’s called a plosive – it’s the mic picking up the sharp release of breath that happens when we pronounce letters like ‘b’ and ‘p’. See if you can move the mic downwards or to the side slightly so that it isn’t directly in front of your mouth. If you hear hissing or whistling on words like ‘stew’ and ‘seats’, that’s called sibilance – try moving the mic a little further away from your mouth.

Play with the distance until you’ve got a good clear voice with minimal background noise, but you’re not getting any hissing or popping. If your mic has a volume control, turn it up until you’re about as loud as you can go without hearing any distortion or seeing any warning lights, then back it off about 5%. Do this every time you change anything – move the furniture, get a new mic, install new drivers. And remember the golden rule: listen to yourself. That’s what your audience is going to be listening to for an hour – if it doesn’t sound good in your own room, on your own headphones, there’s no way that sending it halfway around the world on a video call is going to make it any better.

Conclusions

A USB headset gives the best balance of quality and price – you can get excellent audio quality from a £20 headset, and it won’t pick up things like typing and background noise. The only drawbacks here are that you’ve gotta have a big chunky thing on your head, and if it’s wired, you can’t get up and walk around. If you’re going to be sat a desk for your recordings, and don’t mind how it looks (or you’re not going to be on video), go for one of these.

If all you have is a phone headset, use it – but be very careful not to move around too much. You’ll get good results if you sit still and speak clearly, but watch out for the mic rubbing on your clothes and be careful you don’t snag the cable.

If you’re using a fixed mic like the RØDE Podcaster or the Blue Yeti, make sure it’s mounted high and insulated from vibration – ideally on a proper microphone stand with a shock mount.

If you have no choice but to use the mic built in to your laptop, work out how to isolate it as much as possible from background noise. If your presentation doesn’t involve any typing, clicking or anything, you’ll probably be OK. If you need to type, try resting the laptop on a pillow or a cushion - maybe even use an external keyboard as well to reduce the typing noise.

And remember: you don’t need to spend a lot of money on specialist equipment. Start with what you’ve already got, make some recordings, listen to yourself, figure out what’s not working, fix it, repeat.

Online Presentation Tips

2020-05-01T12:00:00+00:00

As we enter the third month of the global lockdown, more and more events are exploring online conferences as an alternative to postponing or cancelling, which means more and more speakers are being asked if they’d be able to do their talks as an online session presented over a live stream.

Since March, I’ve spoken at four virtual meetups, presented six talks at four online conferences, and helped the crew from NDC turn Copenhagen and Porto into virtual events. I’ve experimented with a ridiculous amount of gear – old and new – and I’ve talked with hundreds of speakers about the challenges we’re all facing to adapt our material, turn our homes into impromptu broadcast studios, and keep on delivering great content and inspiring presentations even when we’re all locked down for the foreseeable future.

I’ve had lots people asking me for tips on this – so here it is: everything you ever wanted to know about online presentations. Well, almost everything. There’s some seriously deep-dive posts coming up about microphones and cameras, but this will get you started. :)

Test, test, test.

The absolute, cast-iron, golden rule of online presentations: Test everything. Test, test, test. It doesn’t matter how good you are in your room – what matters is what the remote audience can see and hear. Figure out how to see through your own camera, how to listen to your own microphone, how to see your own screen share. Put yourself in your audience’s shoes and see how much of your content actually makes it out the other end – and then tweak, improve and adapt until it’s perfect. Every time you change anything, test it.

Rehearse your content.

This sounds obvious, but it’s easily overlooked. Online isn’t just another meetup or conference stage, it’s a completely new format. Even if you’ve done a talk a dozen times, the first time you do it remote, it’s going to be weird. Rehearse. Do your talk by yourself, in the room where you’re going to be presenting. Don’t worry about equipment yet – this is about content. Rehearse until you can deliver your talk flawlessly, to a blank wall that doesn’t laugh at any of your jokes or raise a hand for any of your questions. If you can handle that, you can handle anything that a virtual conference is going to throw at you.

Microphones.

You don’t need an expensive specialist mic to do a great remote presentation – but you will need something a little better than the built-in mic in your laptop. The golden rule here is distance. You want your mic as close to your mouth as you can get it – between 2” (5cm) and 6” (30cm) is ideal – and as far away from any other source of noise as you can. Gaming headsets, Bluetooth earbuds, a clip-on microphone, even the cord mic on the earphones that came with your phone – any of them give a better result than relying on the mic built in to your laptop.

Cameras and video.

The classic setup here is a webcam for your face and a “screen share” for your slides – and in most cases, that’ll work absolutely fine. As with microphones, you’re much better off with inexpensive equipment and a bit of know-how than spending a fortune on gear you don’t know how to use properly.

Here’s the secret. Even the cheapest webcams are good enough for a remote presentation if you give them enough light to work with. You want light shining onto your face, and a neutral background behind you. If there’s a window in the room where you’re presenting, move things around so you’ll be facing towards the window. I took these photographs using the same camera, same day, same time, same distance – the only difference is that I turned 180° between shots:

https://twitter.com/TessFerrandez/status/1246444749961277440/

Oh, and one final camera tip… stick some googly eyes on your webcam to remind you to look at it. If you look straight at the camera. That’s your audience now, and every speaker knows that making eye contact with your audience goes a long way.

Join from a second device.

Get a second device – your old laptop, your iPad, even your phone – and connect it, as an attendee/guest, to the same call where you’re presenting. Mute it, switch off the video, and you can see exactly what your audience will see. For rehearsals, you can also run a screen recorder on the second device, and then review the recording to see how you look and sound to your remote attendees. If you’re on macOS, Quicktime has a built-in screen recorder. If you’re on Windows, there’s a built-in screen recorder hidden in the X-Box Game Bar (no, really – press Win+G, and look for the Capture window).

Note that if you’re on a slow internet connection, you won’t want two devices fighting over the available bandwidth, so consider tethering the second device to your phone to provide an isolated internet connection.

Make a pre-flight checklist.

One of the big challenges with the online format is that we’re having to present from home – which means the room where you’re presenting is also your living room, or your bedroom, and you can’t leave it set up as a live-streaming studio 24/7. You’ll do a tech check, it’ll look and sound great… and then you’ll unplug everything, put the furniture back where it’s supposed to go, open the curtains, switch the aircon back on. And tomorrow, when it’s time to do your talk, you’ll try to put everything back how it was in rehearsal – and you’ll forget something.

Once you find a setup that works, make yourself a pre-flight checklist. Then run through this half an hour before you’re due to speak, to make sure everything’s set up just right.

Connect with your audience.

Different events have done different things here, but having some kind of connection with your audience makes a huge difference.

If you’re using something like Zoom or WebEx to deliver your talk, see if you can get attendees to join the call directly, mute their mics, switch their cameras on, and be your live audience. You won’t be able to hear them, but if you enable the “gallery view” so you can see their faces, it affords a certain amount of live interaction – you’ll be able to see people laughing, raising hands, that kind of thing. If you’re streaming via a platform like YouTube or Twitch, you won’t be able to see any of your audience, but you can interact with them via chat. Get the YouTube chat window up where you can see it, keep one eye on it whilst you’re speaking, and respond directly on the stream – “hey, I see a question in chat from Carol about accessibility. Hi Carol! What I’d do is… {answer}”

For larger events, a hybrid approach works well, especially for keynote talks which might have many hundreds of attendees. Invite the other speakers to join the call directly, and run a YouTube stream for everyone else. The speaker gets a tame audience of tech-savvy nerds and some decent Q&A and interaction, and the rest of the attendees tune in via the YouTube stream and ask questions via chat.

Bear in mind YouTube and Twitch can have a bit of latency – anything up to 20 seconds – so you’ll tell a joke… n response… then 20 seconds later you’ll get a little flurry of laughing-face emojis in the chat. It’s weird, but you’ll get used to it.

Stand up, dress up, remember biology.

You know your desk. You sit there, day in, day out, alt-tabbing between Slack and Visual Studio and Twitter and Amazon and Facebook and Stack Overflow – and since the lockdown, that’s also where you play poker with your buddies, where you sing happy birthday to your niblings… and now you’re going to sit there for an hour, giving your absolute, undivided attention to an audience you can’t see or hear? That’s gonna be on tough gig before it even starts… so mix things up a bit. Get your webcam up on a tripod, or a bookshelf, or, hell, put a chair up on the table and put your laptop on the chair – and stand up. Standing up changes your posture, it changes your voice, it keeps you focused on the fact that whatever else is happening around you, you are on a stage right now and people all over the world are watching you do your thing.

On the day you’re presenting, dress up. Dress like you would if you were doing this for real – shower, do your hair, makeup if that’s your thing… hit that call looking like a million dollars, ‘cos even on the ropiest connection in the world, a few hundred thousand bucks will make it out the other side. It’s also a good way of persuading your brain that something special is happening and this isn’t just another pyjama day in lockdown.

Finally, remember that you’re going to be presenting for an hour. Your brain doesn’t quite get this. Your brain thinks you’re at home, where the bathroom is right down the hall and you can always pop to the kitchen for a snack or a glass of water… but you’re not. You are at a conference, on a stage, with hundreds of people watching you. Now is no time to take a toilet break – so go before you join the call for your talk. Eat a snack, make sure there’s a glass or two of water within easy reach (having a coughing fit during a talk is bad enough at the best of times, but right now it’s a really bad idea) – and be prepared to take a break afterwards. If you do it right, you’ll forget you’re at home; you’ll be 100% focused on your talk and your audience – and without the feedback you’d get from a live crowd, you’ll hit the end and you’ll crash, hard. The part where you’d normally be buzzing from the adrenaline of live performance… you’ll get that, but you’ll be in a room, by yourself, staring at a Slack chat. Give yourself some time to recover. Take a break, walk around the block, sit and drink a glass of water, decompress.

Relax. You can do this.

The transition to online events is hard. We’re having to adapt quickly, in difficult circumstances, and it’s not like you can invite a couple of friends over to help you figure this out, or organise a (physical!) meetup to talk about how to make it work.

There’s a new set of skills to learn here. There’s new tools and technology, sure, but there’s also a completely new presentation style to master – one that’s familiar to radio DJs and studio TV presenters, but completely alien to most of us over here in the tech industry. And we’re doing it in some seriously weird circumstances. As somebody put it the other day:

You are not “working from home”. You are at home, trying to work, in the middle of a global pandemic that has turned the world upside down. Cut yourself a little slack.

So relax. Rehearse, learn the tech, ask your friends to help, and get out there and be awesome.

I mean stay home. Don’t get out there. STAY HOME and be awesome.

And don’t forget to wash your hands :)

Going Virtual with NDC Copenhagen

2020-04-03T12:00:00+00:00

A month ago, I was in Denmark doing a promotional event for NDC Copenhagen - and nobody was even considering the possibility that we might have to turn it into an online-only event – but as the agile manifesto says, developers should be prepared to respond to change instead of following a plan. And, wow, the last four weeks have been intense. We’ve had less than a month to turn a physical conference into fully virtual online event - two days of workshops, one day of talks across three tracks, nearly 250 attendees and speakers…

We did it. Right now there’s three talks streaming live, there’s 247 people online on Slack, there’s Q&A happening in the Slack channels, I’m watching Scott Helme stream his “Stories from the Trenches” infosec talk live on YouTube, and it’s all working remarkably well. Here’s how we did it.

Transparency and Trust > Technology and Tools

We worked hard to keep speakers and attendees in the loop about what was going on. I ran webinars and Q&A sessions last week for all of our workshop trainers and presenters, we ran another online session on Monday for attendees, and I’ve been running ad-hoc equipment checks and Zoom sessions with speakers to help them make sure everything’s set up and good to go. It’s been a very steep learning curve for all of us, but it’s worked.

We also solved a lot of potentially complex problem by deciding, early on, that were going to trust people. Attendees would get invited to a Slack workspace but we weren’t going to require authentication; we’d run live streams as unlisted YouTube streams and trust our attendees not to share the links. This meant we didn’t have to worry about passwords and security and managing access, and could just focus on the attendee experience, the speakers and the content.

Familiar Tools

Over the last few weeks, I’ve tested every single video conferencing and online chat platform I’d ever heard of, and a few that I hadn’t. WebEx, Whereby, GoToMeeting, Zoom, Hangouts, Skype, Slack, Discord, Jitsi - and I’m sure there’s a few that I missed.

Let’s put it this way. None of them is remotely close to meeting together in real life, but all of them are vastly better than getting sick with coronavirus - so it comes down to what constraints and capabilities we’re optimising for, and what we’re prepared to compromise on.

There are virtual conference platforms out there - but there’s no way we were going to have enough time to evaluate the options, choose a platform, get the crew and volunteers up to speed, set up accounts for the speakers and partners, and have it all ready to go by April 1st.

We also wanted to make sure that, wherever possible, our speakers could present their sessions using equipment they already had; some of our speakers create a lot of video training and so are already tricked-out with greenscreens, cameras, and microphones - but a lot of them just have their development laptop, maybe a USB webcam and a headset microphone, and the last thing we wanted to do was ask speakers to go out and buy specialist equipment when we weren’t ourselves confident about the format and the technology yet. So we decided to go with familiar tools, hooked together in interesting ways - Zoom, Slack, Slido, YouTube, and e-mail.

We set up a dedicated Slack workspace for NDC Copenhagen 2020, and decided that would be the backbone of the virtual conference. Folks who bought tickets got invited to join the Slack workspace, we use Slack for discussion, announcements, Q&A, chat, and - here’s the key part - at the end of the conference, we’re going to delete the workspace. That’s it: conference is over, see you at the next one.

How We Set Up Slack

Every Slack workspace has a #general channel. We renamed #general to #announcements, and locked it down so that only administrators could post to it and nobody could start threads.
We created dedicated channels for each conference track, which we used for speaker Q&A and audience feedback.
We set up a #hallway-track for attendees, a #speaker-lounge private channel for speakers, and a dedicated #organisers channel for the NDC crew and volunteers.
We locked down most options, so that attendees couldn’t invite other people, couldn’t create custom emoji, that kind of thing.

How We Streamed Talks

We went with a hybrid approach here. My initial idea was to use Zoom for all the sessions - to start a Zoom meeting for each track, and have speakers and attendees join and leave those meetings over the course of the day, just like folks wandering in and out of session rooms at a physical conference. I’d run a bunch of meetups and social events on Zoom, with 50+ people in the same meeting, and it worked pretty well.

We had several requests from attendees to be able to watch the live streams without actually joining a room. That might sound a bit weird, but NDC Oslo has always had an overflow room; all nine tracks are projected on big-screen TVs, you can sit with a pair of headphones and jump between talks without disturbing anybody, and we wanted to try to replicate that for the virtual event.

So, with some help from the crew from Video for Web in Norway, here’s the setup we ended up running.

Each conference track is a Zoom meeting, that starts around 8am and runs right through the day. One of the NDC crew stays in that meeting all day - they’re the track host; they’ll help speakers get set up and connected.
The Video for Web crew have three separate PCs, each one connected to a different Zoom track. These are running Open Broadcast System (OBS) - they’ll do a live capture of the speaker video and screen share from the Zoom call, combine this with the Slido feed and some NDC branding, and stream the result to YouTube.
We email every speaker an invitation in advance, including the Zoom link and password to join the appropriate track, and the time of their meeting, including the timezone - more about that in a moment
The speaker joins the Zoom track 10-15 minutes before their talk session, checks their camera and audio are working, and begins sharing their screen.
When the talk starts, the Video for Web team switch over each stream to the live feed coming from Zoom via OBS.
We embed the YouTube stream link in each of the Slack channels for tracks 1-3.

For a couple of my talks, I also shared the direct Zoom link in the Slack channel in case folks wanted to join directly, which actually worked really well - more on this later. We also ran Slido alongside every session, giving the audience the chance to ask questions during the talks.

What We Learned

Some Folks Like Zoom, Some Folks Like YouTube

For the talks where I shared the Zoom link, I got 5-10 attendees joining the Zoom call directly, with the rest watching over YouTube.

As a speaker, it’s amazing how much difference it makes even just having a few people “in the room” - we kept the audiences muted, but being able to see people smiling and raising hands makes the whole thing feel so much more like a live presentation. The YouTube stream has about 30 seconds of latency - so the quality’s great, but if you tell a joke, people on YouTube laugh 30 seconds later, so there’s really no way to use the YouTube stream as part of any realtime audience interaction.

So for future events, I’m going to try to run a hybrid format where there’s a real-time option like Zoom or Hangouts for people who want to be part of the “live” audience, and a close-to-real-time YouTube stream for folks who want to dip in and out or have it on in the background.

Zoom socialising? Nice idea… but doesn’t really happen

One of my favourite things about conferences is the ad-hoc conversations that happen - in the coffee queue, over lunch, over drinks at the end of the day. We tried to replicate that using Zoom by spinning up chatrooms between sessions, but for some reason it just doesn’t work - nobody joins the rooms and it never really catches on. We did run an online quiz event last night, which got about 20 people online at one point - but most of them stayed muted with their webcams switched off; I could see them playing along and answering questions on Slido but I wouldn’t say we had much in the way of conversation.

That said, there’s been some great conversation happening in the workshop groups over the past two days, so it’s not the tool or the format that’s the problem… I’ve already written several posts about the weirdness of trying to socialise online; I suspect this is just one more thing that people haven’t figured out yet. We’ll get there.

Slido needs a nudge to get people using it

We ran Slido live Q&A sessions for all the conference talks. Some talks, it didn’t get used at all; but those talks where it did get used, it was completely silent until somebody asked the first question - and then it got very lively.

I think there’s two reasons for this… first, until somebody asks a question, it’s just a static panel in the corner of the live stream. Once the first question appears, there’s some movement and some action and people go “ooh, I want to use that!” I also know there’s some weird cultural things about certain audiences where folks are more than happy to interact but nobody wants to go first, and I haven’t worked out yet whether that’s more or less pronounced when the events are running online.

So - if we’re going to use Slido in future, we’ll have a couple of prepared questions standing by to start things off; once folks see it being used, they’ll probably dive in and get the conversation going.

USB-powered lights are literally the devil… 😈

One of the speakers was having huge problems with audio - pops, crackles, weird electrical interference. We spent literally hours trying to track down the problem - reinstalling drivers, checking different mics on different platforms… and eventually worked out that they were running a USB-powered light off one of the computer’s USB sockets, which was generating some really, really weird interference.

There’s a separate post coming up about how to set up and test equipment for doing live remote presentations, but it looks like one of the golden rules is: don’t have anything plugged in to your USB ports that isn’t required to do your talk. Webcam, microphone, whatever you need to do your presentation or live demos - but if you’re using USB ports to power lights or charge your phone or anything, unplug it until you’re done.

So… What’s Next?

In another life, I was getting on a plane tomorrow, flying from Copenhagen to Saint-Petersburg via Helsinki, presenting The Art of Code at DotNext, travelling from there to Riga and then on to Kyiv for .NET fwdays - and then back on the NDC bandwagon at the end of April for NDC Porto.

But here in the upside-down, that’s all postponed, cancelled or gone virtual, so the next big thing on the schedule is NDC Porto, which we’re now running as another online event. Copenhagen’s gone great, we’ve learned a lot about what works, what doesn’t work; we’ve got lots of interesting new ideas to try and lots of valuable feedback and information to share with all our speakers…

For now, though, I’m gonna open a beer, fire up a Zoom call and enjoy a bit of post-conference relaxation. See y’all for the next one!

Going Virtual, Part 3

2020-03-22T12:00:00+00:00

Once upon a time, a very long time ago now, about last Friday, people used to go to pubs and bars and talk to each other face to face. You remember. Now that we’re living here in Generation Lockdown, that’s not a thing any more, and so we’re all socialising on Zoom and Hangouts and WebEx. Until last week, I’d never joined a “Zoom party” in my life… this week, I’ve joined four. It’s been interesting, and fun, but humans, we got some details to figure out if this thing’s gonna work.

Let’s start with some observations. Human gatherings exist on a spectrum. At one extreme there’s the business meeting, in which a group of named invidivuals meet at a specified time, for a specified time. During this time, all the attendees are expected to give the meeting their undivided attention. One person speaks at a time, the rest listen, maybe somebody takes notes, and when it’s over, everybody leaves. Now I know as well as any of you that meetings in the real world don’t necessarily work like that, but most of the software we’re using to talk to each other now was designeс сердечным приветомd for business, and so it is designed to try and replicate that experience.

A little further along, there’s dinners, tabletop gaming sessions - probably still the same group for the duration of the event, but it’s a little more relaxed. Then there’s house parties and pub nights, where there’s a definite location where “the thing” is happening, but people are wandering in and out, splitting into conversation groups, branching and merging and mingling and going outside for a cigaratte. And then there’s one of those nights when you head out alone, visit four or five different bars, seeing who’s where, bumping into random friends and friendly randoms.

I love parties and pubs and bar crawls, but the social dynamic of those events is not 20 people sat around taking it in turns to speak. I miss the ebb and flow of social events. What’s the online equivalent of going to the kitchen for a beer and staying in there chatting for half an hour because everyone in the living room is currently talking about school fees and you’re bored off your ass? Where’s the quiet corner where you can sit for half an hour and have a proper catch up with a friend you’ve not seen in a while before going and rejoining the main party? Video chats are very binary - you’re in, or you’re out. Sure, you can be in with your mic muted and your video switched off, but at some point when you decide you’ve had enough, do you interrupt everyone to say goodbye, or just disconnect? Interrupting feels rude. Disconnecting feels rude. The cues around body language that we rely on to know when it’s OK to speak – or that we subconsciously pick up on to know that somebody else would like to say something – are much, much harder to pick up on. And the medium amplifies people’s naturally extrovert or introvert tendencies. People who are loud tend to get louder, people who are quiet tend to get quieter.

There’s also weird stuff here about boundaries. We’re all stuck at home, nobody’s going outside. Everybody’s struggling with work/life balance, and sharing neat ways to deal with it. I’ve spoken to people who go out their front door in the morning, walk around the house and go in through the back door to start work. I’ve spoken to people who have sleeping pyjamas, work pyjamas and relaxing pyjamas - because hey, pyjamas are comfortable, but wearing the same ones for 24 hours is gross. But the psychological cues we get from physical spaces are gone. Office: drink coffee, do work. Pub: tell jokes, drink beer, it’s OK to swear. Home: sit around in your underwear watching TV. Now those spaces are gone and we need to work out how to simulate them. I live alone in a house with a lot of technology; right now I’m trying to work out how to join Zoom calls from my living room TV so I’m not doing all my socialising from my office – partly ‘cos it feels friendlier somehow, and partly because drinking beer at my desk regularly is likely to be the start of a very steep and slippery slope.

I think we’ll figure this out. Some new social conventions are going to emerge, and quickly. Stuff that was just about tolerable when we used video chat for paid meetings will quickly become socially taboo. We’ll figure out how to digitise liminal spaces, we’ll come up with ways to use the tools and tech we’ve already got in interesting ways. This crisis is going to do for online communication tools what World War II did for aviation; we’re going to see an unprecedented wave of innovation and invention. Not all of it will work terribly well, but the things that do will be genuinely useful and a year or two from now we’ll be astonished that we ever tried to do online comms without them.

Now if you’ll excuse me, I have to go. Somebody’s telegrammed me a whatsapp with a link to the slack with the link to the discord where they’re trying to decide whether to go to zoom, webex, jitsi or hangouts later, and I should probably join in. It’d be rude not to, right?

Going Virtual, Part 2

2020-03-19T12:00:00+00:00

There’s this whole Einstein relativity space/time continuum thing - you move faster in space, you move slower in time, and apparently vice versa. You know the one. And I gotta say, it feels like since we all decided we were going to stop moving around in space, the moving-through-time thing has gone into overdrive… today is March 19th. Two weeks ago I was in Copenhagen, celebrating my partner’s birthday at Warpigs with a bunch of old and new friends from all over the world. We spent Friday cycling around Copenhagen on rented bikes; on Saturday we went to Sweden for the day. Sunday we flew back to London. A week later, Denmark had gone into total lockdown; as of today, Copenhagen is deserted, and pretty much the entire software industry is hunkered down, working from home and waiting for this thing to blow over.

Last Friday I ran an online lunch over Zoom. On Tuesday I ran a virtual meetup. Tuesday night I had online drinks with the Linebreakers - and then somebody messaged me to say there was a party going on on Microsoft Teams as part of the Virtual MVP Summit, so I stopped by there for a bit, and then invited everyone back to Zoom… but let’s face it, it wouldn’t be a proper conference if I didn’t turn up to some Official Corporate Drinks and drag half the attendees away to a sleazy rock bar somewhere. Nice to know that some traditions work just as well online as they do out in the Big Room.

And yes, this is all still very weird; the news cycle has gone into overdrive, nobody has the faintest idea what new restrictions and cancellations are going to be announced in the next 48 hours, and we’re all figuring a lot of things out very, very quickly. As you probably saw, NDC Conferences has hired me as their Online Community Ambassador for the next few months whilst we’re working this stuff out - which is awesome, because it means I can give this stuff my undivided attention until we’ve all worked out the details.

Today I ran our second virtual meetup - we had Filip Ekberg talking about pattern matching in C#, with lots of technical content and code samples; Clifford Agius talking about building a mobile flight simulator out of a burger van, and Christine Seeman talking about ashtanga yoga and even showing us a couple of yoga positions live via Zoom, which was really cool. We had 66 live attendees at one point, most of them live on camera and voice chat, and we tried out a couple of different things based on feedback and ideas from Tuesday; here’s what we learned.

Timezones are Weird

I mean, we knew that anyway - but we had a bunch of folks in Europe on their lunch break, Christine presenting live from Omaha, Nebraska where it was still early morning, and folks tuning in from as far away as Australia where it was already evening.

Doing some evening coding and joining some UK friends on their lunchtime talk testing 😁

For some reason @dylanbeattie is now on my TV 😂 pic.twitter.com/zeR85c4jej
— Amy 🏡👩‍💻🐺 (@Amys_Kapers) March 19, 2020

It Takes Two: A Producer and a Presenter

For today’s meeting, we had Martin Thwaites “producing” the meetup - I added him as a co-host on the Zoom call, and he was the one muting and unmuting people, switching the ‘Video Spotlight’ onto the speaker, answering questions in the chatroom, and that kind of thing. This was a huge improvement over Tuesday - it meant I was free to host the meetup, stand in front of a camera and actually talk to people without having to remember to click this and select that and keep one eye on the chat session.

We might have found some weird behaviour around “mute all” when working with multiple meeting hosts… further research required, but at the end of the session we found a few folks were stuck on mute and couldn’t unmute themselves. We’ll run some experiments and report back.

But generally, I think we file this one under “best practice” - the producer runs the meeting, the presenter does the talking. It also wasn’t a problem at all that Martin and I weren’t in the same room - or indeed in the same city; the platform coped with this just fine.

Martin’s comments on Zoom:

“Going full screen you can undock the speaker view - having the the speaker view, spotlight view and chat all on different screens helped”
“Switching the Video Spotlight during the Q&A needs some thought - it’s good to see you asking the question and the speaker answering it, but I’m not sure how jerky that was for the audience.”
“Having speakers as co-hosts might help - finding them (in the gallery view) to spotlight them was a pain”

Use Sli.do for questions

We set up a Slido event for this, and shared the link in the chat and as part of my intro slides. One great thing to remember about online events - if you want people to go to a URL, you don’t have to rely on web addresses and QR codes on a PowerPoint slide; you can just get the producer to put the link into the chat.

During this part of the meetup, I used the Open Broadcast System (OBS) rig to superimpose my ‘talking head’ onto the Sli.do “presenter view” so that video attendees could see the questions, which worked great (and looked quite professional as well!)

During discussion at the end, Shahid Iqbal - who recently hosted a virtual meetup as part of NDC DevOps - pointed out that it can be really confusing having questions going on Sli.do and people posting questions in the Zoom chat. Something else that we’re going to need to figure out as we go along.

Martin’s comments:

“Dismissing questions is an easy way to moderate, they disappear from the screen. Having it behind you helped. Promoting the questions needs to be a bit more insync so that the video switching happens better.”
“I’m thinking that it might be an idea to think about streaming to a location that then outputs to Zoom, then something like OBS can combine the streams – so the idea would be that all the hosts somehow join a stream, and that stream is sent to the zoom… would make coordinating the speakers easier, but has some fairly significant downsides”

Equipment Makes a Difference

I’m sure this doesn’t come as a surprise to anybody, but when you’re running an event like this, it’s quite apparent that people fall into three broad categories when it comes to online presenting.

People who haven’t really ever done this before - which is fine; the main reason I’m running these virtual meetups is to give everyone a chance to practise, refine their setup, adapt their material and so on.
People who are clearly experienced presenters and have also done a lot of video calls and remote meetings, but haven’t really tried presenting online before. These are the folks who don’t need a whole lot of practice, but for whom investing in a better mic or a decent lighting setup will make a huge difference to how well their material works in a virtual format
People who basically talk to cameras for a living. The folks with the professional-quality microphones, the hi-def cameras and green screen backdrops.

Now, there’s definitely a law of diminishing returns here - particularly for live streaming, the difference between a built-in microphone and a £20 headset is obvious, but you’re probably not going to hear much difference between a £100 mic and a £500 mic once it’s been compressed through Zoom or Google Hangouts and streamed across the internet. I’ll be doing some more research around this over the next week or so, and hoping to share some recommendations with you all for mics, headsets, cameras, lights, etc. that’ll let you up your remote presentation game without breaking the bank.

If you want to see what a really top-notch remote presenting rig looks like, check out this YouTube video of Aaron Parecki running through his seriously impressive setup to see how deep the rabbit-hole really goes:

What’s next? Well, Friday afternoon we’ll be running a remote live coding workshop with Mathias Brandewinder - 16:00 UK time / 09:00 PDT - to figure out some details around running workshops with a lot of hands-on coding and collaboration; next week I’ll be running a couple of webinars with NDC for speakers and attendees at their forthcoming events, and I’m talking with the crew from BuildStuff about doing some online stuff with them as well.

Exciting times. Stay tuned.

NDC Online Community Ambassador

2020-03-18T12:00:00+00:00

Hi everyone.

Things are weird, and a little scary, right now. People, companies and countries are all responding to the COVID-19 crisis in different ways, but I think we’re all agreed that getting hundreds of people together in a conference centre for days at a time isn’t a good idea over the next few months.

For technology conferences and events, this creates some interesting challenges. I’ve been working with the team from NDC Conferences since January, helping to build stronger relationships between NDC’s flagship conferences and the developer communities that they serve. None of us is going to be meeting face-to-face for a while - but we’re absolutely committed to supporting those people and communities while this situation is ongoing. Big events like NDC are a lot of fun, but they’re also an incredibly important part of our industry. Conference workshops are a critical source of revenue for a lot of people working on open source software. Sponsors and partners use conferences to connect with potential customers and employees – and for attendees, an event like NDC is a chance to meet the experts, discover new technology, connect with new people, maybe even find their next job. Turning a physical conference into an online event isn’t easy; there are challenges around content, technology, infrastructure, scheduling - all sorts of things that we’ll need to figure out. So that’s exactly what we’re going to do.

To that end, I’ll be joining the NDC Conferences team for the next few months as their Online Community Ambassador, to help them – and all of us – figure out how to run the best possible virtual events. I’ll be running live trials, online workshops and virtual meet-ups to help speakers figure out the best way to adapt and present their material online. I’ll be testing equipment, platforms and formats, and working closely with all our speakers and experts to make sure we give all our virtual attendees the best possible experience. Obviously our initial focus will be on the technical content - workshops and talks - but we’re also going to try and virtualise the hallway tracks, the social elements and all the wonderful conversations that take place during an event like NDC.

It’s been a really difficult couple of weeks for everybody, but in a way, the timing is fortunate. NDC Copenhagen at the beginning of April is one of the smaller NDC conferences; NDC Porto at the end of April is a little bigger, and NDC Oslo in June is our biggest event. Now, obviously when everything’s gone virtual the physical locations don’t matter so much - but by the time we get to Oslo, we’ll have had plenty of opportunities to experiment with formats, figure out the details and make sure it all goes smoothly. And I know as well as you do that there’s no way an online virtual event can replicate all the conversations, coffee, drinks, dinners and boat cruises and atmosphere that make events like NDC so memorable, but we’ll be doing everything we can to maintain the character of our events whilst we’re running things online. Same speakers, same sessions, same conversations - but you won’t have to pay Norwegian prices for a beer afterwards.

We’ll be contacting all the confirmed speakers and workshop trainers soon to start working out some details. In the meantime, hang in there, take care of yourselves, and if you feel like hanging out online with some friendly nerds whilst we start figuring this stuff out, I’m running an open Slack at ursatile.slack.com and you’re all very welcome to stop by.

Thanks all.

Dylan

Going Virtual, Part 1

2020-03-13T12:00:00+00:00

It’s March 2020 and everything is cancelled. COVID-19 has been declared a pandemic by the World Health Organisation, Norway is closed, Italy is closed, the US has banned travel from most European countries, and the British government is, um, advising us to wash our hands while singing songs until we develop “herd immunity”.

With many countries now banning all large public gatherings, this has obviously hit IT conferences and community events hard. Within the last 24 hours, NDC Conferences has announced that their forthcoming events in Copenhagen, Porto and Oslo will be virtual online events; JUG.RU - the largest conference organiser in Russia - has postponed all of their forthcoming events until later in the year; local groups are switching to running virtual online meetups, and we’re generally all trying to figure out how to make the best of a pretty rough situation.

We’ve all done remote conference calls – and a lot of the time, they suck. A bunch of bored people dialled into a Slack call or something but not really paying attention; glitchy video, distorted sound. But if we’re all going to be working from home and running virtual meetups for the next three months, we probably need to figure out how to make things better.

Earlier this week, the London Women in Agile group switched their regularly monthly meetup to a be a virtual online event. I joined as a participant and was absolutely blown away - it was orders of magnitude better than any video conference or remote meeting I’ve ever seen before. This was partly thanks to the speaker, Judy Rees, who in a stroke of serendipity was actually talking about how to have difficult conversations with remote teams – but who also brought a wealth of experience and expertise about how to actually run a large online meetup. The meetup was run over Zoom, and Judy used features like breakout rooms to mix up the format and keep people engaged - and it worked brilliantly.

I came away from this meeting - or rather, disconnected from the call - thinking that there’s clearly a lot more to modern video conferencing platforms than I realised, and I’m sure I’m not the only one. So earlier today I organised a virtual online lunch - put links up on Twitter, Facebook and a couple of Slack groups I’m in, just asking if anybody wanted to hang out on Zoom and eat lunch together and play around with the capabilities of the platform…

It worked great. Not perfect, but very, very good. I think everybody on the call saw a couple of new features they’d never seen before - the great thing about this kind of informal session is you can goof around and explore things; we were playing with virtual background, screen sharing, sharing jokes and links in the chat, folks helping each other figure out microphone muting settings and howto get stuff set up properly. And it was also a lot of fun - at the busiest point I think we had 32 people simultaneously online, and the platform handled it all pretty well.

Some specific things to think about:

At one point my home broadband dropped out for a minute. I switched my laptop to tether from my phone and reconnected - and, thankfully, everybody was still there; one of them had been promoted to the meeting organiser whilst I was disconnected, but it didn’t shut down the meeting or kick everybody out.
It actually worked pretty well via a tethered phone on 4G - not quite as good as broadband, but still pretty good.
A handful of participants were clearly on slower connections, and it makes a huge difference - in fact, I’d say if you’re going to try and participate in any kind of remote session, you absolutely need to get a connection that provides a solid 8Mbit+ of bandwidth. If you start breaking up whilst you’re speaking, or folks can’t hear you clearly, they’re probably just going to nod and smile and hope it wasn’t important - and that’s not a great experience for anybody involved.
The Gallery view works brilliantly - you can see everyone’s faces, you can see who’s talking, you can see who’s muted.

On the whole, it worked really well… so I’m going to do more. I’ll be using Zoom to run regular online lunches, drinks after work, that kind of thing – partly to give more people a chance to experiment with Zoom in a friendly context, but mainly because it’s just nice chatting to people in real time. (Yes, this means another hour or two of staring at screens instead of going outside… but hey, maybe outside isn’t all it’s cracked up to be…)

I’m also going to be running some experiments with virtual online events, to try out different formats and tools. A lot of big events this year are switching to virtual-only, and obviously a virtual event is never going to be the same as a physical one – but it’s also a chance to connect with a lot of folks who wouldn’t be able to attend the physical conference in person anyway. Besides, a lot of the people expressing reservations about virtual conferences are the same people who will happily spend hours playing Zelda or Minecraft, or sitting watching Netflix and then chatting about it on Twitter. It’s not the medium, it’s the content.

Wanna help us? We’ll be doing experiments, with Zoom and Twitch and YouTube and OBS and Slido and TwistCam. We’ll be fooling around with greenscreens and virtual slides. We’ll be running virtual meetups and online sessions, and we need content, we need participants, we need audiences who will honestly tell us all what works and what doesn’t, and bear with us while we figure all this out. If you want to participate in any of it, get in touch - email me, find me on Twitter, join my new Ursatile Slack team (yay another Slack! Just what you always wanted!) and get involved. Stock your fridge with snacks and drinks, figure out how to tether your phone in case your broadband goes wonky. Get a decent headset microphone - the mic built in to your laptop is not good enough for this, but even a pair of £15 earbuds with a cord microphone is probably good enough.

Oh yeah, and don’t forget to wash your hands.

Spleeter: Like Unbaking a Cake, but for Music

2020-02-25T12:00:00+00:00

I occasionally still get actual emails – you know, like letters, but sort of digital – that make me stop what I’m doing and go “oh, wow, that’s a thing now?” And so it was a few days ago, when my old friend and occasional musical sparring partner Megan dropped me an email containing the phrase “yes, Spleeter is the thing, it separates music into drums, bass, vocals, piano and everything else in exactly the way unbaking a cake works” – and accompanied by an absolutely sublime mashup of Tori Amos’ “Cornflake Girl” with Soundgarden’s “Black Hole Sun” created using said eldritch un-cake-baking technology.

So… I absolutely had to poke this thing around to see how it works. And I did. And I made a video. Check it out.

The track is one of my parody covers, “HTML” (a cover of AC/DC’s “Highway to Hell”), recorded entirely in Logic Pro X - software drums, live guitar, bass and vocals. I’ve then bounced the whole thing down to an uncompressed AIFF file, fed that through Spleeter using the 4stem preset to extract vocals, bass, drums and “everything else”; you can hear the results in the video. Spleeter is available under an MIT license and it’s on GitHub at https://github.com/deezer/spleeter - you’ll need to install Python and the Conda package manager, but I got it running on macOS without too much difficulty.

And yes. Living in the future is weird.

Look! I Made a Company!

2020-01-28T12:00:00+00:00

October 31, 2019 was an interesting date for several reasons. First, it was the date when the United Kingdom was supposed to leave the European Union. Well, one of many such dates. It didn’t. Second, it was Hallowe’en, and I had plans to fly to Cluj-Napoca, the regional capital of Transylvania, to speak at a software conference. Yep – flying to Transylvania on Hallowe’en! Pretty cool, right? #goth

Things didn’t quite work out that way. I found out around lunchtime on the 31st that Skills Matter, the company where I’d worked as CTO since 2017, was being placed into administration with immediate effect. Now, I’ve left jobs before. There’s a couple of months of notice and handover, time to plan your next move, figure out where you want to go, what you want to work on. Not this time. On Thursday I was a salaried employee; on Friday I was unemployed. Or “unexpectedly promoted to freelance”, if you prefer.

I took a late flight to Romania on Friday so I could still speak at Devoxx on Saturday – and by the time I got back to London on Sunday evening, I’d decided I wasn’t going to look for another job. Not that the idea of applying for every “rockstar developer” job on LinkedIn didn’t have a certain appeal… but it was definitely time to try something else. Respond to change, instead of following a plan.

I’ve always loved working with computers, but over the past five years I’ve also discovered that I love teaching. I love travelling, I love working with new people and unfamiliar technologies, seeing how different teams have solved the same problems. The sort of work I wanted to do didn’t seem particularly compatible with any kind of conventional job, so, like a lot of folks who work in IT, I decided to set up my own consultancy, specialising in training, software development and technology strategy. I set myself a three-month deadline, to get something up and running in time for NDC London – and here we are.

Everybody say hello to Ursatile Ltd, Company No. 12414586. That’s right, folks – I have a certificate and everything! I have a website, I have a name, I have a logo, and I have over 20 years’ experience working with computers (which are dumb, but occasionally do some very smart things) and with people (who are smart, but occasionally do some very dumb things) – in fact, the only thing I don’t have right now is clients. Now, I’m no business expert, but I’ve taken some very solid financial advice and apparently the whole “start your own company” thing works a lot better if you have some paying customers. So I’m gonna come right out and say it:

Hire me!

I can teach you to code in .NET and JavaScript. I can teach you about software architecture. I can teach you public speaking, how to write amazing presentations, and how to communicate more effectively with your customers and teammates. I can show you how to design a hypermedia API – ideally after helping you figure out whether you need one and what you’re going to use it for. I can help you grow your team, build your next prototype or untangle your legacy codebase.

If we think face-to-face is the best way to do something, I’ll travel. I grew up in Africa, I’m based in London, and I genuinely love airports. I’ve taught and spoken everywhere from Sydney to Siberia, I’ve met amazing people, I’ve connected with wonderful communities all over the world, and I’d love the chance to spend a couple of days with your team, helping you figure things out but also learning about your company, your technology and your culture.

Remote is good too. I run workshops about how to use Slack, email and online tools like Trello and GitHub effectively; I’m not only happy to work remote, I might even be able to make it more effective for everyone involved.

My calendar for 2020 is pretty open right now, and I’m doing introductory rates for new training courses for the next few months – but get in quick: according to my business plan, by July I’ll be booked solid until the end of 2025 and charging £75K for a five-minute Skype call, so get in touch and let’s book something in before that happens. ;)

So, yeah. My new website is at ursatile.com. That’s not really for you. I mean, take a look by all means, but you’re already here and you’re still reading this, so I figure we’re already cool. The website is so that search engines will find me, and people like your boss will hire me.

I’ll be at NDC London this week – and I’m hosting PubConf London on Friday if you can’t make it to NDC – so if you’re around, come and say hi, and if you’re not, you can always reach me on email at hello@ursatile.com. I’d obviously love to talk to anybody who’s interested in hiring me, but I also want to know what you’re all working on and what you think your challenges are going to be this year.

And finally, a huge thanks to all the people who’ve helped me get this far. It’s been a difficult few months, and there’s still a lot of stuff I need to figure out, but so far everyone I’ve turned to for advice and help has been absolutely lovely – so thank you, all of you!

Speaking and Playing at GOTO Oslo 2020

2020-01-17T12:00:00+00:00

You’ve all heard the phrase ‘real developers ship’ – but have you ever been on a real developers’ ship? I’m really excited to be speaking at GOTO Oslo 2020 – the conference that’s taking place on a ship! We’ll kick off on board the MS Crown Seaways in Copenhagen on Tuesday morning. At 4pm, there’s a break in the schedule while we set sail for Oslo. We arrive in Oslo on Wednesday morning, for another full day of conference sessions and masterclasses - and at 4pm we set sail back to Copenhagen. I’ll be presenting The Art of Code on Wednesday evening, and then joining The Linebreakers on stage for a live set of classic rock tunes with a software twist.

Early bird tickets are on sale until February 1st from https://goto-oslo.com/2020/registrations - it should be a really great event and I’d love to see you there.

Shaving the Jekyll Yak

2019-12-12T12:00:00+00:00

UPDATE: Thanks to the amazing power of Detective Twitter, I think we figured out what’s going on…

I use Jekyll and Github Pages for pretty much all my standalone websites these days, and I love it – the combination of static HTML, Markdown and YAML provides just enough data-driven behaviour to avoid lots of unnecessary duplication, but without any of the costs or overheads of running databases and server-side processing.

Until today. Today, dear reader, I hit a bump. And it starts, like so many things, at the pub. At PubConf, to be more exact. Todd Gardner is taking a bit of a break from travel in 2020, so I’m going to be running PubConf London at the end of January – which means I get commit access to pubconf.io (yay!) for the next two months. It’s built using Jekyll and hosted on Github Pages, and I’ve had a local version of the PubConf website running on my Macbook for the last 4-5 months without any hassles… except last week, I repaved my Macbook with a clean install of macOS Catalina. And apart from a couple of weird quirks that I’ve managed to isolate, it’s all good – including all my other Jekyll sites.

So I grab a fresh clone of the PubConf source tree, do the bundle install / bundle exec jekyll serve invocation, and… boom.

pubconf.github.io $ bundle exec jekyll serve
Configuration file: /Users/dylanbeattie/Projects/pubconf.github.io/_config.yml
NOTE: Inheriting Faraday::Error::ClientError is deprecated; use Faraday::ClientError instead. It will be removed in or after version 1.0
Faraday::Error::ClientError.inherited called from /Library/Ruby/Gems/2.6.0/gems/octokit-4.14.0/lib/octokit/middleware/follow_redirects.rb:14.
            Source: /Users/dylanbeattie/Projects/pubconf.github.io
       Destination: /Users/dylanbeattie/Projects/pubconf.github.io/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
jekyll 3.8.5 | Error:  wrong number of arguments (given 2, expected 1)

That’s not good. And I hadn’t even changed anything yet. So… here begins a protracted bout of yak-shaving.

First thought: perhaps macOS Catalina uses a different default version of Ruby, that’s no longer compatible with this particular Jekyll configuration. So I spend an hour or so installing rbenv, the version manager that lets you run different Rubies side-by-side. No luck – 2.4.5, 2.4.9 and the default 2.6.3 all produce the same result.

I Google the error message. Now, what I’m looking for here is something that’s recent – looks like various folks have had this error message over the years, but I’m trying to work out what might have changed with that stack recently that could be causing this error to start happening. I find this, which looks pretty promising, including this comment from 15 October:

I believe this is actually an issue with a change in sprockets 4.0.0:

I’ve never heard of Sprockets, but according to the internet “Sprockets is a Ruby library for compiling and serving web assets.” Now, there are two things here which I think are interesting:

The code for pubconf.io uses a plugin called jekyll-assets – which I don’t use on any of my other Jekyll sites.
jekyll-assets relies on Sprockets.
Sprockets released version 4.0.0 on October 8th.

At this point, I remember another detail that might be interesting: the PubConf site is built using TravisCI, which means there’s a separate build pipeline I can look at that’s nothing to do with my laptop. Now, here’s where it gets really interesting. I ran a fresh build on TravisCI, using the exact same source tree that built cleanly on November 8th -and boom. Failing build. But, interestingly, it failed with a different error message:

Configuration file: /home/travis/build/PubConf/pubconf.github.io/_config.yml
NOTE: Inheriting Faraday::Error::ClientError is deprecated; use Faraday::ClientError instead. It will be removed in or after version 1.0
Faraday::Error::ClientError.inherited called from /home/travis/build/PubConf/pubconf.github.io/vendor/bundle/ruby/2.4.0/gems/octokit-4.14.0/lib/octokit/middleware/follow_redirects.rb:14.
            Source: /home/travis/build/PubConf/pubconf.github.io
       Destination: /home/travis/build/PubConf/pubconf.github.io/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
  Liquid Exception: Liquid syntax error (/home/travis/build/PubConf/pubconf.github.io/_includes/head.html line 111): Unknown tag 'stylesheet' included in /_layouts/default.html
jekyll 3.8.5 | Error:  Liquid syntax error (/home/travis/build/PubConf/pubconf.github.io/_includes/head.html line 111): Unknown tag 'stylesheet' included 
The command "./_tools/build" exited with 1.

OK, so let’s take a closer look at what happens if I lock my local Ruby for this project to 2.4.5, the same version that’s used on TravisCI:

pubconf.github.io $ rbenv local 2.4.5
pubconf.github.io $ ruby --version
pubconf.github.io $ ruby 2.4.5p335 (2018-10-18 revision 65137) [x86_64-darwin19]
pubconf.github.io $ rm Gemfile.lock
pubconf.github.io $ gem install jekyll
pubconf.github.io $ gem install bundler
pubconf.github.io $ bundle
pubconf.github.io $ bundle exec jekyll serve

Now, it still didn’t work – but check this out: I’m now getting the same error locally as I get on TravisCI:

Liquid Exception: Liquid syntax error (/Users/dylanbeattie/Projects/pubconf.github.io/_includes/head.html line 111): Unknown tag 'stylesheet' included in /_layouts/default.html
jekyll 3.8.5 | Error:  Liquid syntax error (/Users/dylanbeattie/Projects/pubconf.github.io/_includes/head.html line 111): Unknown tag 'stylesheet' included 

This is progress! Because if my local system fails the same way as TravisCI does, then it stands to reason if I can get it working locally, the same fix will work on the Travis setup.

So… what the hell does Unknown tag 'stylesheet' included mean? Well, like before, there’s a bunch of search results for this, but none of them suggest anything that’s changed within the last few months.

But I did find this issue on the jekyll-assets Github repository, which includes this comment from envygeeks:

Our website is an unreliable source of documentation (right now, I’m working on adding that to my pipeline.. it’ll be a few weeks yet.) Yes, every tag but {% asset %} has been removed. As for the GitHub-pages issue, this issue belongs to them, we’ve not changed anything by way of integration with Jekyll, other than how we hook in, we have people who happily use Jekyll-Assets 3.x with Github Pages and have no problems.

The smoking gun here:

Yes, every tag but {% asset %} has been removed.

Now, this comment is from January 2018. Which suggests that the {% stylesheet %} tag that’s used in the PubConf code templates has been deprecated since the release of jekyll-assets v3.0.0, in November 2017.

Now this is seriously weird. Sure, it explains what’s broken, and the fix is easy – add gem "jekyll-assets", "~> 2.3.2" to the group: jekyll_plugins do section in the project’s Gemfile – but doesn’t give us any clues at all as to why this didn’t break until now. Because as far as I can tell, on 8th November 2019 – just over a month ago – TravisCI was quite happily running that build and ending with jekyll-assets version 2.something and everything just worked. Despite the fact it, apparently, should have stopped working on November 2017, two years earlier, when the first 3.x version of jekyll-assets was shipped.

So, the sprockets thing was a rabbit hole, it’s nothing to do with macOS Catalina, and chatting with Todd, turns out he’s got another site with an identical configuration, on a freshly-paved laptop, that’s pulling down jekyll-assets 2.4.0 without having to specify a version.

If anybody has any bright ideas as to what’s going on, I’d be really curious to hear them. But, as happens so often in the wonderful world of modern web development, I got better things to do than try to figure out what caused the weird impossible bug that’s not only now fixed, but according to all the available evidence should never even have happened in the first place…

UPDATE: THE SOLUTION

One of the best reasons I’ve found for writing blog posts like this is that you get a lot more eyeballs on the problem than just your own – and thanks to some sterling work by @shiftkey, we figured it out. Or at least came up with a pretty plausible explanation.

Travis CI has a 28-day bundler cache, which means if you run a build within 28 days of the previous build, it’ll reuse the same set of dependencies. And it’s been a busy few years for PubConf, with events taking place frequently enough that it’s entirely possible this is the first time since 2017 that 28 days has passed without somebody or something kicking off a Travis CI build. So when I kicked off that build yesterday, that was the first time in literally years that Travis has built the whole project from scratch – and so picked up v3.x of the jekyll-assets plugin.

But really, what’s at the root of all this grief is our old friend, the leaky abstraction. Jekyll and Github Pages offer a really simple, elegant solution for creating static websites – and most of the time, you can just run the handful of commands in the documentation, write your Markdown, git push it and everything works. You don’t have to know about Ruby versions and bundler and Gemfile.lock – all that stuff is supposed to be abstracted away so you can focus on writing content. And it all works great, right up until it doesn’t. We could probably have avoided this problem by locking the plugin to a specific version, or by committing Gemfile.lock to revision control – but there’s drawbacks to both of those approaches, and neither of them warrants any mention in the documentation for Jekyll or for the jekyll-assets plugin.

The other problem with abstractions is that the more complexity they’re hiding, the harder it is to figure out what’s going on when something stops working. Remember, this thing started out as:

jekyll 3.8.5 | Error:  wrong number of arguments (given 2, expected 1)

when the actual thing that went wrong was something closer to:

jekyll 3.8.5 | Error: you haven't run a fresh build in over two years and we've just picked up a major release of the jekyll-assets plugin that hasn't been used in this project before, and which is no longer compatible with the syntax that's used in your website templates. You'll either need to update your templates so you're using the new asset tags required by jekyll-assets 3.x, or modify your _config.yml to specify that you need version 2.x of the jekyll-assets plugin.

NDC Meetups in Portugal in January

2019-12-11T12:00:00+00:00

I’m going to be heading out to Portugal in January to speak at some free meetups – if you’re in Lisbon or Porto and want to kick off 2020 with some cool tech events, come along! I’m gonna be in Porto on January 7th and Lisbon on January 8th talking about the secrets to happy code. It’s a talk that covers a whole range of ideas, from psychology and neurochemistry to user experience design, logging and monitoring, and how we developers can use those ideas to ship better software and create happier teams. Plus pizza, drinks, awesome people and plenty of time to chat.

Thanks to NDC Conferences for putting the whole thing together, and to Farfetch for hosting. And if you like the sound of this, you should check out NDC Porto – two days of workshops and two days of conferences with a fantastic lineup of international speakers, all set in one of the most beautiful cities I’ve ever visited.

The January meetups are free, but you’ll need to register if you’re coming along:

And if anyone wants to join me for francesinhas after the Porto meetup, I’m definitely up for that. :)

Better Meetings with Agenda Defender

2019-12-02T12:00:00+00:00

Have you ever been to one of those meetings that has an agenda something like this?

13:00 Update from CEO (5 mins)
13:10 Update from CFO (5 mins)
13:20 Update on the Singapore project (Alex)
13:30 Update on the OAuth2 project (Chris)
13:45 Any other business
13:55 FINISH

Everyone arrives on time, you kick off at 13:00 as scheduled… but then after twenty minutes, the CEO is still talking, the agenda has been completely forgotten, and you’re all trying to work out whether they’re going to just skip everyone else’s updates so they can wrap up at 2pm as scheduled, or the one-hour meeting is just going to drag on and on until it’s taken up most of the afternoon and that’s the day wiped out?

I have. And, yes, on a handful of occasions I’ve been the person doing the talking – sorry! But sticking to an agenda when you’re talking is hard – like a lot of so-called “soft skills”, participating in meetings effectively is way harder than most people think. Being able to plan and deliver a five-minute update in five minutes, or a half-hour presentation in exactly half an hour, is a skill that takes time and effort to improve.

So I built a thing that can help. It’s called Agenda Defender, it’s at https://agendadefender.app, and it can help you and your team keep your meetings on schedule.

First, type (or paste) your agenda. Local times, 24-hour clock, one line per item, with the finish time as the last entry.

Click “Let’s go!” and it’ll turn each agenda item into a live progress bar. And that’s it.

Sure, one little web app probably isn’t going to fix dysfunctional communication styles overnight – if you work somewhere where long rambling meetings that wander all over the place are a regular occurrence, introducing something like this might come as a bit of a shock. First few times you try it, you’ll get some resistance – especially from those people who think having a slot on the agenda means they can talk for as long as they want. But if we’re going to improve, we really need two things – feedback, so we know what we’re doing wrong, and incentive, so we have a reason to pay attention to the feedback. And once you’ve tried running meetings with Agenda Defender a few times, folks will get the hang of it – the live schedule will make it easier to pace yourself when you’re speaking, and the fact everybody can see who’s hitting their scheduled slots gives everybody an incentive to keep things on track. And if you do need to reschedule, or shuffle things around mid-meeting, no problem – just edit the agenda and start again.

Tired of meetings that should have been an email? Fed up with delivering features that you thought were perfect, only to hear “um… that’s not quite what we wanted”? Wasting time on writing documentation nobody’s ever going to read? Check out my Communication for Developers workshop, and find out how I can help you and your organisation build the right thing, faster, and go home smiling when you’re done.

About That Meetup.com Announcement

2019-10-15T12:00:00+00:00

Remember: everything on this site, including this post, is my personal opinion, and does not reflect any official position of either Skills Matter or of the London .NET User Group.

https://www.meetup.com/lp/paymentchanges, 15 October 2019

The online community platform Meetup.com (hereafter known simply as Meetup) has just announced that “beginning in October, members of selected groups will be charged a small fee to reserve their spot at events”. (archive.org link) From the phrasing of their announcement, it sounds like Meetup will be charging people a $2 fee to register to attend an event – even if the event is free – and that 100% of that revenue will go to Meetup and none of it is passed on to the event organisers.

I’ve organised a .NET meetup group in London since 2016. Since 2018, I’ve also been CTO at Skills Matter, a company based in London that hosts hundreds of free meetups and tech events every year. That’s given me some fairly unique insights into the relationship between the Meetup platform, the people who organise and attend free community events, and the logistics behind running a free event.

Meetup is an odd platform. It’s a lowest common denominator, that’s used by songwriting clubs, running groups, coffee mornings and book clubs, as well as the tech meetups that we all know and love. And over the last few years, they’ve made changes to that platform that’s made it harder to run a tech meetup on their platform. For example, they’ve deprecated their rich text editor in favour of a minimal plain-text interface that makes it much more difficult to include speaker biographies and photographs in an event description.

Many people also aren’t aware that group organisers have no access to our group members’ contact information. They provide tools that let us contact attendees by email, but there’s no way to download a list of members’ email addresses. And it costs. The London .NET User Group is part of the .NET Foundation, which means they pay our subscription costs, but even running free events on their platform costs $19.99 per group, per month.

As a technology platform, it’s not great. It has mediocre usability and a very limited feature set. We can’t share slides or videos. There’s no way to share feedback with our speakers. There’s no facility for attendees to share their dietary requirements. There’s no sort of statistical modelling or forecasting to give organisers some idea of how many people will actually show up – I know of people in the LDNUG group on Meetup who have RSVPed ‘yes’ to every single event since 2016 and have never actually showed up to one, but it will still quite happily list them as ‘attending’ every single time, so good luck figuring out how much beer and pizza you’re going to need on the night. Patricia Aas very neatly summarised it thus:

This is basically three things:
1) A mailing list
2) A RSVP/waiting list
3) An event page @Meetup does none of these things well and already is way too expensive...
— Patricia Aas (@pati_gallardo) October 14, 2019

But we still run London .NET on Meetup, and here’s why. When I announce one of our events on Meetup, I know I’ll get 50-100 RSVPs within 48 hours. If it’s a well-known speaker, I’ll get more like 200. And around 20% of those RSVPs will be complete strangers – people who have never been to one of our events before, who aren’t on any of our Slack chats or following us on Twitter. The only way those people even know that our event is even happening is that they found it on Meetup – and Meetup is so dominant in this space that if your event’s not on Meetup, it may as well not exist.

The recent announcement here about charging attendees to register is interesting. Now, $2 is not a big deal for tech events – here in London that’s significantly less than a cup of coffee. And I wouldn’t be surprised if it’s part of some long-term strategy by Meetup to pivot to a subscription model by jacking the prices up now, and then offering a subscription model in a few months time that looks very reasonable by comparison (“Hey, check out Meetup Unlimited – only $5/month to register for as many events as you want!”). After all, Meetup is owned by WeWork, who are losing vast amounts of money right now and probably desperate for revenue; I suspect if you took a look at their business plan, free meetup attendees are listed under ‘stuff that doesn’t make enough money’. There’s a great write-up by Quincy Larson, the founder of freeCodeCamp.org, about this as well.

But the impact of that change, from free to $2, is far more psychological than financial – and that’s why it might be a game-changer. The announcement has already inspired people around the world to start working on their own version of Meetup – I’m on a Discord channel right now where the FreeCodeCamp folks are coordinating one such solution, and over 100 people have joined that channel in the time it’s taken to write this post. Only time will tell whether this is a flash in the pan or genuine critical mass, but their solution is called Chapter, it’s over at github.com/freeCodeCamp/chapter, and it’s definitely worth keeping an eye on.

Now, I love the way the tech community rallies like this around an opportunity – but we really don’t need a dozen different open-source codebases hosted on hundreds of isolated websites, each of them running a handful of local groups. And when you get into mobile apps, where you can’t just spin up a Docker container and import your data, the fragmentation problem is even more pronounced. The problem that Meetup solves is not a hard technical problem. It’s a social problem. If people can’t find your group, nobody’s going to come to your event.

That’s not a problem you can solve with database schemas and JavaScript frameworks. Sure, tech plays a role here – but if there’s one thing we can all learn from Meetup, it’s that mindshare and user adoption are way more important than features and design when it comes to growing user groups, events and communities. People will put up with poor usability and terrible design if it connects them to the right people – and if it doesn’t, there’s no amount of shiny technology and beautiful UX that’s going to make up for it.

I’ll be really interested to see what happens next.

Publishing Talks

2019-10-06T12:00:00+00:00

I used to blog a lot more than I do now. There’s a couple of reasons for that, not least that a lot of the discussion and engagement that used to happen on blog post comments has now moved to Twitter and StackOverflow. But mainly, it’s because these days almost all of my ‘writing’ time goes into creating talks for conferences and user groups - and that takes a huge amount of time.

Most of my talks start life as an outline - a title and a very broad overview of what I want to talk about - and an Evernote file full of code snippets, web links, observations and notes and scribblings. The first stage of turning that into an actual talk is to sit down and write it. I find it much, much easier to develop, edit and restructure ideas when they’re still text, before I’ve got as far as any slides or visuals or anything - and that means ending up with a written version of pretty much the entire talk, or one possible version thereof.

This is not an article or a report. It’s written for a very different audience, and intended to create a very different kind of engagement. When you’re talking on a stage, you have a connection with the audience. If something doesn’t work, you can normally tell immediately and can explain, clarify, apologise if necessary - live presentation is a daunting medium, but it’s also a very forgiving one if you know how to work with it.

It’s also not a script. Every time I deliver a talk, it’s different - there will be phrasing and jokes and references in the written form that get skipped because of timing, or because they didn’t fit with the context of a particular event, or because I just plain forgot to mention them. It happens.

It might help to think of it as an essay, in the original sense of the word:

Essayer is the French verb meaning “to try” and an essai is an attempt. An essay is something you write to try to figure something out.

Figure out what? You don’t know yet. And so you can’t begin with a thesis, because you don’t have one, and may never have one. An essay doesn’t begin with a statement, but with a question. In a real essay, you don’t take a position and defend it. You notice a door that’s ajar, and you open it and walk in to see what’s inside.

- Paul Graham, The Age of the Essay, 2014

Now, I’ve written five new talks this year - six, if you include collaborations. I write the way I talk, and I talk at about 200 words per minute - so a 60-minute talk is roughly equivalent to writing a 12,000-word essay. And that essay then forms the backbone of the talk itself - the structure, the flow and the narrative; it’s what I use to create slides, work out where to use code samples, animation and video. But until now, I’ve never actually published it.

So I’m going to try an experiment. I’ve taken one of the talks I prepared this year - a talk called ‘The Cost of Code’ - and cleaned it up a bit. I’ve added most of the slides used in the live presentation, incorporated links to my source material, and put it all online.

It’s at https://dylanbeattie.net/articles/the-cost-of-code/ - check it out, and let me know what you think.

Migrating from Blogger to GitHub Pages

2019-08-14T12:00:00+00:00

So, you might have noticed I’ve done a bit of decorating… welcome to the 2019 incarnation of dylanbeattie.net. If you’re interested in how I migrated ten years’ worth of blog posts onto a Jekyll site hosted on GitHub Pages, read on. If you’re not, there’s funny videos over on /music that you might enjoy.

Still here? Cool! OK, so the very first version of dylanbeattie.net was a site I built in PHP back in 1999 or so, which I hosted on my own physical server – way back in the days when both of those things were still considered a pretty neat idea. About ten years I ago I started using Blogger, and before long I scrapped the old PHP site, pointed the main domain at my Blogger site, and just sort of got on with it.

Blogger’s been a pretty good platform, but these days I find I actually need a website to do a lot more than just host blog posts and the occasional ‘about me’ page, so a few months back I started looking around for alternatives. I’d considered setting up something like Umbraco, or a custom Wordpress site, but even those felt a bit like over-engineering for what I actually needed to do. My requirements basically boiled down to:

A really easy way to post articles – text and links with some simple formatting and the occasional image
The option to use rich HTML pages and custom layouts
Preserving the URIs of all my existing posts and pages – remember, cool URIs don’t change. Besides, I’ve got a decades’ worth of Google-fu invested in those pages. It’d suck if all those links and bookmarks stopped working.
Something that looked good, and a responsive layout that worked well across devices.

There’s also a couple of things I decided I definitely didn’t want:

Comments. It’s been years since I saw a decent discussion in the comments thread of a blog post. We have Twitter and Reddit for that now.
Bootstrap. I’m sure it’s lovely. I just don’t like it.

Back in January, I was chatting with Todd Gardner about how he created the PubConf website, and he suggested I take a look at GitHub Pages and a thing called Jekyll. I spent an evening playing around with it, and was basically hooked.

Jekyll and GitHub Pages

Note: the code for this site is available on GitHub at github.com/dylanbeattie/dylanbeattie.net

So here’s how it works. You write your site in HTML or Markdown, and create datasets in YAML. Jekyll compiles them into a static HTML site. It supports a templating language called Liquid, which lets you create templates, conditionals, navigation – you can actually do some pretty sophisticated stuff with it, but everything happens at build time. Which is actually very cool.

The really great part is that GitHub Pages has built-in support for Jekyll – so you build your site locally, using bundle exec jekyll serve to view it, then you just push the repo to GitHub and it’ll build and deploy it for you. It’s a beautifully simple workflow, but one that affords a surprising amount of flexibility once you get the hang of it.

Here’s some fun little things I learned along the way, and a couple of gotchas to watch out for if this inspires you to try something similar.

Design and layout

The design for my new site is the Arcana template by HTML5 UP, which is not only beautiful, responsive and really well put together, but is also released – like all HTML5 UP’s templates – under a Creative Commons Attribution 3.0 License:

“…which means you can use them for personal stuff, use them for commercial stuff, change them however you like … all for free, yo. In exchange, just give HTML5 UP credit for the design and tell your friends about it :)”.

Which is lovely. You see, I’m quite happy doing my own design work – and that’s the problem. I enjoy it. I get sidetracked. I spend hours – days – tweaking layouts and playing around with things, when I should be writing content and fixing bugs… and I end up with something that’s OK, but nothing like as good as some of the amazing templates and layouts that are available online, for free. So credit (and thanks!) to HTML5 UP for the design. I’ve made a couple of tweaks and added a few extra bits, but the layouts, grid, responsive design, navigation, typography – that’s all them. Oh, and it’s all built on SASS – which is also supported by GitHub Pages. Did I mention how much I love this platform?

Migrating old blog posts

This turned out to be a lot easier than I expected. There’s a Ruby gem designed to do exactly this – check out the Jekyll documentation and this blog post from Kris Rice which goes into a bit more detail about it.

Once I’d migrated all the posts from my old Blogger site, I wanted to make sure all the old page URLs would still work. I didn’t want to mess too much with Jekyll’s conventions about structuring posts, though – Jekyll uses a YYYY-MM-DD-title format for URLS compared to Blogger’s YYYY-MM-title format, and so I simultaneously wanted to adopt the Jekyll convention for new posts, to make things easier, but also to preserve the addresses of all my old posts so bookmarks and links still work.

Turns out the jekyll-migrate gem adds some lines to the top of each migrated HTML file – what Jekyll refers to as ‘front matter’ – that can help:

---
layout: post
title: How to *really* break the internet.
date: '2016-03-23T12:08:00.000Z'
author: Dylan Beattie
tags: 
modified_time: '2016-03-23T14:06:37.095Z'
blogger_id: tag:blogger.com,1999:blog-7295454224203070190.post-8996121207980120854
blogger_orig_url: http://www.dylanbeattie.net/2016/03/how-to-really-break-internet.html
---

There’s also a Jekyll plugin called redirect-from, which is part of the github-pages bundle (and, yes, it works a little like the infamous COMEFROM flow control statement beloved of sadistic esolang designers). So all I had to do was trawl through every file in the _posts folder, find that line with the blogger_orig_url in it, parse the path part out of the URL, and add a line underneath like this:

blogger_orig_url: http://www.dylanbeattie.net/2016/03/how-to-really-break-internet.html
redirect_from: "/2016/03/how-to-really-break-internet.html"

(Sorry, awk fans – I actually did this with a global search & replace in VS Code…)

That’s it – Jekyll now hosts all those pages at their new /2016/03/23/how-to-really-break-internet.html addresses, and requesting the original URL returns this:

<!DOCTYPE html>
<html lang="en-US">
<meta charset="utf-8">
<title>Redirecting&hellip;</title>
<link rel="canonical" href="/2016/03/23/how-to-really-break-internet.html">
<script>
  location="/2016/03/23/how-to-really-break-internet.html"
</script>
<meta http-equiv="refresh" content="0; url=/2016/03/23/how-to-really-break-internet.html">
<meta name="robots" content="noindex">
<h1>Redirecting&hellip;</h1>
<a href="/2016/03/23/how-to-really-break-internet.html">Click here...</a>
</html>

It’s not quite perfect – an HTTP 301 Moved Permanently would strictly speaking be a better way of handling this – but hey, perfect is the enemy of good enough. It works. Ship it.

Events schedule and flags

You see that little sidebar there, with all the events I’m speaking at and the flags in it of the countries I’m gonna be visiting? The flag images are from GoSquared, and available under an MIT license. To display them in the schedule, I’ve used a SASS list to generate CSS rules for each flag in the set:

/* _flags.scss: generate .flag-xx CSS classes for elements with flag backgrounds */

$countries: _abkhazia, _basque-country, _british-antarctic-territory, _commonwealth, 
  _england, _gosquared, _kosovo, _mars, _nagorno-karabakh, _nato, _northern-cyprus, 
  _olympics, _red-cross, _scotland, _somaliland, _south-ossetia, _united-nations, _wales, 
  AD, AE, AF, AG, AI, AL, AM, AN, AO, AQ, AR, AS, AT, AU, AW, AX, AZ, BA, BB, BD, BE, BF, 
  BG, BH, BI, BJ, BL, BM, BN, BO, BR, BS, BT, BW, BY, BZ, CA, CC, CD, CF, CG, CH, CI, CK, 
  CL, CM, CN, CO, CR, CU, CV, CW, CX, CY, CZ, DE, DJ, DK, DM, DO, DZ, EC, EE, EG, EH, ER, 
  ES, ET, EU, FI, FJ, FK, FM, FO, FR, GA, GB, GD, GE, GG, GH, GI, GL, GM, GN, GQ, GR, GS, 
  GT, GU, GW, GY, HK, HN, HR, HT, HU, IC, ID, IE, IL, IM, IN, IQ, IR, IS, IT, JE, JM, JO, 
  JP, KE, KG, KH, KI, KM, KN, KP, KR, KW, KY, KZ, LA, LB, LC, LI, LK, LR, LS, LT, LU, LV, 
  LY, MA, MC, MD, ME, MF, MG, MH, MK, ML, MM, MN, MO, MP, MQ, MR, MS, MT, MU, MV, MW, MX, 
  MY, MZ, NA, NC, NE, NF, NG, NI, NL, NO, NP, NR, NU, NZ, OM, PA, PE, PF, PG, PH, PK, PL, 
  PN, PR, PS, PT, PW, PY, QA, RO, RS, RU, RW, SA, SB, SC, SD, SE, SG, SH, SI, SK, SL, SM, 
  SN, SO, SR, SS, ST, SV, SY, SZ, TC, TD, TF, TG, TH, TJ, TK, TL, TM, TN, TO, TR, TT, TV, 
  TW, TZ, UA, UG, US, UY, UZ, VA, VC, VE, VG, VI, VN, VU, WF, WS, YE, YT, ZA, ZM, ZW;

@each $country in $countries {
  .flag-#{to-lower-case($country)} {
    background-image: url(images/flags/flat/64/#{$country}.png);
  }
}

The actual schedule is a YAML file, and Jekyll generates the markup with the correct CSS classes based on the country codes from schedule.yml. It’s easy to add new dates. The only thing it won’t do is automatically archive old ones- ‘cos hey, static content, remember? – but I reckon I can live with that for now.

(And yes, you can absolutely invite me to speak at your event by editing schedule.yml and sending me a PR. That would be really cool. :) )

Speaker bios

I’ve got a bunch of different speaker bios over at my about me page, and I really wanted a way to pick one and copy it to the clipboard as either HTML, Markdown or plain text – different events use different formats when you’re submitting to their CFP or filling out speaker details, and it’s a little thing that would make life easier.

The bios themselves are stored as multiline Markdown snippets in /_data/speaker_bios.yml. Instead of allowing Jekyll to just render Markdown > HTML automatically, I’ve got this little loop in the code for the about me page:

{% for bio in site.data.speaker_bios %}
<article>
  <hr />
  <h3>
    <span class="clipboard-links">
      <a href="#" data-src-id="{{bio.id}}-html">copy html</a>
      <a href="#" data-src-id="{{bio.id}}-markdown">copy markdown</a>
      <a href="#" data-src-id="{{bio.id}}-text">copy text</a>
    </span>
    {{ bio.word_count }} Word Bio ({{ bio.char_count }} characters)
  </h3>
  {{ bio.content | markdownify }}
	<input type="hidden" id="{{bio.id}}-markdown" 
	  value="{{ bio.content }}" />
	<input type="hidden" id="{{bio.id}}-html"
	  value="{{ bio.content | markdownify | escape_once }}" />
	<input type="hidden" id="{{bio.id}}-text"
	  value="{{ bio.content | markdownify | strip_html }}" />
</article>
{% endfor %}

so each snippet is rendered to the page as HTML (via the markdownify filter), and also captured in three hidden variables – one markdown, one HTML, one plain text. Finally, there’s some JavaScript attached to the ‘copy xxx’ links that’ll copy the hidden input value into an invisible textarea and copy it.

Syntax Highlighting

One of the great things about writing posts and pages in Markdown is that it’s so easy to embed code samples - just wrap them in three backticks either side, something known as a fenced code block. Jekyll has a code formatting plugin called Rouge, that’s included with the github-pages bundle, which allows you to specify a language for your code snippets and it’ll highlight them for you:

``` html
<p>This HTML snippet will get <a href="http://rouge.jneen.net/">highlighted</a></p>
```

To get this to work, I had to enable highlighting in _config.yml

markdown: kramdown
# enable rouge syntax highlighting
highlighter: rouge

I also had to add a stylesheet to the site with the various coloring rules - all Rouge does is wrap all the code keywords, etc. in <span> tags with CSS classes on them, so I found this syntax.css file on GitHub, dropped it into my site’s CSS, and it worked.

Gotchas

Pretty much everything I tried to do with Jekyll and GitHub Pages worked as documented, but I did hit two weird gotchas that caused a fair bit of head-scratching until I figured out what was going on…

Case sensitive filesystems

I’ve done most of the dev work for this site locally, on macOS 10.14, using bundle exec jekyll serve to preview and test things. And when I finally pushed the whole thing up to GitHub and switched on the GitHub Pages feature, everything worked – except the flag images in the schedule sidebar. And it took me a good couple of hours to figure out what was going on.

My original flags.scss file looked like this:

$countries: ad, ae, af, ag, ai, al...

@each $country in $countries {
  .flag-#{$country} {
    background-image: url(images/flags/flat/64/#{$country}.png);
  }
}

which generated a bunch of CSS rules that looked like this:

.flag-ad { background-image: url(images/flags/flat/64/ad.png); }
.flag-ae { background-image: url(images/flags/flat/64/ae.png); }

Now, if you look closely at the files in the GoSquared flagset I’m using, you’ll notice the filenames are:

AD.png
AE.png
AF.png

I’d dropped their entire flag set into my project, and pushed the whole thing to GitHub.

Now, I’m guessing that GitHub Pages is hosted on Linux. And Linux uses case-sensitive filesystems, whereas macOS – where I’d been testing everything – uses a ‘case preserving’ filesystem. In other words, if you ask macOS for ae.png, it’ll happily give you AE.png, but on Linux, ae.png and AE.png are different files.

So my CSS rule was telling my browser to ask GitHub Pages for ae.png, and GitHub’s Linux servers are going “nope. Not found.” – ‘cos the only file they’ve got is AE.png, which is COMPLETELY DIFFERENT. Obviously.

I could see two ways to fix this. One was to rename all the files in the GoSquared collection… but it turns out it’s surprising fiddly to rename a file from FILENAME to filename on macOS. The other, easier way is just to uppercase all the ISO country codes in the SCSS list, like this. I threw a to-lower-case($country) into the class name there because, well, I think uppercase CSS rules are vulgar.

$countries: AD, AE, AF, AG, AI, AL...

@each $country in $countries {
  .flag-#{to-lower-case($country)} {
    background-image: url(images/flags/flat/64/#{$country}.png);
  }
} 

DNS, HTTPS, Cloudflare and GitHub Pages

The last piece of the puzzle was to get the whole site using dylanbeattie.net (no www) as the canonical URL, and to get everything running over HTTPS. I host all my DNS with Cloudflare, because it’s free and works really well – and, like a lot of people, I’d relied on Cloudflare’s HTTP+DNS proxy service for a while to provide HTTPS for my old Blogger site (check out Troy Hunt’s Here’s Why Your Static Website Needs HTTPS for more on why this is a good idea.)

I wanted to use GitHub Pages to enforce HTTPS, but when I switched the Cloudflare DNS for dylanbeattie.net to point to GitHub’s servers, I couldn’t switch on the option to enforce HTTPS – instead, I got this error:

Enforce HTTPS — Unavailable for your site because your domain is not properly configured to support HTTPS

and when browsing my new site, I got a warning that:

This server could not prove that it is dylanbeattie.net; its security certificate is from www.github.com. This may be caused by a misconfiguration or an attacker intercepting your connection.

After about a day of head-scratching – change a DNS setting, wait six hours to see if anything happens, change something else, repeat – I contacted GitHub Support.

Turns out that Cloudflare’s DNS+HTTP Proxy feature actually interferes with the certificate issuing mechanism used to support HTTPS on GitHub Pages. GitHub asks for DNS servers so it can issue a certificate, but if you’ve enabled Cloudflare’s HTTP proxy feature (which is on by default, even on their free plan), Cloudflare responds to the DNS query with its own server addresses and so GitHub can’t see that your domain is pointing at GitHub Pages.

Logged into Cloudflare, switched the records for dylanbeattie.net over to DNS only, and boom – certificate was issued within the hour and everything was up and running.

Saint-Petersburg, Russia - Tips for Tech Travellers

2019-05-14T09:42:00+00:00

As I'm heading back to Saint-Petersburg for DotNext 2019, this seemed like a nice moment to repost something I wrote after my first visit to Russia back in 2017.

1. As a guest in Russia, it is vitally important to keep moving at all times. This is because if you stop moving for more than, say, fifteen seconds, your hosts will assume this is because you have run out of roast pork and sausages. It is impossible to sit down at a table in Saint Petersburg without somebody serving you a massive plate of meat. My hosts explained that the Russian phrase for 'no thankyou' is 'nyet, spasiba" but based on my experience I think this literally translates as "please bring me roast duck and cabbage now, and after that some more sausages".

2. Sausages in Russia occupy the culinary niche that is normally reserved for, say, some carrots in Western cuisine. You will find sausages chopped in a salad, boiled, fried, steamed, and served as accompaniments to all sorts of things.

3. Saint Petersburg is BIG. The average street here is wider than most London postcodes. If you have to walk three blocks, wear good shoes and take a bottle of water with you. Monuments and war memorials here are built to such a scale I can only assume they are intended to leave civilisations in neighbouring star systems in no doubt as to the nobility and sacrifice of the Russian military. We should give a special mention to the churches, which are not only huge, but we can conclude from the style of their decorated domes and minarets that the builders thought God had a bit of a thing for cupcakes.

4. Riding the Saint Petersburg metro will seem uncannily familiar to anybody who has read Jules Verne's "Journey to the Centre of the Earth". After buying your metro token from the ticket machine and passing through one of the metal detector arches - for which you are not required to empty your pockets or anything, meaning that they go off constantly and are consequently ignored by everybody including the police - you step onto an escalator approximately fourteen miles in length. The tunnels and station interchanges suggest that tunnelling machines in the former Soviet Union were available in two sizes - Extra Large and Stupidly Extra Large - and the interchanges are so big that the connected stations have different names. If, say, the Red Bull Air Race ever needed to be held indoors due to inclement weather, the pedestrian interchange tunnel at Spasskaya/Sadovaya would provide an ideal venue.

5. For typography enthusiasts, the Cyrillic alphabet is an absolute delight... except when it comes to handwriting. Cursive Cyrillic is a minefield of hilarity and ambiguity. Any doctors who feel they have exhausted the possibilities of the Latin alphabet when it comes to writing illegible prescriptions will find Cyrillic a rich seam of possibility.

6. The Russian people are lovely and friendly... once you get used to the fact that a Russian telling you a joke will initially sound like they're interrogating about some war crimes you may or not have committed. It helps to keep a Polish person with you, since they seem to know the correct point to start laughing, thus giving a handy cue to the slightly baffled English speakers participating in the conversation.

7. An English person trying to speak Russian is the funniest thing that has ever happened. The Russian equivalent of the Edinburgh Festival consists entirely of English people attempting to pronounce the names of Saint Petersburg metro stations whilst the audience drink vodka and roar with laughter.

8. Vodka must be served no warmer than -273.1499 degrees Celsius. To offer someone vodka that is merely refrigerated could cause a serious diplomatic incident.

9. Most consonants in Russian have a 'hard' and a 'soft' pronunciation, which, like tonal Cantonese or the tongue-clicks of the Khoisan language family, is completely impenetrable to foreigners. It is very important, however: based on my attempts to speak Russian to waiters, I have concluded that the elusive hard vs soft 'T' sound must be the difference between saying "no thank you, I have eaten so much food I think I need to to go the hospital" and "could I please have some more roast pork and boiled sausages"

10. There are a lot of Chinese tourists in Saint Petersburg. Local regulations prohibit them from travelling in groups of fewer than fifty. If you arrive to check in to your hotel moments after a Chinese tour party has arrived, you may wish to pass the time whilst you wait by reading the collected works of Dostoyevsky or walking to Vladivostok and back.

11. Every single car in Russia has a dashboard camera recording video footage of the journey, presumably so that when your cab gets cut up by another one and causes a six-car pile-up, the driver can pay the repair bills by sharing the crash footage on YouTube and hoping it goes viral. Most cab drivers keep their radio tuned to the local high-energy Europop station, so when they do inevitably have a massive crash, the resulting YouTube footage already has the appropriate soundtrack.

In short... it was AWESOME. The city is beautiful and vast and unlike anywhere I have ever been, the people could not have been more welcoming and friendly; arrival and departure was an absolute breeze thanks to the brand new, hyper-modern airport terminal building, and the metro is a great way to get around (and clearly signed in English throughout.) And if you don't fancy jumping through the administrative hoops of getting a Russian visa, you can visit SPB on a cruise ship from Tallinn or Helsinki and stay for up to 72 hours without having to get a visa, which is kinda cool.

Just don't stay up drinking vodka the night before your 5am departure. Trust me on this. :)

A release notes bookmarkdownlet for Pivotal Tracker

2019-03-22T18:58:00+00:00

One of the best ways to keep the rest of your team up to speed with what your dev teams are doing is release notes - even if all you're doing is gently reassuring the rest of the organisation that yes, you are patching security vulnerabilities, fixing bugs and quietly making things better.

I love using Slack for this - set up a channel where you post a friendly summary of everything that's being released whenever you deploy to production. Now, here at Skills Matter we're not quite doing continuous deployment yet - we work off short-lived branches that merge to master several times a day, but then once master has passed regression testing on our staging environment, the actual deployment to production is a manual process - we open a master > production pull request in GitHub, merge it, and Heroku does the rest.

We track work in progress using Pivotal Tracker, and we use the various ticket state transitions as:

Start > a developer has created a branch and begun coding the features
Finish > the code is done; time for code review
Deliver > the code has been reviewed, merged to master and deployed to the staging environment
Accept > the code has been tested on staging; stakeholders know it's ready, and we're good to go live.

Now, this definitely isn't the best branch/merge strategy in the world, but it's the one we've inherited and the one we're using until we've made enough changes to the codebase to be able to deploy PRs directly to review environments.

So when we do a production release, one of the things I do is to check which features are included in that release - that'll form a note that's part of the master > production pull request, and we'll also share it with the company via Slack. And this is a bit tedious, so today I threw together a little JavaScript bookmarklet that'll automate it for you.

Select the stories you want in Pivotal Tracker, click the bookmarklet, and it'll copy them to your clipboard as Markdown-formatted bullet points with the story IDs linked to your Pivotal project.

The JS code is here - add a bookmark, paste this whole lot (including the javascript: into the URL field:

And here's what it looks like in action:

PostgreSQL, Heroku, .NET Core and npgsql

2018-11-20T16:17:00+00:00

I've been having fun this week building data visualisation dashboards that pull information straight out of our PostgreSQL databases and use ASP.NET Core to do some LINQ transformations and aggregation on the data. All our data is hosted on Heroku, I'm using Npgsql as an ADO.NET Data Provider, and it works beautifully - once you've worked out the exact connection string syntax needed to connect to a Heroku PostgreSQL database using ASP.NET Core.

So here it is. You'll need to get the host, port, username, password and database from your Heroku dashboard - and don't forget that if you're connecting from apps that aren't attached to Heroku directly, you'll need to manually update the configuration if you rotate your Heroku database credentials.

const string herokuConnectionString = @"
  Host=<host.domain.com>;
  Port=<port>;
  Username=<user>;
  Password=<password>;
  Database=<database>;
  Pooling=true;
  Use SSL Stream=True;
  SSL Mode=Require;
  TrustServerCertificate=True;
";

Happy querying!

The Horrors Lurking in your Legacy Codebase

2018-08-29T10:06:00+00:00

We’ve all come across design patterns, right? Common solutions to common problems, more specific than a language or a platform, less prescriptive than a component or a framework. The pattern movement originated in building architecture, and in the years since the Gang of Four published their groundbreaking work Design Patterns: Elements of Reusable Object-Oriented Software, we’ve seen patterns embraced right across the spectrum of software development. We’ve got architectural patterns and infrastructure patterns and organisational patterns. We’ve even got anti-patterns.

Patterns are intentional. They have purpose. Even anti-patterns generally involve some element of premeditation. But patterns aren’t the only things in software where you’ll see familiar structures and characteristics playing out across different systems and organisations… There are also the emergent phenomena. The creeping horrors that we’ve all unwittingly summoned in the course of our illustrious careers. Things that don’t happen on purpose, that nobody ever set out to do, and yet which keep spontaneously manifesting in organisations all over the world. Here, for your education and entertainment, I present a bestiary: a field guide to the monsters and the creeping horrors that are lurking somewhere in your IT systems.

1. The Reliquary

The reliquary is that one repository full of really good ideas. Clean code. Brilliant algorithms. The OpenID implementation that you optimised until it shone. Classes so beautifully designed and perfectly documented that they’d make a senior architect weep.

You remember the big rewrite? The project that was going to fix everything, only you never worked out how to actually launch the thing, or get any revenue from it? The reliquary is where you’ve preserved it, pickled in revision control like a fabulous museum specimen. A treasury of good code and good ideas; maybe even an entire codebase that was “a couple of weeks” away from shipping before somebody finally looked at the number of critical features the team had somehow forgotten to include and discovered — to everybody’s surprise — that validated XHTML, normalised data models and 95% test coverage are not actually features any of your end users cared about. Like Buran or the Spruce Goose, the surviving artefacts stand as a testament to the quality of your engineering… and a poignant reminder of just how much fun engineers can have building high-quality stuff that nobody actually wants to use.

2. The Doctor Gonzo

Named for the attorney in Hunter S. Thompson’s ‘Fear and Loathing in Las Vegas’, the Doctor Gonzo is that application that’s “too weird to live, and too rare to die”. It’s written in Visual Basic 6, or Delphi, or maybe even Microsoft Access. Your dev team has to keep a couple of antique-grade virtual machines around to fix the occasional show-stopping bug — with instructions that the machines are absolutely not to be Windows Updated on pain of immediate defenestration.

Of COURSE you’ve tried to replace it. Team after team, project after project has come up with a plan, hired some contractors, captured some requirements, and shipped a couple of prototypes. And, as inevitably as night follows day, they have failed… all of their engineering brilliance powerless against the unholy triumvirate of bureaucracy, Stockholm syndrome and undocumented use cases.

In fifty years time when we’re all running genetic algorithms on bioengineered quantum hardware that eschews physical user interfaces in favour of superimposing consciousness patterns directly into our brains by inducing cross-dimensional electrical fields in a neighbouring parallel universe, at least one company will have made a fortune creating a post-singularity hosting environment for running Visual Basic 6 line-of-business applications in the quantum realm.

3. The Epic of Gilgamesh

You know this one. It started out as a simple database query — something that pulled out the sales figures for the last quarter. Then somebody tweaked it to account for currency fluctuations. Somebody else cross-referenced it against website traffic logs. Somebody else added a half-dozen LEFT OUTER JOIN statements so you could find out which web browsers the customers who created the accounts who raised the invoices that generated the revenue were using.

Sometime around 2008, the SQL query in question surpassed Queen’s Bohemian Rhapsody in length and scope. By 2012 it was longer than Beowulf - and about as readable. It now stands as one of the great literary epics of our generation, a heartbreaking work of insane genius that is as incomprehensible as it is breathtaking.

4. The Chasm of Compliance

“It is a truth universally acknowledged that a profitable organisation in possession of a rigorous security policy must be ignoring a few things”
— Jane Austen.

Some friends of mine used to work in the software division of a company that made scientific instruments. Big government clients, universities, hospitals, research laboratories. As part of the conditions of doing business with these kinds of clients, they had a very strict policy that forbade sending email attachments — which was backed up with a set of firewall rules that would block incoming and outgoing mail with any files attached to it.

If you needed to share a file with a colleague who was working from home or away on a business trip, standard operating procedure was to attach it to a draft email in a Hotmail mailbox and then invite your colleague to download it and delete it when they were done. I believe they learned this particular trick from al-Qaeda. It satisfied the letter of the policy — after all, nobody ever SENT an email with any confidential material attached to it — and it got around the firewall blocks and restrictions.

You see that chasm? That gulf between ‘playing by the rules’ and ‘getting caught’? In there is a rich, thriving ecosystem of free-tier AWS accounts, Dropbox folders full of Excel spreadsheets and SSH proxies running on port 53 so the firewall thinks they’re DNS queries.

It would terrify you how much of your operating revenue comes out of that chasm.

5. The Shibboleth

Once upon a time, there was The Password. Being entrusted with The Password was a rite of passage. The Password was the nuclear launch codes, the keys to the city. Maybe it was the root password to the production web server. Maybe it was the sa password to the main database stack, or the master account password for the COBOL mainframe that handled all the financial records.

Of course, in these enlightened times, we have much, much better ways to restrict access to our systems. Federated authentication, PEM keys and one-time codes and 2FA. Single sign-on, cross linked to Sarbanes-Oxley auditing mechanisms so sensitive that if you so much as exhale whilst you’re logged in to production, it’ll analyse your breath content and record the fact that you had a beer with lunch, just in case they ever need to throw you under a bus in court.

Naturally, when you rolled out your new GDPR-compliant single sign-on, you changed the password. Of course you did. And then, when every single piece of software in your organisation went into a screaming panic, you immediately changed it back. And somewhere, there’s a backlog of the applications that need to be updated before you can change The Password. You’ve done the easy ones, obviously. But you haven’t got around yet to remoting into the VM where the Doctor Gonzo (qv) resides, and even if you could, you’ve no idea what algorithm the developers used to encrypt the connection string before pasting it into the INI file that’s eventually got transplanted into the Windows registry. And besides, you’re going to be replacing the Doctor Gonzo any day now, so you can change the password once that’s done.

The shibboleth is a powerful incentive to ensure that your tech staff leave on good terms. They know full well that if they give you any reason to doubt their integrity and trustworthiness following their departure, it’ll be a lot cheaper and easier just to have them killed than it will be to change the master password.

6. The Masquerade Column

This one’s a doozy. Somewhere in your organisation there’s a text column in a database. It was designed for your staff to make notes. It’s probably called something like ‘Comment’ or ‘Description’. 250+ characters of beautiful, unvalidated text. And, for many years, that’s exactly what it was used for… until that one fateful afternoon, when a couple of developers were sat around trying to plan a feature. And one of them said ‘can we add a field to the database to store the order type?’ And somebody else said ‘we COULD — but then we’d need to update the stored procedures and regression test any apps that are using column indexes and… it’ll turn a three-point ticket into a couple of weeks of work’.

At which point some bright spark says ‘hey, what if we use the Comment field? We can parse it, and if there’s a pipe character in there, anything after the pipe indicates the order type. And we’ll just tell the sales team not to edit anything in there that looks funny.’

Of course, it’s not always a pipe. Sometimes it’s a comma-separated list of product IDs, that your code knows to split, parse, translate back into orders and use those to populate the invoices. Or maybe you decided a comma was too obvious so you used a backtick instead — clever, huh? Sometimes, if you’re REALLY smart, you’ll use a newline character because you know full well that the UI element that’s bound to the field is a single-line textbox and so your users won’t see the hidden data you’ve stashed on line 2.

Note that if you’ve gone as far as storing actual JSON or XML in a free text field, that’s not really a masquerade any more, that’s just creative repurposing. The whole point of the masquerade is that your code has to run half-a-dozen different parsing and validation routines on that little piece of text before deciding whether it’s doing anything special or not.

Back in the glorious days of the first dotcom bubble, I once proposed using the description field on a table to store a SQL statement that needed to be generated when an order was raised, but not actually executed until the order was confirmed. I may have used the phrase ‘continuation passing’ to make it sound impressive. Fortunately the client pulled the plug on the entire project before it went anywhere.

7. The Folly

This is probably linked from your homepage. It definitely features prominently in your sales literature. Maybe you even ran a campaign about it. It’s a massively complex feature that was designed, built, shipped… and in the five years since it went live, it’s been used by exactly nine people. The folly is normally built as a sweetener. Like the senator who won’t sign off on a nuclear power bill unless they put a clause in it about reducing the excise duty on liquor sold in golf clubs, it was some weirdly specific scenario that one of your key stakeholders was absolutely hellbent on pushing through to production. Eventually, your team gives up trying to sell them on the idea of an MVP, and builds the whole damn thing just to shut them up.

To be a genuine folly, the system in question needs to be so inextricably wired into the rest of your software platform that shutting it down would be a major undertaking in its own right, and so it quietly purrs along in production, using up a couple of hundred bucks a month of cloud hosting and not really hurting anybody.

8. The Slow Loris

(https://www.worldatlas.com/articles/how-many-species-of-slow-lorises-live-in-the-world-today.html)

The slow loris is a nocturnal primate that’s indigenous to south-east Asia. Slow lorises have big round eyes and curious furry faces, and grow to about 30 cm long. They’re small, they’re cute, they’re not remotely threatening. And if you touch them, they can literally kill you.

Does that remind you of anything in your software? Sure it does. You know. That one innocent-looking database table that if somebody adds a row to it the entire website crashes. The button in your intranet dashboard that says something like ‘download CSV’ and if anybody clicks it the main database server instantly hits 100% CPU and stops accepting any more connections for a good twenty minutes or so. The DLL that has to be installed into C:\Users\Temp\ (because Reasons) and if it’s not there it’s a toss-up as to whether the phone starts ringing before the website goes down, or immediately afterwards.

That file that can’t possibly be doing anything? Probably best not to touch it. No matter how cute and fluffy it might look.

9. The Phantasm

Somewhere in your organisation, there’s a piece of critical physical infrastructure that is so well hidden it’s indistinguishable from magic. We call this the phantasm. And a true phantasm always starts out with somebody trying to make things look nice.

Back in the days of wired networks, a really good phantasm was hard to accomplish — you could always follow the wires. But in these days of wireless networks, it’s the easiest thing in the world to stick a wi-fi access point up above a ceiling tile somewhere and forget about it. Ten years later, your successors will thank you when they get asked to find out why the internet has stopped working. After furiously arguing for three or four hours that the internet NEVER worked because there’s clearly no wi-fi points anywhere on that floor, they end up ripping the entire ceiling down in a desperate attempt to work out what’s going on — and find a dust-bunny the size of a basketball with the last few inches of a wifi antenna forlornly poking out of it.

10. The Paradox

The paradox is a rare and beautiful artefact in modern software systems. It’s something that can’t possibly exist according to its own rules, and yet it does. The classic paradox is a table full of customers with no email address, in a system that (a) defines a customer as invalid unless the email address is populated, and (b) rejects any update to objects that are not valid. This can lead to HOURS of fun chasing your own tail up and down the aggregate graph trying to work out why you’ve managed to get seventeen validation errors without changing a single value.

There are other paradoxes as well. There’s the server that appears to be responding to pings despite the fact you’ve switched it off, removed the power supply and disconnected all the hard drives. The misconfigured IAM security policy that eventually turns out to serve absolutely no purpose other than preventing it from knowing about itself. The dashboard that’s reporting 100% availability because every single other system in the organisation has failed, including the systems that are supposed to report downtime to the dashboard aggregator.

A really good paradox will vanish without trace when you begin investigating it, like some sort of quantum phenomenon that can only exist until an observer attempts to measure it. This shouldn’t be confused with a Heisenbug, which is a bug that only becomes invisible whilst actively being observed. No, the paradox doesn’t just hide — it vanishes, leaving you sat in a retrospective meeting insisting that you really saw it whilst your team-mates look at you with raised eyebrows and wonder whether it’s time you took some holiday. The experienced paradox hunter doesn’t even run a database query until they’ve loaded Camtasia with silver bullets and started a screen recording.

Survival Tips for Aspiring Code Archaeologists

As you explore the canyons and catacombs of an unfamiliar codebase, keep your eyes peeled, for these are just a few of the weird and wonderful creatures you’ll find lurking in the dark places. You may seek to understand them. You may even seek to catalogue a few of your own — but be wary as you study their ways, for to familiarise yourself with these beasts is to walk a narrow and dangerous path. For as the great systems architect Nietzsche once said:

“Beware that, when fighting monsters, you yourself do not become a monster… for when you gaze long into the abyss, the abyss gazes also into you, and when you truly understand the legacy codebase, only then will you realise you yourself have become part of the legacy.”

Reflections on Alt.NET Birmingham 2018

2018-07-02T11:05:00+00:00

Around ten years ago, a bunch of developers got together in London under the banner of ‘alt dot net’ to talk about code, the universe and everything. Ten years doesn’t sound like much, but it’s actually quite amazing to look back on how much has changed. Back at that first alt.NET unconference, a handful of people in the room had a first-generation iPhone. I don’t think anybody had an Android device yet. Stack Overflow didn’t exist. Most of us weren’t on Twitter yet — In fact, my first-ever blog post was a writeup of that first alt.NET UK unconference and I didn’t join Twitter until three months later…

Most .NET developers in 2008 were still running Windows XP, apart from the unlucky few who’d upgraded to Vista and couldn’t work how to go back. Visual Studio 2008 was the new shiny. Lots of shops were still hosting on Windows 2003 or even Windows 2000 Server. That first wave of alt.NET was a bunch of developers who found C# and Visual Studio to be a powerful and productive platform for building software, but a platform that came bundled with a particular set of ‘best practises’ that weren’t necessarily the way we wanted to be working. The .NET community, such as it was, was heavily concentrated around Microsoft’s own conference and events, and there was very little discussion in that community around ideas like extreme programming, agile, model-view-controller unit testing, browser automation, dependency injection… you know. Ideas that are widely acknowledged here in 2018 to be, if not magic bullets, then well worth knowing and understanding how they might apply to the work you’re doing.

And now here we are, ten years later, on a glorious sunny day in Birmingham. Twenty-odd developers getting together on a Saturday under that same Alt.NET banner to talk about… well, whatever we want to talk about. That’s the beauty of the unconference format. The attendees create the schedule on the day. There’s no prepared presentations, no speakers, no keynotes or programme committee — you come along, you bring your questions and ideas and the things you want to talk about, and we see what happens.

An hour of spirited discussion and a LOT of post-it notes later, and we had an agenda — and what a wonderful range of topics:

Messaging and reference data, Blazor, XAML, WebAPI and EF Core and OData, liberating structures and agile processes, quantum computing and Q#, the SAFE stack, devops for mobile apps, Docker and Kubernetes and network security, gRPC and Service Mesh, dot net command line tooling, “are micro services dead?”, functional programming (and functional-style programming in C#), running dot net core on Linux, Systems Thinking, building JavaScript services in .NET Core, diversity and inclusivity, risk based software architecture, Xamarin and F#, and a crash course in Git and GitHub.

One of the great things about the unconference format is that you’ll end up with a bunch of curious people in a room wanting to learn about something… but ‘cos there’s no speakers and no ‘experts’, we’ll just fire up a laptop, find a quickstart or a ‘hello world’ tutorial or something, and start hacking on it. So we started off the day with a mob programming session on Blazor — one of the gang had a half-finished experimental chatbot they’d been working on, so we got that up on a big screen and played around with it for an hour until we had asynchronous callbacks and model binding working.

The next session I went to was one I’d proposed about quantum computing — I saw an intro session from Anita Ramanan and Frances Tibble at DDD13 in Reading last week, and I’ve been itching to download Q# and try it out… and so, in the absence of any quantum computing experts to tell us the answers, we had another group hack session with a laptop plugged into a big screen. What I found really remarkable about this session was how easily we got the whole thing up and running. I’m running the .NET SDK on macOS, using VS Code and JetBrains Rider, and it only took a few minutes to download the quantum computing preview, install it, fire up a ‘hello world’ app (or the Q# equivalent thereof) and start playing around with simulating quantum entanglement.

Next up was a freeform discussion all about command line tooling — gulp, grunt, yeoman, yarn, bower, npm, the dot net cli templating system — how did we get here, what did we like, and why the hell does running ‘dotnet new mvc’ drop a bunch of .something.bower files into your repository? We talked a lot about boilerplate code — about the days when File > New > Project would drop a few hundred lines of .designer.cs and .csproj files into your solution, which was never intended for human consumption, but you could guarantee that eventually SOMETHING would go wrong with one of those files and you’d have to roll your sleeves up and work out how to fix it. And we talked about how for people that have experienced this, Microsoft’s move towards a far more minimalist project file syntax is a Good Thing — but that alongside that, there’s a whole bunch of complexity that we’ve adopted wholesale from the npm ecosystem for managing web assets and front-end dependencies. We talked a bit about Andrey Taritsyn’s BundleTransformer, and the idea of treating SASS, SCSS, TypeScript et al as part of your solution and relying on the .NET pipeline to compile them at request time, and how the npm/node mindset has encouraged developers to regard these sorts of concerns as compile-time rather than runtime operations. The consensus — amongst our small and opinionated but by no means authoritative group — was that there’s still a pretty steep learning curve for the old-school .NET developer trying to understand modern web build tooling, but that it’s generally all getting a lot better.

I also made an honourable mention of the latest .NET CLI tooling by describing how an hour previously I’d installed the Quantum Computing SDK on macOS and got a couple of demos up and running, without having to reboot into Windows or fire up Visual Studio, and how that was actually quite awesome.

We broke for lunch — which was entirely thanks to Ian Russell, who I gather ended up not only organizing but also subsidising quite a lot of the event. For which we are all eternally grateful, but if there’s anyone out there whose company might be interested in supporting this kind of event, here’s a bit of advice for you. Community groups and events like this can make a few hundred pounds go a long, long way, but if your corporate sponsorship process means filling out forms, signing waivers and NDAs, submitting corporate investment opportunity proposals… the sort of people who run events like this are unlikely to appreciate your ‘support’. Work out how to support this kind of event with £500 out of petty cash, or offer to pick up the bill for lunch or something, and we’ll just say nice things about you and tell everyone on Twitter how grateful we are. You can get a lot of good will and positive buzz (not to mention sandwiches!) for less than it costs to hire a .NET contractor for one day — but when it comes to this kind of thing, please remember that your accounting procedures are your problem and not anybody else’s.

But I digress. Lunch was delicious and Ian, next time you’re down our way, the London .NET gang are taking you out for dinner to say thank you.

And on to the afternoon. First up was a session about functional programming — what is it, why does it matter, and how do you do it in C#? Partly inspired by some of Mike Hadlow’s blog posts about refactoring object-oriented C# into a more functional style, it was a great discussion; with some folks in the room who have very much embraced the functional paradigm, some folks who have spent years dabbling in F# but still prefer C# for everyday development, and some folks who haven’t really encountered functional programming but have heard a lot about it and would love to know why it sounds so important. Ian Cooper shared the ‘devil’s argument against F#’ — “I’m not sure I agree with all of this, but I know it well and I think it’s useful to share it” — about how pure functional programming is never going to cross the chasm from the bleeding edge to mainstream adoption; about how a lot of the ‘textbook’ benefits of functional programming are that it scales easily across multiple cores but that as an industry we’ve very much embraced the idea of scaling using services and containers rather than building monolithic applications that scale effectively on high-performance hardware. I pointed out that this notion of ‘scaling’ is very much rooted in ideas around scaling straightforward CRUD processes to cope with massive volumes of users — the predominant computational model of high-traffic web applications — and that there’s still lots of challenges around simulation, video rendering, gaming and machine learning where scaling a local process across multiple cores is still the most compelling route to improved performance. We also briefly discussed the dynamic that exists between the F# community and the rest of the .NET ecosystem (TL;DR: it’s mostly very friendly and lovely, but there’s still a perception that the functional community can be a bit elitist when it comes to sharing ideas with their object-oriented cousins).

There were also several recommendations to check out Scott Wlaschin’s book ‘Domain Modelling Made Functional’, both for an excellent overview of domain-driven design covered in the first few chapters, and for some worthwhile insight into the value of using functional programming patterns in your domain models.

Next was a session that ended up on the board as a combination of ‘diversity’ and ‘attracting new speakers’ — but ended up running a broad gamut from conferences and community, to speaker tips, suggestions and ideas for encouraging more people to get involved in what we do. I’m the first to admit that .NET has a massive problem with diversity — the vast majority of .NET developers in the UK are middle-aged white guys, and whilst there’s many brilliant speakers, engineers and evangelists on our platform who don’t conform to that particular stereotype, if you turn up to a .NET user group pretty much anywhere in the UK, it’s going to be overwhelmingly white males of a certain age. We talked about this at some length, including the observation that a lot of the people who are still passionate about .NET as a platform are people who have been here since the beginning, since the days of Visual Studio .NET and C# 1.0, and there’s not a whole lot of people who have made a conscious decision to embrace .NET as a development platform who didn’t already have some kind of investment in the Microsoft/Windows/SQL Server platform. The session convened around a question — how do we get more people looking at .NET as a development platform? It’s free. It’s open source. It’s cross-platform. It runs on macOS, Linux and Windows, it’s got excellent hosting and infrastructure support from multiple cloud providers… so why aren’t we seeing more interest in it? (When we figure out the answer, I’ll let you know. Promise!)

Finally, we wrapped up with a park bench discussion about “The .NET Renaissance — Where Are We?”. Following on from a series of blog posts and conference talks (and at least one DotNetRocks podcast) that various members of the .NET community have shared over the last eighteen months or so, and overlapping a lot with some of the questions and ideas from the session on diversity and inclusivity, we talked about .NET — past, present and future. Ian Cooper shared some slides, graphs and stats based on various metrics — recruitment websites, TIOBE — along with a healthy disclaimer that all programming language metrics were probably wrong and there’s “lies, damned lies, and percentages”. We talked about how Unity is creating a path into programming for a lot of first-time developers who are interesting in creating games and immersive 3D environments, and how the increasing interest in devices like GearVR and Hololens is only going to encourage this. Jim Bennett (who I should point out was here sharing his own opinions and not any official Microsoft position) spoke about how the diversity of platforms and runtimes almost guarantees long-term investment in C# as compared to platforms like Flutter/Dart, which are still tightly coupled to specific hardware and OS platforms. .NET has got some absolutely first-class tooling support. It runs on a huge number of devices — not just Windows, Linux & macOS, but thanks to Xamarin it’s now running on Android, iOS, Samsung’s Tizen platform (yeah, did you know your Samsung smart TV can run .NET Core applications?) — and with companies like AirBnB talking openly about the challenges they’ve had trying to use frameworks like React Native to deliver cross-platform solutions across web and mobile, it’s clear that there’s still a big unsolved problem around building cross-platform consumer apps, and that .NET Core and Xamarin Forms is working really hard to offer a compelling solution to that problem that could turn out to be the incentive that attracts a new generation of developers and startups onto the platform.

We took a few moments to plug a couple of other .NET events that are coming up in the UK over the next few months — the ProgNET conference and tutorials that we’re running at Skills Matter in September, and DDD East Anglia, which is another free community event that’s taking place in Cambridge in September.

And then, after eight hours of inspiring conversations and insightful observations, we went to the pub to reflect on how the day had gone. One thing that stood out was that, asking people how they’d heard about the event, there was very little correlation. Many of the people who came along today heard about it through word of mouth, and despite promoting the event widely on Twitter, email and social media, we probably knew a few dozen people between us who would have come along if they’d known it was happening. Asking people to give up a Saturday to come along to an unstructured conference is a bit of a big ask, sure — but actually, I think it’s one of the things that makes the unconference format work well. If you ran an event like that on a Friday, you’d probably get a lot more signups from people who persuaded their boss to give them a day off to go to a free conference… but you’d probably also get a lot more people turning up late, sitting quietly without really contributing anything, and I daresay sloping off the pub at lunchtime and not coming back. Most of the people who came along today weren’t based in Birmingham, and even though the event itself was free, somebody who gives up a day of their time and pays out of their own pocket to travel to an event like Alt.NET is exactly the kind of person you want involved.

One thing was overwhelmingly evident, though. If you’d taken today’s agenda back in time to 2008 and showed it to the people who came along to that first Alt.NET UK event, they wouldn’t have understood half of it (“coober-netties? What the hell is coober-netties?”) — and there’s no way they’d have believed the other half. C# running on an iPhone? HA! ASP.NET on Linux? HAAA! And when you sat down an hour later and worked out how install a quantum computing simulator on macOS and then hack together a quantum entanglement demo using a free open-source code editor with support for an experiment language for writing quantum algorithms, they would not have believed for a second that Microsoft and .NET was the driving force behind all those amazing innovations.

It was one of the most positive and inspiring events I’ve been to in a while — and when I think of how many excellent conferences and meetups I’ve been to in the last few months, that’s really saying something. Huge thanks to Ian Russell and Dave Evans for putting it together, to ImpactHub for the venue, but most of all to everyone who came along. An event like this is only as good as the people who show up, and today was absolutely awesome. See you all at the next one.

Oh, and the session on ‘are microservices dead?’ Yeah. Nobody turned up. I guess that makes it official. RIP microservices. It’s been… emotional :)

Tech and travel in Russia, Belarus and Ukraine

2018-01-09T14:02:00+00:00

Over the past couple of years, I’ve spoken at quite a few conferences in Russia, Ukraine and Belarus – DotNext in Saint-Petersburg and Moscow, BuildStuff in Kyiv and Odessa, and .NET Summit in Minsk. Later this year, I’ll be heading back to Saint-Petersburg for DotNext, and travelling to Novosibirsk in Siberia for CodeFest. All these events have open calls for papers, and if you’ve ever wanted to visit these places, tech conferences are a great way to do it. Lots of people have asked me what’s involved and whether it’s something they should look into - so here’s what I did, how it all worked for me, and some tips that might be useful.

Please remember I’m not an immigration lawyer. This is all based on my own experience, and I’m a straight white British cis male with no special dietary or medical requirements who is quite happy travelling alone in strange places. Laws, customs and cultures can vary wildly, so do your own research if you’re concerned about anything.

First up – yes, do it! I’ve had fantastic experiences on all these trips. The conferences I’ve spoken at have been first-class, professional events, with excellent facilities and really attentive, engaged audiences. Travelling in a country where you can’t even read the alphabet can be a bit of a culture shock at first, but I’ve had no problems navigating hotels, public transport and conference venues as an English speaker. Apps like Uber are great for this: being able to call a taxi and see exactly where you’re going and how much it’ll cost is hugely reassuring when you’re somewhere unfamiliar and can’t speak the language.

If you’re up for a challenge, have a go at learning to read the Cyrillic alphabet. The Russian language is hard, but you’ll find lots of the words on things like street signs and restaurant menus are familiar words, once you can read the language they’re written in. Being able to recognise things like Банка (banka – a bank), Гамбургер (hamburger) and Феттучини (fettucine) will get you a long way. (Russian, Ukrainian, and Belarusian are distinct languages with some broad similarities, but being able to recognise familiar English words in Cyrillic will stand you in good stead throughout the region.)

Visiting Ukraine and Belarus

If you’re a British passport holder, visiting Ukraine is easy; you can enter Ukraine without a visa for up to 90 days. Belarus has an interesting visa waiver: British citizens can enter without a visa as long as they stay less than five days and they enter and leave via Minsk international Airport. You need to show proof of medical insurance to clear passport control (and the authorities don’t consider a PDF on your phone to be documentary evidence). If you don’t have insurance, you can purchase it at the airport. It’s priced in euros, and you can pay in cash in Belarusian rubles, Russian rubles or “freely convertible currency” – I paid in euro coins without any problems.

Visiting Russia

Visiting Russia is more complicated. British citizens will need a visa to enter Russia, which means you’ll need to apply through VFS Global, the agency that handles Russian visa applications in the UK. (Note that Russian embassies in other countries often won’t process visa applications for British nationals living abroad – if you’re a Brit living elsewhere in the EU you’ll probably need to travel to London to apply for and collect your visa.) There’s a dizzying array of visa types – tourist, private, student, work, business – and you’ll find various opinions online about what kind of visa you need if you’re speaking at a technical conference. Given this ambiguity, I’ve always insisted on a business visa for my trips to Russia; I’ve never been asked anything on arrival other than when I’ll be leaving, but your experience may differ.

Here’s how it works. The whole process takes about two months.

TL;DR:

Ask the conference organisers to prepare your visa invitation
Fill out the form on the VFS Global website
Wait for confirmation that your invitation has been issued
Go to the VFS Global office in London with your passport, form and documents
Submit your application
Wait 5 days (or 24 hours if you've paid for overnight processing)
Go and pick up your passport and visa
Go to Russia!

Your hosts (normally the company organising the conference) will issue you with a visa invitation. This can take a while, particularly if there are public holidays in Russia; they won’t do it more than a month in advance so you need to time everything quite carefully – especially if you have other travel commitments for which you’ll need your passport. The invitation will be sent by telex from the Russian Ministry of Foreign Affairs, to the Russian consulate in London. Your first application must be for a single-entry visa; if you’ve already done one trip and it looks like you’ll be doing a lot more you can ask them to prepare an invitation for a multi-entry visa valid for one year.

Meanwhile, take a look at the online application form on the VFS Global website. You can’t submit the form electronically – once you’ve finished filling it in, it’ll produce a PDF that you need to print. It asks for things like your parents’ dates and places of birth, details of every country you’ve visited in the last 10 years, information about your marital status (and, apparently, current contact information for your former spouse if you’re divorced or separated). It might take a while for you to dig out all the details you’ll need, so make sure you look at it well in advance. You won’t be able to complete your application until you’ve got your invitation, but you can make a start and save your progress.

Once your invitation has been issued, your hosts (or their travel agent) will send you an email with a ‘telex reference’ on it, which you you’ll need in order to complete the visa application form. Take the printed form, along with your passport and supporting documents, to the VFS Global application office in Gee Street in London, just north of Barbican tube station. There is a PDF on their website that lists the supporting documents you’ll need – passport, application form, passport photo, invitation (telex), introductory letter, and certified bank statements if you’re self-employed. I’ve never been asked to show the introductory letter or the bank statements, but take them with you anyway, just in case.

The VFS Global office is open 08:30-15:00 Monday to Friday, but if you turn up to submit a business visa application between 12:00 and 14:00 they’ll ask you to come back later – they’ll need to phone the Russian consulate to confirm receipt of your telex before they’ll accept your application, and they can’t do this between 12:00 and 14:00 because the consulate closes for lunch.

They’ll take your passport and all your paperwork. You’ll get fingerprinted, and pay the visa application fee and service charge. Cash or credit/debit card is fine, but they don’t take Amex. It’s about £120 total for a single-entry visa ready in 5 working days; if you’re in a hurry, they have an express overnight service which costs more. They’ll give you an application receipt. Don’t lose it – you can’t get your passport back without it.

When your visa’s ready, you’ll have to go back to pick it up – the office is open for collections between 16:00 and 17:30. Check that your visa details are all correct – it’ll be a full-page colour sheet stuck in your passport. One thing I’ve noticed is that names may not be transliterated into Russian consistently – I’ve got two visas in my passport right now; one says I’m Дилан Битти and the other one says I’m Дилан Бети (that’s roughly ‘bitti’ and ‘beti’ written in Cyrillic), but this doesn’t appear to be a problem.

That’s pretty much it. When you arrive in Russia you’ll be given an immigration card at passport control, which you’ll need to show when you check into your hotel to prove you’re in the country legally, and which you’ll need to go through passport control when you leave. Take a picture of this card on your phone (or ask the hotel to photocopy it for you), so you can keep the original with you and leave a copy in the safe in your hotel. The UK foreign office says you must carry your original passport with you at all times in Russia, and can be fined if you don’t produce it when asked. I’ve never been asked to produce mine, but I keep my passport, visa and immigration card on me, with printed photocopies of them all in the hotel room safe and digital copies in Dropbox, just in case.

Practicalities

Most hotels will change major foreign currencies into local currency. There are also foreign exchange bureaux everywhere, but I’ve always changed money in my hotel or used an ATM. If you’re planning to change money, carry cash in a combination of euros and US dollars, since these are most widely accepted by foreign exchange bureaux. It’s easy enough to get Russian rubles from a UK foreign exchange bureau before you leave for your trip, but you probably won’t be able to get Ukrainian hryvnia or Belarusian rubles. There are also ATMs everywhere and most restaurants will take Visa and Mastercard. If you’re using something like a Monzo card or other prepaid payment card, remember you may not have roaming data on your phone so top it up before you leave the hotel.

Roaming data outside the EU is seriously expensive. It’s a lot cheaper than it was a few years ago – during my first visit to Ukraine in November 2015, roaming data was £8/MB – but it’s still not cheap; in Russia, EE currently charges £7.00 for 100Mb, valid for 24 hours. Public wifi is widely available; in Kyiv it’s pretty ubiquitous and often completely unsecured (although you’ll need to know how to accept the terms and conditions in Ukrainian!) but in Russia and Belarus you’ll often find you need a local mobile phone number to register on public wifi spots. Make sure your phone’s unlocked before you go, pick up a local pay-as-you-go SIM card, and ask one of the conference organisers or a friendly local to help you activate and register it (remember that all the confirmation messages from the phone network will be in Russian or Ukrainian).

It’s also worth caching Google Maps offline using the “ok maps” feature, and downloading the Google Translate app and the language files for Russian to your phone so that even if you’re not online you’ll be able to navigate and translate some basic phrases.

And finally – don’t forget to actually see the place! It’s all too easy to travel halfway around the world for a conference and never see anything except the inside of the Radisson Blu, so if the conference organisers have arranged a tour of the city or any excursions, go along – you’ll get to see some really interesting things. Or head out on your own. Moscow, Saint-Petersburg, and Kyiv have fast, efficient metro networks, with stations and maps in English as well as Russian/Ukrainian. The metro is great for travelling around without having to talk to anybody – buy tickets at a machine, use the automated barriers, watch the locals when it comes to navigating around the stations and do what they do. There are metal detectors everywhere, which go off constantly. It looks like they’re being ignored but apparently they give the police probable cause to search anybody they don’t like the look of - personally I’ve never had any problems at all.

The great thing about being at a tech conference is that many of the attendees will be locals who know the city and will be perfectly happy to take you their favourite bar or a good place for dinner or some live music or whatever you fancy doing with your spare time. Enjoy the conversations, soak up the culture, try the local food, and remember not to stay up drinking vodka until 2am when your flight is at 06:30... unless you really, really want to.

Sounds like fun?

If all that’s piqued your interest... awesome! Here are a few events that are accepting papers at the moment.

DotNext, Saint Petersburg, Russia, 22-23 April (CFP)
.NET Summit, Minsk, Belarus, 2-3 March (CFP)
CodeFest, Novosibirsk, Russia, 31 March – 1st April (CFP)

Увидимся!

Thank you for flying Analysis Airways

2017-12-05T18:44:00+00:00

If you work in software – or even if you don’t – it’s likely that, at some point, you’ll find yourself working with a team who are completely unfamiliar with your systems. A management consultancy, a development partner, a new supplier. A team of smart, capable people who are hoping to work with you to deliver something, whether that’s process improvements or a reduction in costs or some shiny new product.

A common pattern here is that, at the start of the engagement, they’ll appoint some business analysts to spend a whole lot of time talking with people from your organisation to get a better idea of what it is you do. They sometimes call this ‘gathering requirements’. You’ll know it’s happening when you get half-a-dozen invitations to four-hour ‘workshops’ from somebody you’ve never met, normally with a note saying something like ‘Hey everyone! Let’s get these in the diary!’

Now, there’s a problem here. Asking people what’s happening is almost never the best way to find out what’s actually happening. You don’t get the truth, you get a version of the truth that’s been twisted through a series of individual perspectives, and when you’re using these interviews to develop your understanding of an unfamiliar system, this can lead to an incredibly distorted view of the organisation. Components and assemblies aren’t ranked according to cost, or risk, or complexity. They’re ranked according to how many hours a day somebody spends dealing with them. And when you consider that in these days of scripted infrastructure and continuous deployment, a decent engineer can provision an entire virtual hosting environment in the time it takes to deal with one customer service phone call, what you end up with is a view of your organisation that ranks ‘phone calls’ equal to ‘hosting environment’ in terms of their strategic value and significance.

When you factor in the Dunning-Kruger effect, the errors and omissions, the inevitable confusion about naming things, and the understandable desire to manage complexity by introducing abstractions, you can end up with a very pretty and incredibly misleading diagram that claims to be a ‘high-level view’ of an organization’s systems.

There’s a wonderful example of this in neurology – a thing called the ‘cortical homunculus’; a distorted representation of the human body where the various parts of the body are magnified based on the density of nerve endings found therein. Looks like this:

https://www.maxplanckflorida.org/fitzpatricklab/homunculus/

It’s recognisably human, sure. But it’s a grotesque distortion of what a human being actually looks like – brilliant for demonstrating neurology, but if you used it as a model when designing clothes or furniture your customers would be in for one hell of a shock. And we know it’s grotesque, because we know what human beings are supposed to look like – in fact, it’s the difference between the ordinary and the grotesque that makes these cortical homunculi interesting.

The problem with software is that it’s made out of invisible electric magic, and the only way to see it at all is to rely on some incredibly coarse abstractions and some very rudimentary visualisation tools.

https://ironmaiden.com/ed-force-one

Imagine, for one second, that we’ve hired some consultants to help us design an aircraft. They send over some business analysts, and book some time with the ‘domain experts’ to talk over the capabilities of the existing system and gather requirements. The experts, of course, being the pilots and cabin crew – which, for a Boeing 747 like Ed Force One, is three flight crew and somewhere around dozen cabin attendants. They spend a couple of very long days interviewing all these experts; maybe they even have the opportunity to watch them at work in the course of a typical flight.

And then they come up with this: the high-level architectural diagram of a long-range passenger airliner:

Now, to an average passenger, that probably looks like a pretty reasonable representation of the major systems of a Boeing 747. Right? Take a look. Can you, off the top of your head, highlight the things that are factually incorrect?

That’s why this diagram is dangerous. It’s nicely laid out and easy to understand. It looks good. It inspires trust… and it’s a grotesque misrepresentation of what’s actually happening. Like the cortical homunculus, it’s not actually wrong, but it’s horribly distorted. In this case, the systems associated with the cabin attendants are massively overrepresented - because there’s 12 of them, as opposed to three flight crew – so 400% more workshop time and 400% more anecdotal insight. The top-level domains – flight deck, first class, economy class – are based on a valid but profoundly misleading perspective on the systems architecture of an airliner. The avionics and flight control systems are reduced to a footnote based on three days of interviews with the pilots, somebody with a bit of technical knowledge has connected the engines to the pedals (like a car, right?) and the rudder to the steering wheel (yes, a 747 does have a steering wheel), the wings are connected to the engines as a sort of afterthought…

Now, when the project is something tangible – like an office building or a bridge or an airliner, it won’t take long at all before somebody goes ‘um… I hate to say it, but this is wrong. This is so utterly totally wrong I can’t even begin to explain how wrong it is.’ Even the most inexperienced project manager will probably smell a rat when they notice that 20% of the budget for a new transatlantic airliner has been allocated to drinks trolleys and laminated safety cards.

But when the project is a software application – you know, a couple of million moving parts made out of invisible electronic thought-stuff that bounce around the place at the speed of light, merrily flipping bits and painting pixels and only sitting still when you catch one of them in a debugger and poke it line-by-line to see what it does – that moment of clarity might never happen. We can’t see software. We don’t know what it’s supposed to look like. We don’t have any instinct for distinguishing the ordinary from the grotesque. We rely on lines and rectangles, and we sort of assume that the person drawing the diagram knew what they were doing and that somebody else is looking after all the intricate detail that didn’t make it into the diagram.

And remember, nobody here has screwed up. The worst thing about these kinds of diagrams is that they’re produced by competent, honest, capable people. The organization allocates time for everybody to be involved. The stakeholders answer all the questions as honestly as they can. The consultants capture all of that detail and synthesise it into diagrams and documents, and everybody comes away with the satisfying sense of a job well done.

That’s not to say there’s no value in this process. But these kinds of diagrams are just one perspective on a system, and that’s dangerous unless you have a bunch of other perspectives to provide a basis for comparison. A conceptual model of a Boeing 747 based on running cost – suddenly the engines are a hell of a lot more important than the drinks trolley. A conceptual model based on electrical systems. Another based on manufacturing cost. Another based on air traffic control systems and airport infrastructure considerations. And yes, producing all these models takes a lot more than arranging a week of interviews with people who are already on the payroll, which is why so many projects get as far as that high-level system diagram and then start delivering things.

And why, somewhere in your system you almost certainly have the software equivalent of a hundred-million-dollar drinks trolley.

Thank you for flying Analysis Airways.

Goodbye Spotlight… Hello Skills Matter!

2017-11-24T17:44:00+00:00

I have some exciting news. I'll be leaving Spotlight at the end of January, to take up a new role as CTO at Skills Matter. After fourteen years at Spotlight, this is a massive change for me, it's a massive change for Spotlight, and (I hope!) it's going to be a massive change for Skills Matter as well - but it's also a very natural next step for me, and a move that I think is going to unlock all sorts of exciting possibilities over the coming years.

I first started working with Spotlight way back in 2000 - in Internet time, that's about a hundred million years ago. My first job after I graduated was working for a company who built data-driven websites in ASP; spotlight.com was the first big commercial site I ever built, and I've been working with them ever since - initially as a supplier, then as webmaster (remember those?), then head of IT, then systems architect. I've come with them on a journey from Netscape Navigator and dial-up modems to smartphones and REST APIs and progressive web apps, and it's been a blast - we've shipped some really excellent projects, we've learned a lot together, and I've had the pleasure of working with awesome people and a lot of very cool technology.

Around ten years ago, up to my eyeballs in ASP.NET WebForms and wondering if everybody found them as unpleasant as I did, I started going along to some of the tech industry events that were happening here in London to get a bit of external perspective. I was at the first Future Of Web Apps conference back in 2007, the first Alt .NET UK 'unconference', the DDD events held at Microsoft a few times a year... and it wasn't long after that that I met Wendy Devolder and Nick Macris, who at the time were still running Skills Matter out of a basement in Sekforde Street, and hiring the crypt underneath St James Church on Clerkenwell Green for bigger events. That's where I did my first-ever technical talk - a fifteen-minute introduction to jQuery which I presented at the Open Source .NET Exchange. And, as the saying goes, I've never really looked back. This year, I've spoken at 20+ tech events in nine countries, from huge international conferences to local user groups around the UK. I'm on the programme committee for NDC London, FullStack and Progressive.NET, I'm helping to run the London .NET User Group, I've started giving training courses and workshops on building hypermedia APIs and scalable systems. And, because I'm one of those kinds of nerds, every time I take part in a tech event I come away from it buzzing with ideas about how to make it better - for attendees, organisers, volunteers, speakers... everybody. The only problem was finding the time to implement those ideas, and so when Wendy got in touch a few months ago to ask if I'd be interested in joining the team at Skills Matter and putting some of those ideas into practice, the timing was just right.

Now, this isn't intended to be a puff-piece. I've known the team here at Skills Matter for many, many years, we’ve done some really excellent things together, and I'm really excited to be joining them. I’ve also spoken to literally hundreds of people whose experience at their conferences and events has been nothing but positive, and that’s played a big part in my decision to come on board here. However, I also know that a few of you have had some frustrating experiences with them in the past - about how they organise their community events, about logistics and speaker invitations for their bigger conferences... even things like having the ‘wrong kind of cider’ in the Space Bar. :) Well, I want to hear all the details. I know we can't please all of the people all of the time, but the fact the team here has hired me means they're up for a bit of spirited discussion and an influx of new ideas, so if you wanna drop me an email or bend my ear over a beer sometime, I'd love to hear from you and see what we can do about it.

For the team at Spotlight, this marks the dawn of a new era. I’ve come dangerously close to becoming a dungeon master, and to me that’s a clear sign that it’s time to hand over the lines and rectangles to somebody new. To quote from Alberto Brandolini’s “The Rise and Fall of the Dungeon Master”:

“I am not suggesting to hire a hitman, but to acknowledge that the project would be better off without the Dungeon Master around. If you’re looking for a paradigm shift, you’ll need a team with a different attitude and emotional bonding with the legacy.”

I'm sad not to be coming with them on the next phase of their journey, but it's clear that Spotlight's future lies along a different path from mine. They've got an absolutely first-class team there, I'm sure they're going to go on to even bigger and better things, and I'm looking forward to catching up with them all over a beer every once in a while and seeing how it's all going.

First and foremost, though, this is a chance for me to get more involved with the global tech community. I absolutely love the part of my life that involves bouncing all over the world talking about tech, sharing ideas and meeting interesting people; and I'll be taking advantage of all those opportunities to chat about what Skills Matter can do to help support the tech community - whether it's open source projects or corporate development teams, tiny meetups or big international conferences. I'll also be working with the software team here to create some really exciting things, and of course I'll be getting even more closely involved in the conferences and meetups that we host here at CodeNode and at various venues around London. For Skills Matter, my experience as a developer, user group organiser and programme committee volunteer - not to mention my connections in the tech industry all over the world - will play a big part in deciding our plans and priorities for the next few years.

And the best part? I even get to work the odd shift behind the bar here at CodeNode once in a while, thus becoming the tech industry’s answer to Neville the Part-Time Barman. So next time you’re round Moorgate of an evening, drop in and say hi.

Exciting times, indeed.

London .NET User Group Open Mic Night

2017-11-09T18:10:00+00:00

Last night’s London .NET meetup was an ‘open mic’ session. Rather than inviting speakers to talk for an hour or so as we normally do, we invited members of the group to come along and talk for 10-15 minutes about their own projects, things they’re working on, or just stuff they think is cool. It’s the first open mic session we’ve had in a long while, but based on the success of last night’s event I think we’ll try to make these more of a regular thing in 2018. Quite a few of the speakers who came along were talking about their own .NET open source projects, and showing off some very, very cool things; here’s a quick rundown.

First up was Phil Pursglove, giving us a whistle stop tour of Cosmos DB, a new database platform that Microsoft are now offering on Azure. I’ve seen a couple of talks this year about Cosmos, and it looks really rather nice. It’s got protocol-level compatibility with MongoDB (what Microsoft call ‘bug compatible’), plus support for SQL and a couple of other language bindings. One of the coolest features of Cosmos is native support for multiple consistency models, allowing you to optimise your own application for your particular requirements - with the ability to override the global consistency model on a per-request basis. There’s a time-limited free trial available here - check it out.

Next up, Robin Minto gave us a run-through of OWASP ZAP, a proxy-based web security tool created by the Open Web Application Security Project (OWASP). ZAP is beautifully simple; you install it (or fire up the Docker image), it acts as a web proxy whilst you navigate through some of the primary user journeys on your web application, but in the background it’s probing your server and scanning your HTML for a whole range of common security vulnerabilities - and when you’re done, it’ll generate a security report you can share with the rest of your team.

James Singleton - author of ASP.NET Core 2 High Performance - gave us a very cool live demo of some of the cross-platform capabilities of .NET Core 2.0, including using his Windows laptop to cross-compile a web app for ARM Linux and running it live on a Raspberry Pi. What I found really impressive about this is that it didn’t require any .NET framework or runtime install on the Pi - it’s just vanilla (well, raspberry!) Raspbian Linux with a couple of things like libssl, and everything else is included in the deployment package created by the dot net tooling.

Ed Thomson, the Git program manager for Visual Studio Team System, did a great walkthrough of his open source project libgit2sharp - a set of C# language bindings for working with local and remote git repositories. If you’ve ever had to parse the output from the Windows git command line tools, you’ll know how painful it can be - but with libgit2sharp, you can use C# or Powershell to create automated build tools, manipulate your git repositories and do all sorts of cool stuff.

Jason Dryhurst-Smith gave us a demo of his CorrelatorSharp project - “your one-stop shop for context-aware logging and diagnostics” - a library that allows you to track the context of operations across multiple services and operations, including support for frameworks like NLog and RestSharp and a client-side JavaScript library.

Ben Abelshausen came along to show us Itinero, an open-source route planning library for .NET which you can use within your own applications to calculate routes and analyse map data.

Finally, we had Matt Ellis from Jetbrains giving us ’Ten Jetbrains Rider Debugging Tips in Ten Minutes’ - including some really neat stuff like cross-platform support for DebuggerDisplay attributes and the ability to define interdependent breakpoints in your code.

Huge thanks to all our speakers and to everyone who came along - it’s great to see so much enthusiasm and activity going in with .NET open source, and with .NET Core 2.0 going fully cross-platform and tools like Rider and Visual Studio Code, it’s only going to get more interesting.

See you all at the next one!

Why You Weren’t Picked For NDC London

2017-10-17T09:23:00+00:00

Over the past two weeks, a small team of us has been putting together the agenda for NDC London 2018, which is happening at the Queen Elizabeth II Centre in Westminster this coming January. As somebody who’s submitted a lot of conference talks over the years, I know how exciting it is getting the email confirming that you’ve been accepted — and how demoralizing it can be getting the email saying ‘We’re sorry to say that…’

Well, now that I’ve been on the other side of that process a few times, I see things very differently, so I wanted to take this chance to tell you all how the selection programme actually works, why you didn’t get picked, and why it shouldn’t put you off.

First, though, I want to say a huge thank you to everybody who submitted, and congratulations to the people who have been selected. It’s been a lot of work to pull everything together, but I’m really happy about the programme of amazing people and top-class talks that we’ve got this year — and I’m particularly excited that we’ve been able to include so many great speakers from the developer community who will be speaking at NDC for the first time. Welcome, all of you.

For all the speakers who didn’t get selected this time around, I really hope this article might help you understand why. (TL;DR: we just had too many good talks submitted!)

First, a basic rule of conferences — they have to generate a certain amount of money in order to operate. Yes, there’s community events like DDD that are free to attend and are run with minimal sponsorship — and I think those events are every bit as valuable to our community as the big commercial conferences — but once you start dealing with international speakers and multi-track events running across several days, you need to start thinking about commercial considerations. Conferences generate revenue from two sources — ticket sales and sponsorship. Both of those boil down to creating an event that people want to attend (and, in many cases, can persuade their boss is worth the cost and the time), so that’s what we, as the programme committee, are trying to do.

I should mention that on every conference I’ve worked with, the programme committee has consisted entirely of volunteers — four or five people from within the industry who are happy to donate their free time to help put the programme together. They’ll normally get a complimentary ticket to the conference in return, but it’s not a job, and nobody’s getting paid to do it.

Most software conferences run a public ‘call for submissions’ (aka ‘call for papers’ or CfP) where anybody interested in speaking can submit talks for consideration. Once the CfP has closed, the programme committee has the unenviable task of going through all the talks that have been submitted and picking the ones that they think should be included. NDC London has five tracks of one-hour talks, over three days. Once you’ve allowed for keynotes and lunch breaks, that gives you fewer than 100 talk slots. We had 732 sessions submitted. If we’d said ‘yes’ to all of them, we’d have ended up with a conference lasting three weeks with a ticket price of over £7,500… good luck getting your manager to sign off on attending that one.

So, how do you pick the top 100 talks from 732 submissions? Well, here’s what we look for.

First — quality. This one doesn’t really help much, because the vast majority of the talks submitted to a high-profile event like NDC are excellent, but there might be a handful that you can reject outright. These tend to be the talks that look like sales pitches — single-vendor solutions submitted by somebody who works for the vendor in question. You want to use a conference to sell your product? No problem — get a booth, or buy a sponsorship package. But we’re not going to include your sales pitch at the expense of someone else’s submission.

It’s also worth pointing out that when we get to the final round of submissions, when it’s getting really hard to make a call, we’ll look very critically at the quality of the submission itself. We’ll look for a good title, a clear, concise summary, a succinct speaker biography and a good-quality headshot — basically, a submission that makes it really clear who you are, what you’re talking about, why it’ll be interesting, and that you’re going to put in the effort required to deliver a great presentation. There’s no hard-and-fast rule to this; there are some excellent articles out there about how to write good proposals. My own rule of thumb when I’m submitting is 100 characters for the title, 2,000 for the abstract and 1,000 for the speaker bio, and I often use Ted Neward’s approach of ‘pain and promise’ — ‘here’s a problem you’ve had, here’s an idea that might fix it’ — when I’m writing proposals. Every speaker is different, and we all have our own style, but if your talk summary is only one or two lines or you’ve pasted in your entire professional CV instead of writing a short speaker bio, you may well get rejected in favour of somebody who’s clearly put more time into drafting their submission.

Second — relevance. Conferences have a target audience; NDC has evolved from being primarily a .NET conference into an event with a much broader scope, but we know the kind of developers who normally come to NDC London and what sort of things they’re interested in. For example, this year we decided to decline any C++ talks, purely because there’s very few C++ developers in our target audience. That said, we do try to include things that might be interesting to our audience, even if they’re not immediately relevant. Topics like Kotlin and Elm are still relatively esoteric in terms of numbers of users, but the industry buzz around them means we try to include things like this on the programme because we’re confident that people will want to see them.

Third — diversity. We could very easily have filled the entire programme with well-known white male speakers talking about JavaScript — but we think a good conference should feature diverse speakers, presenting diverse topics, with a range of presentation styles. We’re not just talking about gender and ethnicity, either — we want to see new faces alongside regular speakers, and bleeding-edge technology topics alongside established patterns and practices. That said, we would never accept a substandard talk purely for the sake of diversity; the way to get more balance in the programme without compromising quality is to start with a wider range of submissions. Many conferences suffer from a lack of diversity in the talks that get submitted — it’s just the same people submitting the same topics year after year — so throughout the year people like me go to tech events and conferences, look out for great speakers and talks that haven’t appeared at NDC before, and invite them to submit.

Finally, there’s the invited speakers — the people who we definitely want to see on the programme. They fall into two broad camps. There’s the big names — the people with 100K+ Twitter followers, the published authors, the high-profile open source project leaders. Now, being famous-on-the-internet doesn’t automatically mean you’re a brilliant speaker — speaking for an hour in front of a few hundred people takes a very different set of skills from running an open-source project or writing a book — but we’re lucky in our industry to have a lot of people who are well known, well respected, and really, really good on a conference stage. And these people are important, because their involvement gives us a fantastic signal boost. More interest translates into more ticket sales — which means more budget for covering speaker travel, catering, facilities, the entertainment at the afterparty, and all the other things that make a good conference such a positive experience for the attendees.

The programme committee also invites a lot of new speakers because it’s a great way of getting some new faces and new ideas on to the programme. As I mentioned above, many of us spend a lot of time going to user groups, meetups and conferences, and when I see somebody deliver a really good talk, I’ll invariably get in touch afterwards and ask if they’d be interested in submitting it to an event like NDC.

So, those are our constraints. We want to deliver a balanced mix of big names and new faces. We want to promote diversity in an industry that’s still overwhelmingly white and male (never mind the ongoing fascination with JS frameworks and microservices). We want to offer a compelling combination of established technology and interesting esoterica; of stuff that’s interesting, stuff that’s relevant, and stuff that’s fun…

And we get to pick fewer than 100 talks out of 750 submissions, which means whatever we do, we’re going to be sending a whole lot of emails at the end of it saying ‘Sorry, your talk wasn’t accepted…’ — and that’s not much fun, because we’re turning down good content from good speakers. In many cases those people are good friends as well — the tech industry is a very friendly place and I count the people I’ve met through it among my closest friends. But that’s how it works.

It’s humbling to be part of an industry where so many talented people are willing to invest their time in sharing their own expertise. Speaking at conferences is a really rewarding experience, but it’s also a huge amount of work, preparation, rehearsal, logistics, and time away from home. Without speakers, there’d be no conferences — but, as I hope I’ve explained here, there are more excellent people and talks out there than any one event can ever hope to accommodate.

The thing is, there’s absolutely no shortage of great conferences and events. Don’t be discouraged. Keep submitting. If you didn’t make the cut for NDC London, submit to Oslo and Sydney. And BuildStuff, and DevSum, and Øredev, and FullStack, and Progressive.NET, and DotNext, and Sela Developer Practice, and SDD, and QCon, and WebSummit. And if none of those grab you, sign up for services like The Weekly CFP and Technically Speaking that will email you about conferences that are looking for speakers and submissions.

Finally, you know the one thing that every really good speaker I’ve ever seen has in common? It’s that they work hard on interesting things, and they love what they do. Maybe it’s their job. Maybe they lead an open-source project, or run a user group, or they’re writing a book. But if you love what you do and you want to share that enthusiasm, it’ll happen. Just give it time.

London, London, Uber Alles

2017-09-23T10:18:00+00:00

I read with some interest yesterday that Transport for London (TfL) are not renewing Uber’s license to operate in London. TfL have cited concerns over Uber’s driver screening and background checks, and Uber’s use of ‘Greyball’, a software component designed and built by Uber to bypass all sorts of regulatory mechanisms, including using a phone’s GPS to recognise when the phone is at being used at Apple HQ so that the Apple engineers who review iOS applications won’t see the hidden features that Apple aren’t supposed to know about.

I use Uber a lot. Their service used to be absolutely excellent, and is still pretty good. It’s not as good as it used to be. It takes longer to get a car than it used to, particularly in central London. But, as a passenger (and yes, I know I’m a white male passenger, although some of my Uber experience dates from a period when I did have quite serious mobility issues whilst recovering from a skiing injury), I have found Uber to be a really good service. I’ve used it all over the world — London, Bristol, Brussels, Kyiv, Saint-Petersburg, and as of last night, Minsk. I missed it in Tel Aviv, where it’s been outlawed and everyone uses Gett instead, although Gett in Israel appears to operate exactly the same as Uber does in London so I’m not quite sure what the distinction is.

I’ve had some seriously impressive experiences as an Uber customer. In London, I once left my guitar in the back of an Uber that dropped me home at 4am after a horribly delayed flight. I realised within minutes. I used the app to phone the driver, who immediately turned around and brought it back, and was very taken aback when I insisted on paying him for the extra journey. In Kyiv I’ve used Uber to travel safely from a place I couldn’t pronounce to a place I couldn’t find, in a city where I couldn’t speak the language or even read the alphabet, with a driver who spoke no English, and I’ve done it with absolutely no fear of getting ripped off or robbed.

It’s not all been smooth. In Bristol I once had an Uber driver — sorry, “partner-driver” — who missed the turning four times in a row and then explained, giggling, that he’d never driven a cab before and didn’t know how to use satellite navigation. In London I’ve actually been in an Uber car that was pulled over by the police for speeding.. the driver panicked, drove away, realised what he’d done, thought better of it and reversed down Moorgate, in rush hour traffic, back to where a rather surprised-looking police officer immediately placed him under arrest. In both of those cases I complained, Uber investigated (something they can do really easily, thanks to GPS tracking of exactly what all their vehicles are doing at any moment) and notified me within 24 hours that the drivers in question had been suspended and would not be driving for the company again. I’ve used their app to claim refunds where I was overcharged — but also to reimburse drivers who undercharged me because of technical problems.

Now, here’s the two points I think are really important. One — Uber didn’t actually solve any hard problems. They didn’t invent GPS, or cellular phone networks, or draw their own maps of the world’s major cities. They just waited until exactly the right moment — the moment everyone had GPS, and online payments were easy, and smartphones were cheap enough that it was cost-effective to use them as the basis of a ride-sharing application — and then they pounced. Now don’t get me wrong, they did it incredibly well, and the user experience — particularly in the early days — was absolutely first-class. Usability features like being able to photograph your credit card using your phone camera instead of having to type the number in were a game-changer — the kind of features that people would show off to their friends in the pub just because no-one had ever seen it before. But Uber didn’t solve any hard engineering problems. There’s no PageRank algorithms or Falcon 9 reusable rockets being invented here. Uber’s success is down to absolutely first-class customer experience, aggressive expansion, and their willingness to enter target markets without the slightest consideration for how their service might affect the status quo. And their protestations about the ruling threatening 40,000 jobs is a bit rich given how adamantly they insist that they don’t actually employ any drivers, despite a court ruling to the contrary.

If it hadn’t been them, it would have been someone else, which brings me to my second point. The London taxi cab industry was just crying out for somebody to come in and kick seven bells out of it. I’ve lived in London since 2003, and I’ve taken many, many taxis over the years. For all the talk about “The Knowledge”, I have lost count of the number of times I’ve had to give a black cab driver turn-by-turn directions because they don’t know where I’m going and they’re not allowed to rely on sat-nav. I’ve lost count of the number of times I’ve had to ask a cab driver to stop at a cash machine because they don’t take credit cards. I’ve lost track of the number of hours of my life I’ve spent stood on street corners, in the rain, with more luggage than I can possibly carry home on the bus, waiting for the glow of an orange light that might take me home. Or might just ask where I’m going and drive away with a shake of the head and a ‘no, mate’ because my journey isn’t convenient for them.

The alternative, of course, was minicabs. Booked in advance from a reputable operator, they were generally pretty good. Not always, but most of the time they’d show up on time and take you where you wanted to go. Or, at the end of a long night out, you’d wander up to one of Soho’s many illustrious minicab offices, and end up in the back seat of an interesting-smelling Toyota Corolla wondering if the driver was really the person who’d passed all the necessary tests and checks, or one of their cousins who’d borrowed the cab for the night to earn an extra few quid. (On two separate occasions I’ve been offered this as an explanation for a minicab driver not knowing where they’re going…)

A lot of people are very upset about yesterday’s news — a petition to ‘save Uber’ has attracted nearly half a million signatures since the announcement — but in the long term, I don’t think it really matters whether Uber’s license is renewed or not. There’s no way people in London are going to go back to standing on rainy street corners waving their arms at every orange light that goes past. Uber has changed the game. It’s now a legal requirement that black cabs in London accept payment by card — something which even a few years ago was still hugely controversial. And that’s actually really sad, because once upon a time, London taxis had an international reputation for innovation and excellence. The London ‘black cab’ is a global icon, but it’s also an incredibly well-designed vehicle — high-visibility handholds, wheelchair accessible, capable of carrying five passengers and negotiating the narrow tangled streets of one of the world’s oldest cities. London taxis have been regulated since the 17th century. “The Knowledge” — the exam all London black cab drivers are required to pass, generally regarded as one of the hardest examinations of any profession anywhere in the world — was introduced in the 1850s, after visitors travelling to London for the Great Exhibition complained that their hackney-carriage drivers didn’t know where they were going. One of the great joys of visiting London in the days before sat-nav was hailing a cab, giving the driver the name of some obscure pub halfway across the city, and watching as they’d think for one second, nod, and then whisk you there without further hesitation. But in the age of ubiquitous satellite navigation, who cares whether your driver can remember the way from Narrow Street to Penton Place after you’ve had to stand in the rain for fifteen minutes trying to hail a cab? What Uber did — and did incredibly well — is they thought about all the elements of the passenger experience that happen outside the car. Finding a cab. Knowing how much it’s likely to cost. Paying the bill. Recovering lost property. And they made it cheap. They created a user experience that, for the majority of passengers, was easier, cheaper and more convenient than traditional black cabs. It’s no wonder the establishment freaked out.

So what happens now? Maybe Uber win their appeal. Maybe someone else moves into that space. Maybe things get a bit more expensive for passengers — and, frankly, I think they should. I think Uber is too cheap. If you want the luxury of somebody else driving you to your door in a private car, — and for most of us, that IS a luxury — then I believe the person providing that service deserves to make a decent living out of providing it, and one of the most welcome features Uber’s introduced recently is the ability to tip drivers via the app.

Uber has taken a stagnant industry that was in dire need of a kick up the arse, and it’s done it. They’re not the only cab hailing app in the market. Gett, mytaxi (formerly Hailo), Kabbee, private hire firms like Addison Lee — not to mention just phoning a minicab office and asking them to send a car round. In a typical month in London, I’ll use buses, the Tube, national rail services, the Overground, Uber AND black cabs — not to mention a fair bit of walking and cycling. I even have an actual car, which I use about once a month to go to B&Q or IKEA or somewhere. None of them’s perfect, but on the whole, transport in London works pretty well, and I’ve found Uber a really welcome addition to the range of transport services that’s on offer.

But Uber vs TfL is just a tiny taste of what’s coming. I honestly believe that within the next ten years, we’ll be using our smartphones to summon driverless autonomous electric vehicles to take us home after a night out. In all sorts of shapes and sizes, too — after all, why should it take an entire Toyota Prius to transport a 5’2”, 65kg human with a small shoulder bag from Wardour Street to Battersea? We’ll be going to IKEA in a tiny little electric bubble-bike, buying a new kitchen, and taking it home in a driverless van that waits outside whilst we unload it and then burbles off happily into the sunset to pick up the next waiting fare. Never mind the cab wars between the black cab drivers and the minicab drivers — what happens when a robot cab will pick you up anywhere in town and take you home for a quid? A robot cab that doesn’t have a mortgage to pay or kids to feed? That runs on cheap, green energy, that doesn’t get bored or tired or distracted?

There are going to be problems. There are going to be collisions, fatalities, lawsuits, prosecutions, appeals and counter-appeals. And the disruptions will keep coming, faster and faster. Just as Transport for London think they’ve got workable legislation for driverless cars, someone’s going to invent a drone that can fly from Covent Garden to Dulwich carrying a passenger, and whilst they’re busy arguing in court over whether it’s a helicopter or not someone’s gonna shoot one down with a flare pistol and all merry hell’s going to break loose.

What Uber shows us is that technology isn’t going to self-regulate. The digital economy moves too fast for pre-emptive legislation and licensing frameworks. There’s fortunes to made in the months or years between your product hitting the market and the authorities deciding to shut you down. The future of Uber doesn’t depend on getting their TfL license renewed. They’ve been kicked out of entire countries before and it doesn’t seem to have slowed them down. To them this is just a bump in the road. Their future is about being the first company to roll out self-driving cabs, and you can guarantee they’re working, right now, on finding the legislative loopholes in their various target markets that will allow them to launch first and ask questions later.

Black cabs aren’t going away. Buses aren’t going away. Much as I’d love it, they’re not going to decommission all the rolling stock and turn the London Underground network into a giant dodgems track. Technology is going to disrupt, government is going to react — and whilst that model doesn’t always work terribly well, I can’t see any feasible alternative. Engineers are going to solve hard problems. Touch screens, cellular networks, GPS, space travel, battery capacity, biological interfaces, machine learning. Companies like Apple and Google and SpaceX and Tesla are going to put those solutions in our pockets, and in our buildings and on our streets, and companies like Uber and Tinder and AirBnB are going to find ways to turn those solutions into products that you just can’t WAIT to show your friends in the pub.

And, short of the nightmare scenarios my friend Chris has so lovingly documented over on H+Pedia, society will continue. Uber will run surveillance software on your phone that’s a fraction of what the Home Office are doing all day every day, but Greyball isn’t going to cause the downfall of society. Self-driving cars will kill people. Yes, they will. But 3,000 people already die in road traffic accidents every day, and we think that’s normal, and as long as it doesn’t impact our lives directly we just sort of shrug and ignore it and get on with our lives.

I like Uber, and I feel guilty for liking them because I know they do some truly horrible things. I also travel by air, and I eat meat, and I wear leather and I use an iPhone and don’t always recycle. And I used to download MP3s all the time, and feel guilty about it, and then Spotify came along and now I don’t download MP3s any more. And I used to download movies and TV shows, and now I have Amazon Prime and Netflix and I haven’t downloaded a movie in YEARS.

Very few people are extremists. For every militant vegan, there’s someone out hunting their own meat, and a couple of hundred of us who just go with whatever’s convenient. If you’re trying to change the world, ignore the extremists. Don’t outlaw things you think people shouldn’t be doing. Change the world by using technology to deliver compelling alternatives. I want vat-grown burgers and steaks that taste as good as the real thing, and yes, I’ll pay. I want boots and jackets made from synthetic textiles that actually look properly road-worn after a couple of years instead of just falling apart. I want to travel by hypersonic maglev underground train instead of flying. OK, maybe not that last part… I’m writing this from 30,000 feet above the Lithuanian border, the setting sun behind us is catching the wingtips as we soar above an endless sea of cloud, and I’m being reminded how much I love flying. But you could definitely make air travel a LOT more expensive before I stopped doing it completely.

If TfL really want to get rid of Uber, revoking their license isn’t the way to do it. They’ll appeal, you’ll fight, a lot of lawyers will get rich and TfL will either lose and look like an idiot, or win and look like an asshole. How about TfL find the company that’s trying to beat Uber anyway — they’re out there, somewhere — and offer to work with them instead? As Rowland Manthorpe wrote in WIRED yesterday, why don’t we create a socially responsible, employee-owned, ride sharing platform that gives passengers everything Uber does without the institutionalised nastiness and the guilty aftertaste? What ethical transport startup would not jump at the fact to sign up the greatest city in the world as their development partner?

You think the future is about using an iPhone to summon a guy in a Toyota Prius, and the important question is who wrote the app they’re using? Come on, people. This is London calling. We can think a hell of a lot bigger than that.

Generating self-signed HTTPS certificates with subjectAltNames

2017-08-07T15:55:00+00:00

We provide online services via a bunch of different websites, using federated authentication so that if you sign in to our authentication server, you get a *.mydomain.com cookie that’s sent to any other server on our domain.

We use local wildcard DNS, so there’s a *.mydomain.com.local record that resolves everything to 127.0.0.1, and for each developer machine we create a *.mydomain.com.hostname record that’s an alias for hostname, so you can browse to www.mydomain.com.<machine> to see code running on another developer’s workstation, or www.mydomain.com.local to view your own local development code.

This works pretty well, but getting a local development system set up involves running local versions of several different apps – and since Google Chrome now throws a security error for any HTTPS site whose certificate doesn’t include a “subject alternative name” field, getting a bunch of local sites all happily sharing the same cookies over HTTPS proved a bit fiddly.

So… here’s a batch file that will spit out a bunch of very useful certificates, adapted from this post on serverfault.com.

How it works

Get openssl.exe working - I use the version that's shipped with Cygwin, installed into C:\Windows\Cygwin64\bin\ and added to my system path.
Run makecert.bat. If you don't want to specify a password, just provide a blank one (press Enter). This will spit out three files:
- local_and_hostname.crt
- local_and_hostname.key
- local_and_hostname.pfx
Double-click the local_and_hostname.crt file, click "Install Certificate", and use the Certificate Import Wizard to import it. Choose "Local Machine" as the Store Location, and "Trusted Root Certification Authorities" as the Certificate Store.
Open IIS, select your machine, open "Server Certificates" from the IIS snapin, click "Import..." in the Actions panel
Select the local_and_hostname.pfx certificate created by the batch file. If you used a password when exporting your PKCS12 (.pfx) file, you'll need to provide it here
Finally, set up your IIS HTTPS bindings to use your new certificate.

Yay! Security!

Deployment Through the Ages

2017-07-31T16:43:00+00:00

Vanessa Love just posted this intriguing little snippet on Twitter:

In what could be the worst idea ever, i've opened up a google doc - add steps from your manual deployment process - https://t.co/Ag56BudiM5
— Vanessa Love (@fly401) July 31, 2017

And I got halfway through sticking some notes into the Google doc, and then thought actually this might make a fun blog post. So here’s how deployment has evolved over the 14 years since I first took over the hallowed mantle of webmaster@spotlight.com.

2003: Beyond Compare (maybe?)

The whole site was classic ASP – no compilation, no build process, all connection credentials and other settings were managed as application variables in the global.asa file. On a good day, I’d get code running on my workstation, test it in our main target browsers, and deploy it using a visual folder comparison tool. It might have been Beyond Compare; it might have been something else. I honestly can’t remember and the whole thing is lost in the mists of time. But that was basically the process – you’d have the production codebase on one half of your screen and your localhost codebase on the other half, and you’d cherry-pick the bits that needed to be copied across.

Of course, when something went wrong in production, I’d end up reversing the process – edit code directly on live (via UNC share), normally with the phone wedged against my shoulder and a user on the other end; fix the bug, verify the user was happy, and then do a file sync in the other direction to get everything from production back onto localhost. Talk about a tight feedback loop – sometimes I’d do half-a-dozen “deployments” in one phone call. It was a simpler time, dear reader. Rollback plan was to hammer Ctrl-Z until it’s working again; disaster recovery was tape backups of the complete source tree and database every night, and the occasional copy’n’paste backup of wwwroot before doing something ambitious.

Incidentally, I still use Beyond Compare almost daily – I have it configured as my merge tool for fixing Git merge conflicts. It’s fantastic.

2005: Subversion

Once we hired a second developer (hey Dan!) the Beyond Compare approach didn’t really work so well any more, so we set up a Subversion server. You’d get stuff running on localhost, test it, maybe share an http://www.spotlight.com.dylan-pc/ link (hooray for local wildcard DNS) so other people could see it, and when they were happy, you’d do an svn commit, log into the production web server (yep, the production web server – just the one!) and do an svn update. That would pull down the latest code, update everything in-place. There was still the occasional urgent production bugfix. One of my worst habits was that I’d fix something on production and then forget to svn commit the changes, so the next time someone did a deployment (hey Dan!) they’d inadvertently reintroduce whatever bug had just been fixed and we’d get upset people phoning up asking why it was broken AGAIN.

2006: FinalBuilder

This is where we start doing things with ASP.NET in a big way. I still dream about OnItemDataBound sometimes… and wake up screaming, covered in sweat. Fun times. The code has all long since been deleted but I fear the memories will haunt me to my grave.

Anyway. By this point we already had the Subversion server, so we had a look around for something that would check out and compile .NET code, and went with FinalBuilder. It had a GUI for authoring build pipelines and processes, some very neat features, and could deploy .NET applications to IIS servers. This was pretty sophisticated for 2006.

2008: test server and msdeploy

After one too many botched FinalBuilder deployments, we decided that a dedicated test environment and a better deployment process might be a good idea. Microsoft had just released a preview of a new deployment tool called MSDeploy, and it was awesome. We set up a ‘staging environment’ – it was a spare Dell PowerEdge server that lived under my desk, and I’d know when somebody accidentally wrote an infinite loop because I’d hear the fans spin up. We’d commit changes to Subversion, FinalBuilder would build and deploy them onto the test server, we’d give everything a bit of a kicking in IE8 and Firefox (no Google Chrome until September 2008, remember!) and then – and this was magic back in 2008 – you’d use msdeploy.exe to replicate the entire test server into production! Compared to the tedious and error-prone checking of IIS settings, application pools and so on, this was brilliant. Plus we’d use msdeploy to replicate the live server onto new developers’ workstations, which was a really fast, easy way to get them a local snapshot of a working live system. For the bits that still ran interpreted code, anyway.

2011: TeamCity All The Things!

By now we had separate dev, staging and production environments, and msdeploy just wasn’t cutting it any more. We needed something that can actually build different deployments for each environments – connection strings, credentials, and so on. And there’s now support in Visual Studio for doing XML configuration transforms, so you create a different config file for every environment, check those into revision control, and get different builds for each environment. I can’t remember exactly why we abandoned FinalBuilder for TeamCity, but it was definitely a good idea – TeamCity has been the backbone of our build process ever since, and it’s a fantastically powerful piece of kit.

2012: Subversion to GitHub

At this point, we’d grown from me, on my own doing webmaster stuff, to a team of about six developers. Even Subversion is starting to creak a bit, especially when you’re trying to merge long-lived branches and getting dozens of merge conflicts, so we start moving stuff across to GitHub. It takes a while – I’m talking months – for the whole team to stop thinking of Git as ‘unnecessarily complicated Subversion’ and really grok the workflow, but we got there in the end.

Our deployment process at this point was to commit to the Git master branch, and wait for TeamCity to build the development version of the package. This would get built and deployed. Once it was tested, you’d use TeamCity to build and deploy the staging version – and if that went OK, you’d build and deploy production. Like very step on this journey, it was better than anything we’d had before, but had some obvious drawbacks. Like the fact we had several hundred separate TeamCity jobs and no consistent way of managing them all.

2013: Octopus Deploy and Klondike

When we started migrating from TeamCity 6 to TeamCity 7, it became rapidly apparent that our “build everything several times” process… well, it sucked. It was high-maintenance, used loads of storage space and unnecessary CPU cycles, and we needed a better system.

Enter Octopus Deploy, whose killer feature for us was the ability to compile a .NET web application or project into a deployment NuGet package (an “octopack”), and then apply configuration settings during deployment. We could build a single package, and then use Octopus to deploy and configure it to dev, staging and live. This was an absolute game-changer for us. We set up TeamCity to do continuous integration, so that every commit to a master branch would trigger a package build… and before long, our biggest problem was that we had so many packages in TeamCity that the built-in NuGet server started creaking.

This started life as an experimental build of themotleyfool/NuGet.Lucene – which we actually deployed onto a server we called “Klondike” (because klondike > gold rush > get nuggets fast!) – and it worked rather nicely. Incidentally, that NuGet.Lucene library is now the engine behind themotleyfool/Klondike, a full-spec NuGet hosting application – and I believe our internal hostname was actually the inspiration for their project name. That was a lot of fun for the 18 months or so that Klondike existed but we were still running the old NuGet.Lucene codebase on a server called ‘klondike’. It’s OK, we’ve now upgraded it and everything’s lovely.

It was also in 2013 that we started exploring the idea of automatic semantic versioning – I wrote a post in October 2013 explaining how we hacked together an early version of this. Here’s another post from January 2017 explaining how it’s evolved. We’re still working on it. Versioning is hard.

And now?

So right now, our build process works something like this.

Grab the ticket you’re working on – we use Pivotal Tracker to manage our backlogs
Create a new GitHub branch, with a name like 12345678_fix_the_microfleems – where 12345678 is the ticket ID number
Fix the microfleems.
Push your changes to your branch, and open a pull request. TeamCity will have picked up the pull request, checked out the merge head and built a deployable pre-release package (on some projects, versioning for this is completely automated)
Use Octopus Deploy to deploy the prerelease package onto the dev environment. This is where you get to tweak and troubleshoot your deployment steps.
Once you’re happy, mark the ticket as ‘finished’. This means it’s ready for code review. One of the other developers will pick it up, read the code, make sure it runs locally and deploys to the dev environment, and then mark it as ‘delivered’.
Once it’s delivered, one of our testers will pick it up, test it on the dev environment, run it past any business stakeholders or users, and make sure we’ve done the right thing and done it right.
Finally, the ticket is accepted. The pull request is merged, the branch is deleted. TeamCity builds a release package. We use Octopus to deploy that to staging, check everything looks good, and then promote it to production.

And what’s on our wishlist?

Better production-grade smoke testing. Zero-footprint tests we can run that will validate common user journeys and scenarios as part of every deployment – and which potentially also run as part of routine monitoring, and can even be used as the basis for load testing.
Automated release notes. Close the loop, link the Octopus deployments back to the Pivotal tickets, so that when we do a production deployment, we can create release notes based on the ticket titles, we can email the person who requested the ticket saying that it’s now gone live, that kind of thing.
Deployments on the dashboards. We want to see every deployment as an event on the same dashboards that monitor network, CPU, memory, user sessions – so if you deploy a change that radically affects system resources, it’s immediately obvious there might be a correlation.
Full-on continuous deployment. Merge the PR and let the machines do the rest.

So there you go – fourteen years worth of continuous deployments. Of course, alongside all this, we’ve moved from unpacking Dell PowerEdge servers and installing Windows 2003 on them to running Chef scripts that spin up virtual machines in AWS and shut them down again when nobody’s using them – but hey, that’s another story.

Securing Blogger with CloudFlare and HTTPS

2017-07-27T13:57:00+00:00

As you may have read, life is about to get a whole lot harder for websites without HTTPS. Now this site is hosted on Blogger – I used to run my own MovableType server, but I realised I was spending way more time messing around with the software than I was actually writing blog posts, so I shifted the whole thing across to Blogger about a decade ago and never really looked back.

One of the limitations of Blogger is that it doesn’t support HTTPS if you’re using custom domains – there’s no way to install your own certificate or anything. So, since Chrome’s about to crank up the warnings for any websites that don’t use HTTPS, I figured I ought to set something up. Enter CloudFlare, who are really rather splendid.

First, you sign up. (bonus points for them NOT forcing you to choose a password that contains a lowercase letter, an uppercase letter, a number, a special character, the poo emoji and the Mongolian vowel separator).

Second, you tell them which domain you want to protect:

They scan all your DNS records, which takes about a minute – and not only is there a nice real-time progress bar keeping you in the loop, they use this opportunity to play a really short video explaining what's going on. I think this is absolute genius.

Finally, after checking it's picked up all your DNS records properly (it had), you tell your domain registrar to update the nameservers for your domain to CloudFlare's DNS servers, give it up to 24 hours, and you're done. Zero downtime, zero service interruption – the whole thing was smooth, simple, and completely free-as-in-beer.

Yes, I realise this does not encrypt content end-to-end. For what we're doing here, this is absolutely fine. It'll secure your traffic against dodgy hotel wi-fi and unscrupulous internet service providers - and if anyone's genuinely intercepting HTTP traffic between CloudFlare and Google, I'm sure they can think of more exciting things to do with it than mess around with my blog posts.

Having done that, I then had to use the Google Chrome console to track down the resources – photos and the odd bit of script – that were being hosted via HTTP, and update them to be HTTPS. The only thing I couldn't work out how to fix was the search bar that's embedded in Blogger's default page layout – it's injected by JavaScript, it's hosted by Google's CDN (so I can't use any of CloudFlare's clever rewriting tricks to fix it), it's stuck inside an IFRAME, and it points to http://www.dylanbeattie.net/search – see the plain HTTP with no S?

After an hour or so of messing around with CSS, I gave up, posted a question on the ProWebmasters Stack Exchange, and – of course, immediately found the solution; go into Blogger, Layout, find the Navbar gadget, click Edit, and there's an option to switch the nav off entirely.

So there you go. Thanks to CloudFlare, https://www.dylanbeattie.net/ now has a green padlock on it. I don't know about you, but I take comfort in that.

Summer 2017 .NET Community Update

2017-07-21T14:01:00+00:00

Summer here in the UK is normally pretty quiet, but this year there's so much going on around .NET and the .NET community that I thought this would be a great opportunity to do a bit of a round-up and let you all know about some of the great stuff that's going on.

First, there's the news of two new .NET user groups starting up in southern England. Earlier this week, I was down in Bournemouth speaking at the first-ever meetup of the new .NET Bournemouth group, and thoroughly enjoyed it. Three speakers – Stuart Blackler, Tommy Long and me – with talks on leadership, agile approaches to information security, and an updated version of my "happy code" talk I've done at a few conferences already this year. The venue and A/V setup worked flawlessly, there was a strong turnout, and some really good questions and discussion after each of the talks – I think it's going to turn out to be a really engaging group, so if you're in that part of the world, stop by and check them out. Their next few meetup dates are on meetup.com/Net-Bournemouth already.

Next month, Steve Gordon is starting a new .NET South East group based in Brighton, who will be kicking off with their inaugural meetup on August 22nd with Steve talking about Docker for .NET developers.

There's a great post on Steve's blog explaining what he's doing and what he's hoping to get out of the group, and they're also on meetup.com/dotnetsoutheast (and I have to say, they've done an excellent job of branding the Meetup site – nice work!)

It's an exciting time for .NET – between the cross-platform stuff that's happening around Xamarin and .NET Core, new tooling like JetBrains Rider and Visual Studio Code, and the growing number of cloud providers who are supporting C# and .NET Core for building serverless cloud applications, we've come a long, long way from the days of building Windows Forms and databinding in Visual Studio .NET.

If you're interested in really getting to grips with the future of .NET, join us at the Progressive.NET Tutorials here in London in September. With a great line-up of speakers including Julie Lerman, Jon Skeet, Jon Galloway, Clemens Vasters and Rachel Appel – plus Carl Franklin and Rich Campbell from DotNetRocks, and a few familiar faces you might recognise from the London.NET gang – it promises to be a really excellent event. It goes a lot deeper than most conferences – with one day of talks and two days of hands-on workshops, the idea is that attendees don't just go away with good ideas, they actually leave with running code, on their laptops, that they can refer back to when they take those ideas back to the office or to their own projects. Check out the programme, follow #ProgNET on Twitter, and hopefully see some of you there.

Then on Saturday 16th September – the day after Progressive.NET – is the fourth DDD East Anglia community conference in Cambridge. Their call for speakers is now closed, but voting is open until July 29th – so sign up, vote on the sessions you want to see – or just vote for mine if you can't make your mind up ;) - and hopefully I'll see some of you in Cambridge.

Finally, just in case any readers of this blog DON'T know about the London .NET User Group… yep, we have .NET User Group! In London! I know, right? We're on meetup.com/London-NET-User-Group, and on Twitter as @LondonDotNet, and we meet every month at SkillsMatter's CodeNode building near Moorgate.

Our next meetup is on August 8th, with Ana Balica talking about the history and future of HTTP and HTTP/2, and Steve Gordon – and on September 12th we've got Rich Campbell joining us for a Progressive.NET special meetup and presenting the History of .NET as you've never heard it before.

New people, new meetups, new platforms and new ideas. Like I said, it's a really exciting time to be part of the .NET community – join us, come to a meetup, follow us online, and let's make good things happen.

Use Flatscreens

2017-07-04T00:09:00+00:00

This started life as a lightning talk for PubConf after NDC in Sydney, back in August 2016… and after quite a lot of tweaking, editing and learning to do all sorts of fun things with Adobe AfterEffects and Premiere, it's finally on YouTube. The inspiration is, of course, "Wear Sunscreen", Baz Luhrmann's 1999 hit song based on an essay written by Mary Schmich. Video footage and stock photography is all credited at the end of the clip, and the music, vocals, video, audio and, well, basically everything else is by me. Happy listening - and don't forget to use flatscreens :)

Ladies and gentlemen of the class of 2017… use flat screens. If I could offer you only one tip for the future, flat screens would be it. The benefits of flat screens have been proved by Hollywood, whereas the rest of my advice has no basis more reliable than my own meandering experience.

I will dispense this advice... now.

Enjoy the confidence and optimism of greenfield projects. Oh, never mind. You will not appreciate the confidence and optimism of greenfield until everything starts going to hell. But trust me, when you finally ship, you'll look back at the code you wrote and recall in a way you can't grasp now how simple everything seemed, and how productive you really were. Your code is not as bad as you imagine.

Don't worry about changing database providers. Or worry, but know that every company who ever used an OR/M in case they needed to switch databases never actually did it. The real problems in your projects are the dependencies you don't control; the leaking air conditioner that floods your data centre at 5pm on the Thursday before Christmas.

Learn one thing every day that scares you.

Optimise.

Don't reformat other people's codebases; don't put up with people who reformat yours.

Rebase

Don't get obsessed with frameworks. Sometimes they help, sometimes they hurt. It's the user experience that matters, and the user doesn't care how you created it.

Remember the retweets you receive; forget the flame bait. If you succeed in doing this, tell me how.

Keep your old hard drives. Throw away your old network cards.

Refactor.

Don't feel guilty if you don't understand f#. Some of the most productive junior developers I've worked with didn't know F#. Some of the best systems architects I know still don't.

Write plenty of tests.

Be kind to your keys; you'll miss them when they're gone.

Maybe you have a degree; maybe you don't. Maybe you have an open source project; maybe you won't. Maybe you wrote code that flew on the Space Shuttle; maybe you worked on Microsoft SharePoint. Whatever you do, keep improving, and don't worry where your next gig is coming from. There's a big old world out there, and they're always going to need good developers.

Look after your brain. Don't burn out, don't be afraid to take a break. It is the most powerful computer you will ever own.

Launch, even if you have no users but your own QA team.

Have a plan, even if you choose not to follow it.

Do NOT read the comments on YouTube : they will only make you feel angry.

Cache your package dependencies; you never know when they'll be gone for good.

Read your log files. They're your best source of information, and the first place you'll notice if something's starting to go wrong.

Understand that languages come and go, and that it's the underlying patterns that really matter. Work hard to fill the gaps in your knowledge, because the wiser you get, the more you'll regret the things you didn't know when you were young.

Develop in 86 assembler once, but stop before it makes you smug; develop in Visual Basic once, but stop before it makes you stupid.

Read.

Accept certain inalienable truths. Your code has bugs, you will miss your deadlines, and you, too, might end up in management. And when you do, you'll fantasize that back when you were a developer, code was bug-free, deadlines were met, and developers tuned their database indexes.

Tune your database indexes.

Don't deploy your code without testing it. Maybe you have a QA team. Maybe you have integration tests. You never know when either one might miss something.

Don't mess too much with your user interface, or by the time you ship, it will look like a Japanese karaoke booth.

Be grateful for open source code, but be careful whose code you run. Writing good code is hard, and open source is a way of taking bits from your projects folder, slapping a readme on them, and hoping if you put them on GitHub somebody else will come along and fix your problems.

But trust me on the flat screens.

Interview with Channel 9 at NDC Oslo

2017-06-28T16:39:00+00:00

I was in Oslo earlier this month, where – as well as doing the opening keynote, a couple of talks, a workshop on hypermedia systems and PubConf – I had the chance to chat with Seth Juarez from Channel 9 about code, culture, speaking at conferences, and… all kinds of things, really.

The interview's here, or you can watch it over on Channel9.msdn.com - thanks Seth and co for taking the time to put this together!

Interview with habrahabr.ru about HTTP APIs in .NET

2017-05-13T11:32:00+00:00

Next week I'll be in Russia, where I'm speaking about HTTP APIs and REST at the DotNext conference in Saint Petersburg. As part of this event, I've done an interview with the Russian tech site Хабрахабр about the history and future of API development on the web and in Microsoft.NET. The interview's available on their site habrahabr.ru (in Russian), but for readers who are interested but can't read Russian, here's the original English version.

Cathedral in Saint Petersburg, Russia. goodfreephotos.com / Photo by DEZALB.

Q: What kind of APIs are you designing? Where does API design fit into software development?

That’s kind of an interesting question, because I think one of the biggest misconceptions in software is that designing APIs is an activity that happens separately to everything else. Sure, there are certain kinds of API projects – particularly things like HTTP APIs which are going to be open to the public – where it might make sense to consider API design as a specific piece of work. But the truth is that most developers are actually creating APIs all the time – they just don’t realise they’re doing it. Every time you write a public method on one of your classes or choose a name for a database table, you’re creating an interface – in the everyday English sense of the word – that will end up being used by other developers at some point in the future. Other people on your team will use your classes and methods. Other teams will use your data schema or your message format.

What’s interesting, though, is that once developers realize that the code they’re working on will probably form part of an API, they tend to go too far in the other direction. They’ll implement edge cases and things that they don’t actually need, just in case somebody else might need it later. I think there’s a very fine balance, and I think the key to that balance is to be very strict about only building the features that you need right now, but to make those things as reusable and self-explanatory as you can. There’s a great essay by Pieter Hintjens, Ten Rules for Good API Design, that goes into more detail about these kinds of ideas.

The biggest API project I’m working on at the moment is a thing I’m building at Spotlight in the UK, where I work. It’s a hypermedia API exposing information about professional actors, acting jobs in film and television, and various other kinds of data used in the casting industry. We’re building it in the architectural style known as REST – and if you’re not sure what REST is, you should come to my talk at DotNext in Saint-Petersburg and learn all about it. There’s lots of different patterns for creating HTTP APIs – there’s REST, there’s GraphQL, there’s things like SOAP and RPC – but for me, the biggest appeal of REST is that I think the constraints of the RESTful style lead to a natural decoupling of the concepts and operations that your API needs to support, which makes it easier to change things and evolve the API design over time.

Q: One of the most famous applications that was "killed" by backward compatibility is IE. The problem of this browser was that it has too large number of applications for which it was required to have backward compatibility. Problem was solved by adding new application Edge, which is updatable and supports all new standards. Can you give an piece of advice on how not to get caught by that backward-compatibility trap? As an example could it be a modularity which doesn't have layers? May be there is a way to replace API with RESTful API, Service Oriented Architecture or something else?

I’ve been building web applications for a long, long time – I wrote my first HTML page a couple of years before Internet Explorer even existed, back when the only browsers were NCSA Mosaic and Erwise. It’s fascinating to look back at the history of the web, and how the web that exists today has been shaped and influenced by things like the evolution of Internet Explorer – and you’re absolutely right; one of the reasons why Microsoft has introduced a completely new browser, Edge, in the latest versions of Windows is that Internet Explorer’s commitment to backwards-compatibility has made it really difficult to implement support for modern web standards alongside the existing IE codebase.

Part of the reason why that backwards compatibility exists is that, around the year 2000, there was a massive shift in the way that corporate IT systems were developed. There are countless corporations who have bespoke applications for doing all sorts of business operations – stock control, inventory, HR, project management, all kinds of things. Way back in the 1980s and early 1990s, most of them used a central mainframe system and employees would have to use something like a terminal emulator to connect to that central server, but after the first wave of the dotcom boom hit in the late 1990s, companies realised that most of their PCs now had a web browser and an network connection, and so they could replace their old mainframe terminal applications with web applications. Windows had enormous market share at the time, and Internet Explorer was the default browser on most Windows PCs, so lots of organizations built intranet web applications that only had to work on a specific version of Internet Explorer. Sometimes they did this to take advantage of specific features, like ActiveX support; more often I think they just did it save money because it meant they didn’t have to do cross-browser testing. This happened with some pretty big commercial applications as well; as late as 2011, Microsoft Dynamics CRM still offered no support for any browser other than Internet Explorer.

So you’ve got all these companies who have invested lots of time and money in building applications that only work with Internet Explorer. Those applications aren’t built using web standards or progressive enhancement or with any notion of ‘forward compatibility’ – they’re explicitly targeting one version of one browser running on one operating system. And so when Microsoft releases a new version of Internet Explorer, those applications fail – and the companies don’t want to invest in upgrading their legacy intranet applications, so they blame the browser. So we end up with this weird situation where here in 2017, Microsoft are still shipping Internet Explorer 11, which has a compatibility mode where it switches back to the IE9 rendering engine but sends a user agent string claiming that it's IE7. Meanwhile, everyone I know uses Google Chrome or Safari for all their web browsing - but still has an IE shortcut on their desktop for when they have to log in to one of those legacy systems. .

So… to go back to the original question: is there anything Microsoft could have done to avoid this trap? I think there’s a lot of things they could have done. Building IE from the ground up with a modular rendering engine, so that later versions could selectively load the appropriate engine for rendering a particular website or application. They could have made more effort to embrace the web standards that existed at the time, instead of implementing ad-hoc support for things like the MARQUEE tag and ActiveX plugins, which would have avoided the headache of having to support these esoteric features in later versions. The point is, though, none of this mattered at the time. Their focus – the driving force behind the early versions of Internet Explorer – was not to create a great application with first-class support for web standards. They were trying to kill Netscape Navigator and win market share – and it worked.

Q: Let’s imagine someone is going to introduce an API. So they collect some requirements, propose a version and gets feedback. That’s rather simple and straightforward thing. But if there are any hidden obstacles there down the road?

Always! Requirements are going to change – in fact, one of the biggest mistakes you can make is to try and anticipate those changes and make your design ‘future-proof’. Sometimes that pays off, but what mostly happens is that you end up with a much more complicated design purely because you’re trying to anticipate those future changes. Those obstacles are often things outside your control. There’s a change to the law that means you need to expose certain data in a different way. There’s a change to one of the other systems in your organization, or one of your cloud hosting providers announces that they’re deprecating a particular feature that you were relying on.

The best thing to do is to identify something simple and usable, ship it, and get as quickly as you can to a point where your API is stable, there’s no outstanding technical debt, and your team is free to move on to the next thing. That way when you do encounter one of those ‘hidden obstacles’, you have a stable codebase to use as a basis for your solution, and you have a team who have the time and the bandwidth to deal with it. And if by some stroke of luck you don’t hit any hidden obstacles, then you just move on to the next thing on your backlog.

Q: Continue with API design. We’ve released the v1.0 of our API and now v1.1 is approaching. I believe many of us noticed http://example.com/v1/test and http://example.com/v1.1/test or something. What are the best practices (a couple of points) you can think of that can help a developer to design a good v1.1 API in respect to v1.0?

It’s worth reading up on the concept of semantic versioning, (SemVer) and taking the time to really understand the distinction between major, minor and patch versions. SemVer says that you shouldn’t introduce any breaking changes between version x.0 and version x.1, so the most important thing is to understand what would constitute a breaking change for your particular API.

If you’re working with HTTP APIs that return JSON, for example, a typical non-breaking change would be adding a new data field to one of your resources. Clients that are using version 1.1 and are expecting to see this additional field can take advantage of it, whereas clients that are still using version 1.0 can just discard the unrecognised property.

There’s a related question about how you should manage versioning in your APIs. One very common solution is to expose URLs via routing – api.example.com/v1/ as opposed to api.example.com/v1.1 – but if you’re adhering to the constraints of a RESTful system, you really need to understand whether the change in version represents a change in the underlying resource or just the representation. Remember that a URI is a Uniform Resource Identifier, and so we really shouldn’t be changing the URI that we’re using to refer to the same resource.

For example – if we have a resource api.example.com/images/monalisa. We could request that resource as a JPEG (Accept: image/jpeg), or as a PNG (Accept: image/png), or ask the server if it has a plain-text representation of the resource (Accept: text/plain) – but they’re just different representations of the same underlying resource, and so they should all use the same URI.

If – say – you’ve completely replaced the CRM system used by your organization, and so “version 1” of a customer represents a record used in the old CRM system and “version 2” represents that same customer after they’ve been migrated onto a completely new platform, then it probably makes sense to treat them as separate resources and give them different URIs.

Versioning is hard, though. The easiest thing to do is never change anything

Q: .NET Core - what do you think about its API?

When .NET Core was first announced in 2015, back when it was going to be be called .NET Core 5.0, it was going to be a really stripped-down, lightweight alternative to the .NET Framework and the Common Language Runtime. That was an excellent idea in terms of making it easier to port .NET Core to different platforms, but it also left a sizable gap between the API exposed by .NET Core, and the ‘standard’ .NET/CLR API that most applications are built against.

I believe – and this is just my interpretation based on what I’ve read and people I’ve talked to – that the idea was that .NET Core would provide the fundamental building blocks. It would provide things like threading, filesystem access, network access, and then a combination of platform vendors and the open source community would develop the modules and packages that would eventually match the level of functionality offered by something like the Java Class Library or the .NET Framework. That’s a great idea in principle, but it also creates a chicken-and-egg situation: people won’t build libraries for a platform with no users, but nobody wants to use a platform that doesn’t have any libraries.

So, the decision was made that cross-platform .NET needed a standard API specification that would provide the libraries that users and application developers expected to be available on the various supported platforms. This is .NET Standard 2.0, which is already fully supported by the .NET Framework 4.6.1 and will be supported in the next versions of .NET Core and Xamarin. Of course, .NET Core 1.1 is out, and works just fine, and you can use it right now to build web apps in C# regardless of whether you’re running Windows or Linux or macOS, which is pretty awesome – but I think the next release of .NET Core is going to be the trigger for a lot of framework and package developers to migrate their projects across to .NET Core, which in turn should make it easier for developers and organizations to migrate their own applications.

Tram on Moscow Gate Square in Saint Petersburg. freegoodphotos.com / Photo by Dinamik.

API flexibility VS. API precision. One can design a method API so it can accept many different types of values. It’s flexibility. We also can design a method API with lots of rules on input parameters. Both ways are correct. Where is the boundary across these approaches? When should I make a “strict” API and when should I make a more “flexible” design? Don’t forget that you should take backward-compatibility into account.

By implementing an API where the method signatures are flexible, all you’re doing is pushing the complexity to somewhere else in your stack. Say we’re building an API for finding skiing holidays, and we have a choice between

DoSearch(SearchCriteria criteria)

and

DoSearch(string resortName, string countryCode, int minAltitude, int maxDistanceToSkiList)

One of those methods is pretty easily extensible, because we can extend the definition of the SearchCriteria object without changing the method signature – but we’re still changing the behaviour of the system, we’re just not changing that particular method. By contrast, we could add new arguments to our DoSearch method signature, but if we’re working in a language like C# where you can provide default argument values, you won’t break anything by doing that as long as you provide sensible defaults for the new arguments.

At some point, though, you need to communicate to the API consumers what search parameters are accepted by your API, and there’s lots of ways to accomplish that. If you’re building a .NET API that’s installed as a NuGet package and used from within code, then using XML comments on your methods and properties is a great way to explain to your users what they need to specify when making calls to your API. If your API is an HTTP service, look at using hypermedia and formats like SIREN to define what parameter values and ranges are acceptable.

I should add that I think within the next decade, we’re going to start seeing a whole different category of APIs powered by machine learning systems, where a lot of the conventional rules of API design won’t apply. It wouldn’t surprise me if we got an API for finding skiing holidays where you just specify what you want in natural language, and so there’s not even a method signature – you just call something like DoSearch(“ski chalet, in France or Italy, 1400m or higher, that sleeps 12 people in 8 bedrooms, available from 18-25 January 2018”) – and the underlying system will work it all out for us. Those sorts of development in machine learning are hugely exciting, but they’re also going to create a lot of interesting challenges for the developers and designers trying to incorporate them into our products and applications.

Thanks to Alexej Sommer for taking the time to set this up (and for translating my answers into Russian – Спасибо!), and if you're at DotNext next week and want to chat about APIs, hypermedia or any of the stuff in the interview, please come and say hi!

It Works On My Machine!

2017-04-27T15:27:00+00:00

I saw on Twitter this morning that Derick Bailey is looking for people to share their own “works on my machine” stories… and halfway through filling out his survey, I decided this would probably be much more fun if I nicked his survey questions and turned them into headings in a blog post. Mainly ‘cos writing for an audience appeals to me far more than filling out survey – but Derick (and anyone else who cares?) is very welcome to use anything in this post as part of their own research.

What's typically going through your head when you say "works on my machine" to a QA person or another developer?

I think the interesting question here is actually – what did somebody say to you that caused you to respond with “it works on my machine?”

Here’s three fairly common scenarios:

Q: This code throws an exception when we run it on the staging environment…
A: It works on my machine.

Q: How are you getting on with that improvement to the search algorithm?
A: It works on my machine.

Q: Did you get anywhere with that really weird solution to the mapping problem that Chris found on StackOverflow?
A: It works on my machine…

See how in each case, there’s a sort of implicit subtext? See, I think we all understand that there’s often quite a big difference between solving a problem and delivering a solution. In almost all development scenarios, the first step is to get the code you’re working on running locally and doing the right thing on your development system – and often to do that, we have to hack things around. Running web servers as a local admin user. Granting “Everyone Full Control” of all the files in the media folder. Manually tweaking registry keys, installing DLLs, reusing credentials for APIs and external services – there’s a whole lot of stuff that has to happen as well as just writing some code, but most of the time, the code is the focus and the rest just feels like friction.

So… to answer the original question, when I tell somebody something works on my machine, I’m thinking “ok, what else, other than my own code, do we need to do to deploy the solution, close the ticket and move on?” When you’re working on spikes and prototypes, that’s a natural part of the conversation. If you’ve submitted a ticket to QA for final pre-release testing and it doesn’t work, there’s naturally a bit of tension because implicit in the conversation is the fact that somebody thinks you haven’t done your job properly – and “well it works on my machine” can come across as defensive.

Can you share a story about a time when you have said, or thought, this?

Ah, dozens. The most common example for me is when I’m making a change that spans code in 4-5 different projects, so I’ve checked them all out… in one project I’ve added a database column, in another there’s some new HTTP request routing, in the third there’s a new message queue subscriber, and then there’s the new feature code that relies on all of those changes to work properly. And it works on MY machine because locally I’ve already made all of those changes, but it get it working anywhere else, all five projects need to be reviewed, built, packaged, configured and deployed onto the same environment. Or, for another developer to work on it, they need to check out five specific branches from five specific projects and then probably run a couple of configuration scripts as well – and there’s invariable one or two little things that didn’t require any explicit configuration on my own workstation but then it turns out your teammate has enabled WebDAV publishing under Windows Programs and Features and so their IIS configuration isn’t the same as yours.

How do you typically feel when someone says, "works on my machine," to you?

First off, I’m happy. See, I’ve worked with a very small number of developers who didn’t bother doing even this most basic validation of the work they were doing. They’d commit something, open a pull request and ask for a code review, and you’d look at what they did and think “that’s a bit weird”, so you’d wander over and say “hey, can you show me how this feature works?” – and they will actually say “I don’t know, I can’t run that project”.

“Works on my machine” at least indicates that they’ve got all the code checked out, they’ve compiled it, and they’ve actually got it running. That’s a good start. That’s something you can work with. And at that point, it’s a great opportunity to explain things like configuration management or deployment scripts.

Can you share a story about a time when someone said this to you?

We had a problem a few weeks ago where we ported an old project from VS2010 to VS2015, and TeamCity wouldn’t build it properly. An absolute textbook example – another developer comes to me and says “well, it works on my machine; it builds fine and I can run all the tests, but TeamCity won’t run any of the unit tests and so the build keeps failing.” Turned out to be a rogue wildcard somewhere in the TeamCity build config settings that was causing it to pick up unit test DLLs from the \obj\ folders instead of \bin\. Which, of course, doesn’t happen when you’re running tests using Resharper or NCrunch, because those tools are smart enough to understand path conventions.

What are the 3 largest causes of someone saying "works on my machine"?

In my experience? The biggest one is dependencies between multiple projects. The “feature”, the unit of business value we’re trying to deliver, requires changes across several different projects and so those changes need to be coordinated and deployed together in order to run and test the new feature, and it’s very easy to miss a step when you’re trying to capture and package all of those code and configuration dependencies.

Second biggest would have to be mismatches between developer environments. We have some people running Windows 8.1, some people running Windows 10, some people working via remote desktop onto virtual machines in the cloud, and then all the various quirks of people’s individual OS configurations like the aforementioned WebDAV Authoring support.

Third? Probably data. Lookup tables, test records, and code that’s brittle because it depends on specific records existing in a particular state, and when you check out the code it doesn’t include the migration steps or SQL scripts that are necessary to set up those records.

Actually, I’m going to go for four, because the one that bites me all the time – probably once a week – is that when you add a new file to a Visual Studio solution, it doesn’t save the .csproj file by default. The new file gets added to the repo and pushed up to GitHub, but the project reference to that new file still only exists on your local machine. Sometimes it’ll crash the build; sometimes – if it’s an image or a script file or something – it’ll build, pass tests, deploy, and then fail on the test server because the new file isn’t included in the .csproj and so wasn’t included when the deployment package was built. If you think your organisation doesn’t have this problem, search your GitHub repo for commit messages including the phrase “csproj file”…

How do you combat the "works on my machine" problem?

There’s a couple of things that have definitely made a big difference to our team at Spotlight. One is setting up an internal NuGet server (we’re running Klondike), and making sure that if your project code references DLLs or any other static components, those dependencies are managed as NuGet packages. That way the first time you build the project, it’ll download all of those obscure DLLs for you instead of waiting for you to get an error message, look it up on the wiki, etc.

One is giving everybody the ability to build and deploy pre-release packages. We have TeamCity and GitHub configured so that as soon as you open a pull request, TeamCity will try and build a deployable package based on the merge head of your feature branch. This means you, the developer, can get packaged builds of your work in progress, deploy it onto one of our testing environments and see for yourself whether it’s going to work or not. Which means you get the chance to fix the bugs and configuration problems before passing it on to anyone else to review or test.

Oh, and we have something called escrow beers. If you want to introduce a new tool, dependency, language or something into one of our projects, you have to put a six-pack of beer (or a box of cookies or something similarly delicious) in escrow, in the kitchen. Put a post-it note on it saying what it’s for – and then when some poor developer is working late to get a feature out and they discover that they need to install grunt or gulp or bower or yeoman or FAKE or PSake or whatever, there’s goodies in the fridge that will help. That’s doesn’t necessarily inhibit the adoption of new tech, but having to go out and buy beer or cookies makes people stop to think about how their changes might affect their teammates, and so they’ll add some checkout scripts to get the new thing working, or document it on the wiki, or organise a demo to show everyone what they need to know. It's also funny how often somebody thinks a new tool or language is ABSOLUTELY TOTALLY AMAZING and there's no way we can possibly live without it… except it's not actually quite amazing enough to justify walking to a shop at lunchtime and buying a box of cookies.

So there you go… more than you ever wanted to know about code that works on my machine. Thanks again to Derick Bailey for the idea – and just to be clear, you’re welcome to use it, and please credit me by name if you use the information provided here in any follow-up posts or other material.

Robert M. Pirsig on "Stuckness"

2017-04-24T22:39:00+00:00

Robert M. Pirsig, the author of "Zen and the Art of Motorcycle Maintenance", died today aged 88. I've read and re-read that book many times over the years. As somebody who has always found tranquillity in tinkering, I found that "Zen" evokes that meditative, transcendental state that one can achieve whilst doing mechanical maintenance better than anything I've read… and in others, it captures perfectly the awful frustration that can only be experienced when a perfectly simple job turns into a protracted bout of yak-shaving.

Of all the passages in the book, the one that has stayed with me the most is the one I've included below, on the subject of 'stuckness'. After countless evenings spent tweaking and tuning mountain bikes in my dad's garage, experiencing first-hand the frustration of a £800 mountain bike rendered completely useless by stripping the head off a 50p bolt, this passage resonated with me more than anything I think I've ever read. I still think of it frequently, normally when I find myself stuck on some hitherto inconsequential detail of a software project that's somehow managed to derail the entire team for days at a time. The book is excellent, and if you haven't read it I highly recommend it, but the passage in question is here. I hope Mr Pirsig's lawyers don't mind. :)

Stuckness. That's what I want to talk about today.
A screw sticks, for example, on a side cover assembly. You check the manual to see if there might be any special cause for this screw to come off so hard, but all it says is "Remove side cover plate" in that wonderful terse technical style that never tells you what you want to know. There's no earlier procedure left undone that might cause the cover screws to stick.
If you're experienced you'd probably apply a penetrating liquid and an impact driver at this point. But suppose you're inexperienced and you attach a self-locking plier wrench to the shank of your screwdriver and really twist it hard, a procedure you've had success with in the past, but which this time succeeds only in tearing the slot of the screw.
Your mind was already thinking ahead to what you would do when the cover plate was off, and so it takes a little time to realize that this irritating minor annoyance of a torn screw slot isn't just irritating and minor. You're stuck. Stopped. Terminated. It's absolutely stopped you from fixing the motorcycle.
This isn't a rare scene in science or technology. This is the commonest scene of all. Just plain stuck. In traditional maintenance this is the worst of all moments, so bad that you have avoided even thinking about it before you come to it.
The book's no good to you now. Neither is scientific reason. You don't need any scientific experiments to find out what's wrong. It's obvious what's wrong. What you need is an hypothesis for how you're going to get that slotless screw out of there and scientific method doesn't provide any of these hypotheses. It operates only after they're around.
This is the zero moment of consciousness. Stuck. No answer. Honked. Kaput. It's a miserable experience emotionally. You're losing time. You're incompetent. You don't know what you're doing. You should be ashamed of yourself. You should take the machine to a real mechanic who knows how to figure these things out.
It's normal at this point for the fear-anger syndrome to take over and make you want to hammer on that side plate with a chisel, to pound it off with a sledge if necessary. You think about it, and the more you think about it the more you're inclined to take the whole machine to a high bridge and drop it off. It's just outrageous that a tiny little slot of a screw can defeat you so totally.
What you're up against is the great unknown, the void of all Western thought. You need some ideas, some hypotheses. Traditional scientific method, unfortunately, has never quite gotten around to say exactly where to pick up more of these hypotheses. Traditional scientific method has always been at the very best, 20-20 hindsight. It's good for seeing where you've been. It's good for testing the truth of what you think you know, but it can't tell you where you ought to go, unless where you ought to go is a continuation of where you were going in the past. Creativity, originality, inventiveness, intuition, imagination..."unstuckness," in other words...are completely outside its domain.
We're still stuck on that screw and the only way it's going to get unstuck is by abandoning further examination of the screw according to traditional scientific method. That won't work. What we have to do is examine traditional scientific method in the light of that stuck screw.
We have been looking at that screw "objectively." According to the doctrine of "objectivity," which is integral with traditional scientific method, what we like or don't like about that screw has nothing to do with our correct thinking. We should not evaluate what we see. We should keep our mind a blank tablet which nature fills for us, and then reason disinterestedly from the facts we observe.
But when we stop and think about it disinterestedly, in terms of this stuck screw, we begin to see that this whole idea of disinterested observation is silly. Where are those facts? What are we going to observe disinterestedly? The torn slot? The immovable side cover plate? The color of the paint job? The speedometer? The sissy bar? As Poincaré would have said, there are an infinite number of facts about the motorcycle, and the right ones don't just dance up and introduce themselves. The right facts, the ones we really need, are not only passive, they are damned elusive, and we're not going to just sit back and "observe" them. We're going to have to be in there looking for them or we're going to be here a long time. Forever. As Poincaré pointed out, there must be a subliminal choice of what facts we observe.
The difference between a good mechanic and a bad one, like the difference between a good mathematician and a bad one, is precisely this ability to select the good facts from the bad ones on the basis of quality. He has to care! This is an ability about which formal traditional scientific method has nothing to say. It's long past time to take a closer look at this qualitative preselection of facts which has seemed so scrupulously ignored by those who make so much of these facts after they are "observed." I think that it will be found that a formal acknowledgment of the role of Quality in the scientific process doesn't destroy the empirical vision at all. It expands it, strengthens it and brings it far closer to actual scientific practice.
I think the basic fault that underlies the problem of stuckness is traditional rationality's insistence upon "objectivity," a doctrine that there is a divided reality of subject and object. For true science to take place these must be rigidly separate from each other. "You are the mechanic. There is the motorcycle. You are forever apart from one another. You do this to it. You do that to it. These will be the results."
This eternally dualistic subject-object way of approaching the motorcycle sounds right to us because we're used to it. But it's not right. It's always been an artificial interpretation superimposed on reality. It's never been reality itself. When this duality is completely accepted a certain nondivided relationship between the mechanic and motorcycle, a craftsmanlike feeling for the work, is destroyed. When traditional rationality divides the world into subjects and objects it shuts out Quality, and when you're really stuck it's Quality, not any subjects or objects, that tells you where you ought to go.
By returning our attention to Quality it is hoped that we can get technological work out of the noncaring subject-object dualism and back into craftsmanlike self-involved reality again, which will reveal to us the facts we need when we are stuck.
Let's consider a reevaluation of the situation in which we assume that the stuckness now occurring, the zero of consciousness, isn't the worst of all possible situations, but the best possible situation you could be in. After all, it's exactly this stuckness that Zen Buddhists go to so much trouble to induce; through koans, deep breathing, sitting still and the like. Your mind is empty, you have a "hollow-flexible" attitude of "beginner's mind." You're right at the front end of the train of knowledge, at the track of reality itself. Consider, for a change, that this is a moment to be not feared but cultivated. If your mind is truly, profoundly stuck, then you may be much better off than when it was loaded with ideas.
The solution to the problem often at first seems unimportant or undesirable, but the state of stuckness allows it, in time, to assume its true importance. It seemed small because your previous rigid evaluation which led to the stuckness made it small.
But now consider the fact that no matter how hard you try to hang on to it, this stuckness is bound to disappear. Your mind will naturally and freely move toward a solution. Unless you are a real master at staying stuck you can't prevent this. The fear of stuckness is needless because the longer you stay stuck the more you see the Quality...reality that gets you unstuck every time. What's really been getting you stuck is the running from the stuckness through the cars of your train of knowledge looking for a solution that is out in front of the train.
Stuckness shouldn't be avoided. It's the psychic predecessor of all real understanding. An egoless acceptance of stuckness is a key to an understanding of all Quality, in mechanical work as in other endeavors. It's this understanding of Quality as revealed by stuckness which so often makes self-taught mechanics so superior to institute-trained men who have learned how to handle everything except a new situation.
Normally screws are so cheap and small and simple you think of them as unimportant. But now, as your Quality awareness becomes stronger, you realize that this one, individual, particular screw is neither cheap nor small nor unimportant. Right now this screw is worth exactly the selling price of the whole motorcycle, because the motorcycle is actually valueless until you get the screw out. With this reevaluation of the screw comes a willingness to expand your knowledge of it.
- from "Zen and the Art of Motorcycle Maintenance" by Robert M Pirsig (September 6, 1928 – April 24, 2017)

There’s a problem with the phalange!

2017-04-21T15:55:00+00:00

Yesterday I was throwing together a quick Entity Framework prototype to inspect and wrangle some data held in one of our legacy databases. I’m using the Entity Framework “Code first from Database” approach, where you use the tooling to generate your initial model for you but thereafter you modify it by hand. One of the tables I’m working with here is called LookupRanges, so when I generated a bunch of entities and DbSet<> mappings, I was a bit surprised when I ended up with a class called LookupRanx in my new model.

It took a minute or two to ascertain that yes, Entity Framework had mapped my LookupRanges (plural) table name onto a class called LookupRanx. But where on earth did that ‘Ranx’ come from? My hunch here is that somebody who worked on this pluralization code remembered that the English word phalanx has the plural form phalanges – and so implemented a rule that says ‘any word ending in –anges should be singularizaed to –anx. Out of curiousity, I dug out my huge ASCII file of English words, found all the words ending in *anges, and hacked up a quick SQL script to create tables named for all these words so I could run them through EF and see what class names were generated.

Well, it gets ‘changes’ and ‘phalanges’ right – and gets literally every other case wrong.

Now this, to me, is an outstanding example of one of the biggest problems in software development… smart people like working on things that are interesting, and will frequently spend time doing something that’s interesting instead of something that’s important.

Writing a library that can singularize and pluralize English words is fascinating. It’s a never-ending problem with dozens of rules and hundreds of edge cases, and you learn a lot of weird and cool esoteric facts about language and etymology whilst you’re doing it. But in this instance, something started out as a good idea (“hey – wouldn’t it be cool if the model generator would convert plural table names to singular class names?”), and got bogged down in edge cases (“is the plural of ‘tableau’ really ‘tableaux’?”) and – in this instance – ended up with a bizarre bug because one of those so-called edge cases actually ended up breaking the default – and entirely correct – behaviour.

First, phalanges is extremely unlikely to ever show up as the name of a table in an Entity Framework database model. I can think of dozens of real-world scenarios where you’d end up with a table name ending with –Ranges, –Exchanges or –Interchanges, but I’m honestly struggling to think of any remotely likely scenario where you have a Phalanges table in a SQL Server database. This is the kind of thing where the ticket or the user story probably just says ‘implement pluralization’, and then there’s no sub-prioritization or further analysis about just how much pluralization needs to be implemented. Second – it’s kind of a stupid edge case. We’re not trying to win points on University Challenge here, we’re building software. In contemporary English, phalange is an acceptable singular form, and phalanxes is an acceptable plural form. There’s no reason at all why they needed to implement support for this particular edge case. And third: if you really found yourself in a scenario where you had to map the Phalanges table to the Phalanx class, you can just rename it. It’s easy. Visual Studio has first-class support for this kind of refactoring.

And, as if that wasn’t confusing enough, there’s actually source code for an EnglishPluralizationService.cs on Microsoft’s GitHub repository. Somebody obviously had a lot of fun building this, tracking down all those bizarre little edge cases like seraph/seraphim, hippopotamus/hippopotami – but according to this implementation, the plural of phalanx is… go on. Go and take a look.

Now, though, I’m going to eat an oranx and check my email. Using Microsoft Exchanx, of course.

The Pursuit of APIness: The Secret to Happy Code

2017-04-03T17:19:00+00:00

I'll be giving a new talk at the London.NET User Group meetup here in London next Tuesday, based on an idea I've had rattling around for a decade or more now. See, it seems to me that over the course of my career, there's been a strong correlation between happy developers and successful projects. I can't think of any examples where a miserable death-march project has resulted in high-quality working software, and I can't think of too many instances where a group of happy, motivated developers has failed to deliver a working product. I've been thinking around this idea for a while, and started looking at it in terms of user experience – both the user experience that we as developers are creating for our end users, but also the 'user experience' that's being provided by the libraries, frameworks and tools that we're using to do our jobs. Here's the talk synopsis:

We spend our lives working with systems created by other people. From the UI on our phones to the cloud infrastructure that runs so much of the modern internet, these interactions are fundamental to our experience of technology - as engineers, as developers, as users - and user experiences are viral. Great user experiences lead to happy, productive people; bad experiences lead to frustration, inefficiency and misery.
Whether we realise it or not, when we create software, we are creating user experiences. People are going to interact with our code. Maybe those people are end users; maybe they're the other developers on your team. Maybe they're the mobile app team who are working with your API, or the engineers who are on call the night something goes wrong. These may be radically different use cases, but there's one powerful principle that works across all these scenarios and more. In this talk, we'll draw on ideas and insight from user experience, API design, psychology and education to show how you can incorporate this principle, known as discoverability, into every layer of your application. We'll look at some real-world systems, and we'll discuss how discoverability works with different interaction paradigms. Because, whether you're building databases, class libraries, hypermedia APIs or mobile apps, sooner or later somebody else is going to work with your code - and when they do, wouldn't it be great if they went away afterwards with a smile on their face?

If that sounds interesting (or if you think I'm completely wrong and you want to come along and heckle!), sign up at the SkillsMatter website and come along on Tuesday 11th. Hope to see you there.

Goodbye ECMAScript; hello UKMAScript!

2017-03-22T18:33:00+00:00

The UK government has announced it will trigger Article 50 on March 29th, beginning the two-year process of the United Kingdom leaving the European Union.

Once the UK trigger #Article50, UK web devs will have only two years to define a replacement for ECMAScript & migrate all their web apps
— Nat Pryce (@natpryce) March 22, 2017

That’s right - it will no longer be legal for British web developers to run ECMAScript, since the ECMAScript specification is controlled by the European Computer Manufacturers’ Association. We’re happy to announce that as of today, top engineers are starting work on a superior British programming language called UKMAScript.

UKMAScript will extend the core language specification with the following enhancements, which we believe will provide a massive boost to the UK tech industry and offset the immeasurable damage caused when all our EU colleagues and collaborators decide to exercise their freedom of movement.

Along with NaN and Infinity, UKMAScript will support a new primitive numeric value called MoneyForTheNhs, whose value is defined to be exactly 3.5x10⁸ until it’s used as an argument to any function, at which point its value will be silently changed to zero after the function has returned.
The Math.round() method will behave as before, except Math.round(0.52) will now return Number.MAX_VALUE. Math.round(0.48) will return a new constant Number.TRAITORS and any attempt to use this value in calculations will throw a TreasonError
A new “illogical implication” operator !#> will be introduced. This is syntactically similar to the notion of logical implication in Boolean algebra, but designed to allow the scope of arguments to be massively exaggerated. For example, the statement (leave_eu !#> leave_customs_union && leave_eea) will implicitly bind the values of leave_customs_union and leave_eea to the value of leave_eu, despite this dependency not being expressed anywhere else in the codebase.
Along with null and undefined, a new language primitive brexit will be introduced. This has the special equality semantics (brexit == brexit) == undefined. typeof(brexit) will return the value “hard”, and attempting to evaluate brexit.valueOf() will throw a TreasonError.
UKMAScript features a new parallel programming paradigm implemented via the Referendum.Invoke() method. This causes a thread to break away from the main sequence of program control and attempt to continue execution despite no longer having access to any processing capabilities or shared resources of the host system. Note that if a thread A has called Referendum.Invoke(), any child process B attempting to call Referendum.Invoke() will be summarily ignored by process A on the grounds that it’s clearly developed a fault.

UKMAScript ships with no standard library or runtime, but the UKMAScript language committee assures us that platform vendors are lining up to deliver first-class support for the new language.

To further promote the popularity of UKMAScript, the only alternative permitted once Article 50 has been invoked is a new language called LABOUR, which takes many of the core language principles of COBOL-64 but is only accessible using the GNU/Corbyn compiler. This compiler has a tremendously exciting installation routine but then doesn’t actually do anything other than occasionally create internal process deadlocks for no reason.

It's a bug! It's a feature! It's… a limitation of the fundamental design of your test framework?

2017-03-07T14:26:00+00:00

As some of you probably know, I'm a big fan of NCrunch. When I'm coding in C#, NCrunch gets a CPU core and a whole screen to itself (yes, I don't really write code on a system that looks like this) and sits there quietly running all my tests, all the time, and telling me the second I break anything.

I'm also a big fan of testing things that are as close to production behaviour as you can. Unit tests are great for informing the design of your components, but without integration testing you can't be sure they're actually going to work when you stick them together.

@ThePracticalDev @testobsessed pic.twitter.com/uKvhzoEPF7
— George Dinwiddie (@gdinwiddie) January 23, 2016

So on my current project, there's a suite of unit tests using FakeItEasy and assertions, and then a suite of integration tests that connect to the live API, follow the various hypermedia links, throw assorted JSON objects at the PUT and POST endpoints to see how they respond, and then call DELETE to clean up when they're done. And, just to keep us honest, we've got a post-deploy step in our Octopus Deploy script that will actually run the integration test suite as part of the deployment process, and roll the whole thing back if any of the tests fail. Another small step on the road to truly continuous deployment.

Anyway. Last week, I push a release to our dev environment, and a whole load of tests fail. Which is weird, because it worked on MY machine. And it worked on my machine when I pointed my local codebase at the database in the dev environment. And – here's the fun part – it worked on my machine when I pointed the entire test suite at the dev environment. So I start eliminating variables. One of the first things I pick up on is that my local test runner is NCrunch, whereas the post-deploy step is using nunit-console. So I run the local integration tests using nunit-console and – bang. Failures. Which is good, because I know what's causing the weirdness, but weird, because tests are supposed to either pass or fail regardless of what test runner you're using.

So I dig a little deeper, and I end up with what looks to me like a bug in NCrunch. See, we're using the TestCaseSource attribute to generate test cases for the API tests, and – because all we need is a bunch of different JSON objects – we're just spinning up new anonymous objects and passing them in as test cases.

Here's two anonymous objects:

var testCase1 = new { forenames = null, surname = "Batman" }
var testCase2 = new { forenames = String.Empty, surname = "Batman" }

What I noticed is that if you generate these two test cases, NCrunch will only see them as a single test – which I assumed was because their ToString() representations are equal, because null and String.Empty both return String.Empty when you ToString() them in this situation. So I opened a post about it on the NCrunch forums, even going so far as to suggest using GetHashCode() when enumerating test names, and got this really interesting response from Remco Mulder, the NCrunch lead developer:

Tests must be uniquely identifiable between execution and discovery runs. This isn't important for a tool like the nunit console runner where a test can be discovered and executed within the same process call (and thus identified by its memory address), but for a tool like NCrunch, there's no way to run the test or collect data from it without this. As you've identified, generated tests with a null parameter and an empty string will return the same result under .ToString(), so NCrunch can't tell them apart.

The only way to solve this is to change the design of your code. Try using the NUnit .SetName() method to give each of your generated tests a distinctive name.

Unfortunately .GetHashCode() is not a reliable solution to this problem as this method is not designed to generate the same identifier across different processes. This method returns different results under x86 vs x64, and under .NET Core it will actually return an entirely different result for each process. Because your code is responsible for generating the tests, the problem can only be solved within your own code.

I thought this was a really interesting insight into how a tool like NCrunch has to deal with situations that an in-process test runner like nunit-console will probably never encounter. It also turns out I’d dismissed that very warning a few weeks earlier – when it cropped up in response to an unrelated issue which produced the same symptons – and sure enough, after clicking the “Show all hidden warnings” button on the NCrunch toolbar, the warning popped back up – along with a very detailed explanation of what was causing it:

Plus, I had no idea that NUnit has a TestCaseData interface with a SetName() method on it, which gives a much nicer way of presenting these test cases in both NCrunch and NUnit. I've ended up with something akin to:

public static IEnumerable TestData() {
  foreach (var data in new[] { null, String.Empty }) {
    var testCase = new { forenames = data, surname = "Batman" };
    var json = JsonConvert.SerializeObject(testCase);
    yield return new TestCaseData(testCase).SetName(json);
  }
}

Oh, and if you're interested, the deployment failures were because of a weird validation rule that treats null as missing, which is fine, but String.Empty as an empty string which violates a string length constraint. Which is wrong, and now the API doesn't do it any more. This is just another reason why integration testing is a good idea. So there you have it – a bug that wasn’t a bug, a crash-course in how NUnit and NCrunch actually work behind the scenes, and a TIL for naming your NUnit tests explicitly. Happy Friday.

Progressive.NET 2017 : Call for Themes

2017-02-22T14:00:00+00:00

In September, SkillsMatter will be hosting the eighth annual Progressive.NET Tutorials. Over the next few months, the programme committee – including me – will be working to create a line-up of themes, workshops, talks and speakers that reflects the state of the art in .NET here in 2017. We’ll be opening our call for papers next month, but before we do, we’d like your help. Yes, you!

What sets the Progressive.NET Tutorials apart from most conferences is our emphasis on deep-dive half-day workshops. Something between a normal conference talk and a full training course, the idea is that you go away afterwards with running code, on your own laptop, that you’ve written during the workshop and can refer back to when you’re trying to implement the things you’ve learned. Over the years we’ve introduced dozens of new ideas and technologies to the wider .NET community – from technologies like F#, NHibernate and OpenRasta, to patterns like machine learning, event sourcing and continuous deployment.

We’ve got loads of ideas for themes, tracks and workshops this year, but we’d like your input. What do you want to see? What’s “progressive” in your corner of the .NET ecosystem? Some of the themes we’re already talking about are:

.NET on Linux in Production

OK, so your .NET Core application runs on Linux – awesome. What else do you need to know? Security? Configuration management? Monitoring, infrastructure? What about tools like Nginx, HAProxy and Varnish? How can you combine the power of .NET Core runtime with the maturity and flexibility of the Linux platform?

Contributing to .NET Core and Open Source

.NET Core is now part of a rich ecosystem of open source projects, but even for experienced developers, the journey from using open source to actually contributing can be daunting. Want to learn more about contributor licenses, workflows, issues and how to find your way around an unfamiliar codebase?

Cloud Native and Serverless

Ten years ago we were talking about dumping physical servers for virtual servers… now we’re talking about getting rid of servers completely. Cloud native is a whole new world for app developers. 12-factor apps, microservices, API-first development and containerisation are changing the way we approach application development – and the “big three” cloud platforms - .NET Core. AWS Lambda, Google Cloud Functions and Azure Functions -now all support running serverless code built with .NET Core 1.1. So what can you do with it? What’s involved in designing, implementing and deploying serverless and cloud native applications?

Mobile, Desktop and Beyond

At one extreme, we’re deploying microservice apps onto serverless infrastructure. At the other extreme, people running .NET on a wider range of devices than ever before. Xamarin gives us a true cross-platform development toolchain for building native apps for Android and iOS devices. Libraries like Unity are helping C# developers build virtual worlds, from interactive data visualisation tools to launching Kerbals into space. HoloLens, Kinect and the latest generation of VR headsets are letting us interact with applications in all sorts of unprecedented ways, and with .NET Core and Windows Nano Server, we’re even seeing .NET running on the Internet of Things.

Agree? Disagree? Did we miss anything?

What do you think? What do YOU want to see? Akka.NET? Hexagonal architecture? ES.Next? What would you love to spend half-a-day learning about – discussing principles and patterns, asking questions, and going away with running code on your laptop that you can refer back to?

Comment here, find me on Twitter (@dylanbeattie), drop me an email, or come and say hi at the next London.NET User Group meetup, and let me know what you think. And let’s make this the best Progressive.NET Tutorials yet.

Bring back alt.NET? But… why?

2017-02-20T12:50:00+00:00

Pull up a chair, dear reader. I’m going to tell you why I think we should revive alt.NET, but first, we need to establish a little context, which means it’s time for a good old-fashioned origin story.

I’ve been building web apps professionally since before they were called web apps, since the days when IIS was part of the Windows NT 4 Option Pack. I cut my teeth writing Active Server Pages, although unlike most of the ASP crowd, I wrote mine in JScript. That’s right, kids — I was running JavaScript on web servers back when The Matrix didn’t have any sequels. And I loved it. The web was a simpler place in those days, and JScript ASP provided a lightweight, expressive language and runtime that did 99% of everything I ever needed to do. But it was obvious that the Microsoft had a vision for the future of web development, and it wasn’t about JScript and what we now call classic ASP.

I stuck it out with JScript and ASP for a long while, but eventually a project came along that clearly required something with a bit more oomph. Multi-tenancy, internationalisation, that kind of thing. So I brushed up on my C# chops and started building WebForms. C# was — and is — a truly wonderful language to work in, but WebForms? Not even close. I was banging my head against a wall, battling with the .NET framework daily to deliver even the most basic features. The elegant, expressive programming models provided by HTTP and HTML were gone, hidden beneath endless layers of leaky abstractions and fragile redirection. I was unhappy and I was unproductive, but it was incredibly easy to rationalise the misery as a necessary part of the learning curve. It didn’t help that the few developers I discussed it with seemed to be using ASP.NET quite happily and didn’t see any problem at all with the idea of <form runat=”server” /> and OnItemDataBound.

So here’s the scene. It’s 2007, I’m fed up, I haven’t shipped any working code for literally months, and — probably via Scott Hanselman’s blog — I start hearing noises about something called “alt dot net”. Now, this sounded exciting. Unfortunately, many of the original posts and articles have been lost to the mists of time and website redesigns, but the original movement defined itself as:

What is ALT.NET?

1. You’re the type of developer who uses what works while keeping an eye out for a better way.

2. You reach outside the mainstream to adopt the best of any community: Open Source, Agile, Java, Ruby, etc.

3. You’re not content with the status quo. Things can always be better expressed, more elegant and simple, more mutable, higher quality, etc.

4. You know tools are great, but they only take you so far. It’s the principles and knowledge that really matter. The best tools are those that embed the knowledge and encourage the principles

To somebody drowning in the prescribed chaos of ASP.NET WebForms, that sounded like a pretty attractive set of principles.

Now, before we go any further, I’d like to share two things by way of qualifying the rest of this post. Firstly, I’m talking very specifically here about the alt.NET movement as it happened here in the UK, and why it mattered to me, personally, as a developer. Alt.NET as an international movement generated a lot of interest, a lot of opinions and a lot of controversy, and yes, to some extent we’re reviving the alt.NET “brand” because this very controversy has given it a degree of recognition among the tech community that transcends geography and specialisation. But here in the UK, I think alt.NET caught just the right people, at just the right time, and I believe many of those people found it to be a really positive thing to be part of.

Second, I’m well aware that it’s not all smooth sailing — to highlight one recent example, last week’s post from JetBrains about the licensing dispute with a Microsoft debugging component they were using in Rider generated a lot of controversy. I know people on both sides of that particular exchange, and until we have a bit more clarification about exactly what’s happened, all I’ll say is that having multiple vendors working on commercial IDEs, targeting an open source .NET Core platform, is somewhat unprecedented, and there’s bound to be the odd bump along the way.

OK, back to alt.NET. In February 2008, the first alt.NET UK ‘unconference’ took place here in London. For me, it was a revelation. It was about openly challenging an orthodoxy that had become so established it was easy to think it was completely non-negotiable. Instead of “you do Microsoft.NET or you do open source”, it was “find what works for you, and don’t be afraid to mix it up”. Here was a loose-knit community of developers who were cherry-picking the bits of .NET that they liked and happily ignoring the rest. Don’t like WebForms? Cool, let me show you FubuMVC and Monorail. Don’t like Windows? Check out the Mono project. Having nightmares about SOAP, WSDL and DISCO? Here, have a look at this thing called REST. Oh, and by the way, here’s a bunch of neat ideas from Ruby and Java and Haskell that you might be interested in.

It suddenly became apparent that over time “.NET” had become an umbrella term for a whole gamut of languages, frameworks and technologies; some of them were really quite good, some of them were pretty poor, but you didn’t have to use all of them.

It was that first alt.NET UK conference that inspired me to start blogging — in fact, my very first blog post ever was a write-up of the event. I stopped relying so much on MSDN documentation, and I started going to user group meetings. I stopped writing stored procedures (I know, right?) and started using ORM’s — Linq-to-SQL, NHibernate, Castle ActiveRecord, and always with a very definite mindset of “use what works, ship working code, but keep your eyes open for something better”. I never took the plunge with Monorail or Fubu, but when Microsoft came out with ASP.NET MVC, I found it a perfect antidote to the bloated misery of WebForms. I remember vividly a quote from a Scott Hanselman podcast: “you don’t need a Repeater control; you’ve got a for() loop.” — that was the ASP.NET MVC ethos, and I loved it. My team launched our first ASP.NET MVC project in August 2008, whilst it was still on RC3. By the time it went beta, we’d put nearly a million pounds in revenue through it. The WebForms project was put ‘on hold’, and we never went back to it.

There were three alt.NET UK conferences. I don’t remember exactly when the third one was — probably 2009? — but I remember Ian Cooper saying at the time that there was unlikely to be another one any time soon. It felt like the right people had found each other, the conversations had started, the changes were happening. It wasn’t just in the UK, either. Dozens of blogs and meetup groups were spawned in the wake of alt.NET — a handful of them are still going, in Melbourne and Sydney, Paris, New York, and Brighton. Twitter was taking off in a big way; StackOverflow (built on ASP.NET MVC) was changing the way developers asked for help and shared solutions. We did a thing, and it worked.

So here in 2017, why are we talking about doing it again? For me, the answer’s simple. Partly, it’s just the passage of time. For every developer who has loudly and publicly abandoned the .NET platform, there’s a company somewhere whose investment in .NET isn’t going away anytime soon — and in the years since alt.NET, thousands of Comp Sci graduates have left university, landed their first job and ended up working on .NET. Sure, a lot of them are probably happy just to show up, write code, get paid and go home — but I suspect there’s also a lot of them who will go on to do really great things, and who haven’t yet grasped the power and the flexibility of the platform they’re using, and the community that’s built up around it.

But more than that, I don’t want to see the post-alt .NET community become an echo chamber. The ideas that were radical a decade ago have ossified into “best practice”, and it’s time to kick things up again. My personal and professional investment in .NET runs deep, because I sincerely believe that it’s a platform and a community worth investing in. I really enjoy working in .NET. C# and JavaScript are my languages of choice. but I like the fact that I can explore F# and TypeScript without having to learn a whole new platform to go with them. I’m a big fan of — and very occasional contributor to — open-source libraries like Dapper, NancyFX, Newtonsoft.Json, Shouldly and Moq. I’m also involved in running the London.NET User Group, and lately it feels like we’re seeing a lot of familiar faces and retreading a lot of familiar ground. Which is comfortable, and reassuring — and, yes, it’s fun; I love being a part of this community, and count the people I’ve met through it among my dearest friends. But I miss the cross-pollination, the fresh faces and the new ideas and that were such a vital part of alt.NET, and I want to see what we can do to reinvigorate that.

I want us to reach the junior developers — and aspiring future developers — who are looking around for a fast and free way to build their first web apps, and help them get started with .NET Core. I want to reach the people who have always dismissed .NET because they don’t want to run Windows and let them know about things like Visual Studio for Mac and JetBrains Rider. I want to learn more about projects like Unity, and new platforms like HoloLens — not just the “ooh, shiny!”, but to understand the patterns and principles that developers are using to create great products using these tools. And I’d like to reconnect with the people who have abandoned .NET in the years since that first wave, and say “hey! What are you working on? How’s it going? And whilst we’re chatting, have you seen what’s happened to .NET since you last looked at it?”

It’s 2017. .NET Core is open source, Entity Framework is open source — even Windows Live Writer is open source. Bash runs on Windows, SQL Server runs on Linux, and Microsoft is doing some genuinely innovative things. Visual Studio 2017 is right around the corner, and .NET is a free, fast, cross-platform development system that you can use to build just about anything. We’ve got cloud-native applications, running on hosting so cheap it’s basically free. We’ve got Universal Windows Apps running on phones, tablets, laptops, desktops and consoles. We’ve got Xamarin and .NET Core bringing .NET to Linux, Mac, iOS and Android. We’ve got VR headsets and 3D printers and autonomous drones and all sorts of fascinating and unprecedented ways to make our software do cool things… and I think it’s high time we broke down some barriers, shared our ideas and tried to restart some of those conversations.

This essay is also published at medium.com/altdotnet.

Naming Things is Hard: Spotlight Edition

2017-02-14T14:58:00+00:00

Like most specialist industries, software is rife with mainstream English words that we’ve taken and misappropriated to mean something completely different. Show business is no different. The software team here at Spotlight sits smack-bang in the intersection between these two specialist fields, and so when we’re talking to our customers and product owners about the systems we build, it’s very important to understand the difference between typecasting and type casting, and exactly what sort of actor model we’re talking about. We therefore present this delightful “double glossary” of everyday terms that you’ll hear here at Spotlight Towers. Because as we all know, there’s only two hard problems in software: cache invalidation, naming things, and off-by-one errors.

Actor

Software: A mathematical model of concurrent computation that treats "actors" as the universal primitives of concurrent computation.
Showbiz: A person whose profession is acting on the stage, in films, or on television.

Agent

Software: A software agent is a computer program that acts for a user or other program in a relationship of agency
Showbiz: A person who finds jobs for actors, authors, film directors, musicians, models, professional athletes, writers, screenwriters, broadcast journalists, and other people in various entertainment or broadcast businesses.

Callback

Software: Any executable code that is passed as an argument to other code, which is expected to call back (execute) the argument at a given time.
Showbiz: A follow-up interview or audition

Casting

Software: Explicitly converting a variable from one type to another
Showbiz: Employing actors to play parts in a film, play or other production. Also the act of doing same.

Client

Software: The opposite of a server
Showbiz: An actor, specifically in the context of the actor’s relationship with their agent or manager Internally at Spotlight we have both internal and external clients/customers

Client Profile

Software: A subset of the .NET framework intended to run on mobile and low-powered devices
Showbiz: An actor’s professional CV, as it appears on their agent's’ website or in various kinds of casting software and directories

Double

Software: Primitive data type representing a floating-point number
Showbiz: A performer who appears in place of another performer, i.e., as in a stunt.

Mirror

Software: A copy of a system that updates from the original in near to real time, often a database or file storage system
Showbiz: An optical device that helps a performer check they’ve applied their makeup correctly

Principal

Software: Used in database mirroring to refer to the primary instance of the database
Showbiz: A performer with lines.

Production

Software: The live infrastructure and code environment
Showbiz: A film, TV or stage show, such as a professional actor might list on their acting CV.

REST

Software: Representational State Transfer - an architectural style used when building hypermedia APIs
Showbiz: What actors do between jobs.

Script

Software: A computer program written in a scripting language
Showbiz: The written dialogue and directions for a play, film or show

“Sequel”

Software: Standard pronunciation of SQL, referring to either the database query language. Also commonly refers to Microsoft’s SQL Server database product.
Showbiz: A published, broadcast, or recorded work that continues the story or develops the theme of an earlier one.

Server

Software: The opposite of a client
Showbiz: Someone working as waiting staff in a restaurant. Who is quite possibly an actor moonlighting as a server to pay the bills between acting jobs.

Spotlight

Software: The native macOS search application
Showbiz: Our company – www.spotlight.com, “The Home of Casting” – and the directories and services we have created since 1927.

Staging

Software: A replica of a production hosting environment used to test new features and deployments.
Showbiz: The method of presenting a play or dramatic performance; also used to refer to the stage structure itself in theatre and live performance.

Semantic Versioning with Powershell, TeamCity and GitHub

2017-01-26T11:46:00+00:00

Here at Spotlight Towers, we’ve been using TeamCity as our main build server since version 6; it’s a fantastic tool and we love it dearly. It got even better a few years back when we paired it with the marvellous Octopus Deploy; TeamCity builds the code and creates a set of deployable packages known as Octopacks; Octopus deploys the packages, and everything works quite nicely. Well, almost everything. One of the few problems that TeamCity + Octopus doesn’t magically solve for us is versioning. In this post, we’re going to look at how we use Git and TeamCity to manage versioning for our individual packages.

If this sounds like your sort of thing, why not come and work for us? That’s right – Spotlight is hiring! We're looking for developers, testers and a new UX/Web designer - check out jobs.spotlight.com and get in touch if you’re interested.

First, let’s establish some principles

We are going to respect the semantic versioning convention of MAJOR.MINOR.PATCH, as described at semver.org.
Major and minor versions will be incremented manually. We trust developers to know whether their latest commit should be a new major or minor release according to semantic versioning principles.
Building the same codebase from the same branch twice should produce the same semantic version number.
Packages created from the master branch are release packages.
Packages created from a merge head of an open pull request are pre-release packages.
Pre-release packages will use the version number that would be assigned if that branch was accepted for release at build time.

Now, here’s the part where we’re going to deviate from the semantic versioning specification, because our packages actually use a four-part version number. We want to include a build number in our package versions, but the official semver extension for doing this – MAJOR.MINOR.PATCH+BUILD - won’t work with NuGet, so we’re going to use a four-part version number MAJOR.MINOR.PATCH.BUILD. Pre-release packages will be appended with a suffix describing which branch they were built from – MAJOR.MINOR.PATCH.BUILD-BRANCH.

OK, here’s an illustrated example that demonstrates what we’re trying to achieve. Master branch is green. Two developers are working on feature branches – blue and red in this example. To create our pre-release builds, we’re using a little-documented but incredibly useful feature of GitHub known as ‘merge heads’. The idea is that if you have an open pull request, the merge head will give you a snapshot of the codebase that would be created by merging the open pull request into master – so you’re not just testing your new feature in isolation, you’re actually building and testing your new feature plus the current state of the master branch. There is one caveat to this, which I’ll explain below.

So, we’ve got TeamCity set up to build and publish packages every time there’s a commit to master or to the merge head of an open pull request, and we’re also occasionally triggering manual builds just to make sure everything’s hanging together properly. Here’s what happens:

That line there that’s highlighted in yellow is a gotcha. At this point in our workflow, we’ve merged PR1 into our master branch, but because we haven’t pushed anything to the blue branch since this happened, the blue merge head is out of date. PR2 does NOT reflect the latest changes to master, and if we trigger a build manually, we’ll end up with a package that doesn’t actually reflect the latest state of the codebase. The workaround is pretty simple; if you’re creating pre-release builds from merge heads, never run these builds manually; make sure you always trigger the build by pushing a change to the branch.

Now let’s look at how can we get TeamCity to automatically calculate those semantic version numbers whenever a build is triggered. We’ll start with the major and minor version. We’re going to track these by creating a version.txt file in the root of the project codebase, which just contains the major and minor version numbers. If a developer decides that their feature branch represents a new major or minor version, it’s their responsibility to edit version.txt as part of implementing the feature. This also means that prerelease packages built from that branch will reflect the new version number whilst master branches will continue to use the old version until the branch gets merged, which I think is rather elegant.

For the patch version, we’re going to assume that every commit or merge to the master branch represents a new patch version, according to the following algorithm

If the current version.txt represents a NEW major/minor version, the patch number is zero
Otherwise, the patch number is the patch number of the latest release, incremented by the number of commits to the master branch since that release.

So – how do we know how many commits there have been since the last release? First, each time we build a release branch, we’re going to use Git tags to tag the repository with the version number we’ve just built. TeamCity will do this for you automatically using a build feature called “VCS labeling”:

Assuming every release has a corresponding tag, now we need to find the most recent release number, which we can do from the Git command line.

git fetch –tags
git tag –sort=v:refname

Git tags aren’t retrieved by default, so we need to explicitly fetch them before listing them. Then we list all the tags, specifying sort=v:refname which causes tag names to be treated as semantic versions when sorting. (Remember that semver sorting isn’t alphanumeric – in alphanumerics, v9 is higher than v12). Once we’ve got the latest tag, we need to count the number of revisions since that tag was created, which we can do using this syntax:

git rev-list v1.2.3..HEAD –count

To use this in our TeamCity build, we'll need to output the various different formats of that version so that TeamCity can use them. We want to do three things here:

Label the VCS root with the three-part semantic version number v1.2.3
Update the AssemblyInfo.cs files with the four-part version number 1.2.3.456 – note that we can’t put any prerelease suffix in the AssemblyInfo version.
Pass the full version – 1.2.3.456-pr789 – to Octopack when creating our deployable packages with Octopus.

I've wrapped the whole thing up in a Powershell script which runs as part of the TeamCity build process, which is on GitHub:

To use it in your project, add versions.ps1 to the root of your project repo; create a text file called version.txt which contains your major.minor version, and then add a TeamCity build step at the beginning of your build process that looks like this:

Finally, it’s worth mentioning that to use command-line git from Powershell, I had to set up TeamCity to use an SSH VCS root rather than HTTP, and install the appropriate SSH keys on the TeamCity build agent. I don't know whether this is a genuine requirement or a quirk of our configuration; your mileage may vary. And I still find Powershell infuriatingly idiosyncratic, but hey - you probably knew that already. :)

Happy versioning! And like I said, if this sort of thing sounds like something you’d like to work on, awesome - we’re hiring! Check out jobs.spotlight.com for more details and get in touch if you’re interested.

Sharing a Dropbox folder using Bootcamp and exFAT

2017-01-09T20:30:00+00:00

My main laptop for the last few years has been a 15” Retina MacBook Pro. As I’m primarily a .NET developer, I use Bootcamp to run Windows 10, Visual Studio and SQL Server, but I also boot into macOS almost daily for things like audio recording and video editing using Logic Pro and Adobe Premiere. I’m also a huge fan of Dropbox – I use Evernote for text documents (notes, talk ideas and lyrics), and Dropbox for just about everything else.

My MacBook Pro has a 500Gb SSD, which sounds like a lot, but once you’ve got two operating systems it’s surprising how quickly you start running out of space – and one of my biggest frustrations was having two separate Dropbox folders. I have about 40Gb of stuff in Dropbox, which means a 40Gb Dropbox folder on my NTFS Windows partition, and another 4Gb Dropbox folder on the ExtFS macOS partition. And they store exactly the same files – so much so that Dropbox actually became my default sync mechanism for getting files from Windows into macOS and vice versa.

Being a huge nerd, I thought it would be fun to kick off 2017 by completely repaving my laptop, and I wondered if there was some way to store my Dropbox folder, and other big things like my music library, on a shared partition that both operating systems could use. It took a bit of tweaking but I think I’ve managed to get it working – here’s how. The secret sauce that made it all work is a relatively new file system called exFAT. The killer feature of exFAT here is that both Windows 10 and macOS can read and write it natively, but it doesn’t have any of the file size or volume size restrictions of older interoperable formats like FAT32.

Caveat: this is all completely unsupported. If you try any of this and you lose all your files, I’m not going to help you – and neither are Apple, Microsoft or Dropbox. If you’re happy using batch scripts to toggle file attributes and manually partitioning your OS drive, then this lot might help – but messing around with disk partitions and unsupported filesystems is a good way to end up with an unbootable laptop and a lot of bad mojo. So before we go any further, back up everything. After all, Dropbox is a file sync tool – if some bizarre combination of macOS and exFAT results in it removing a load of files, the last thing I want is for Dropbox to conveniently remove the same files from all my other devices. I know it’s easy enough to revert to an older version of anything that’s in Dropbox, but I figured having a copy of it all on an external HD probably couldn’t hurt.

Next, work out how you want to partition your disk. In my case, I wanted to go for a ~150Gb partition for each operating system, and a 200Gb shared partition that both of them could see. Your mileage may vary, and – as I did – you might find you can’t get exactly what you wanted, but I managed to get pretty close. Now do a completely clean reinstall of macOS. Give it the entire drive and let it do its thing. Once you’re done, use the Boot Camp Assistant to set up and install Windows. When Boot Camp Assistant asks how you’d like to partition your drive, make the Windows partition equivalent to the total size of your desired Windows partition plus your shared partition – so in this case, I temporarily had a 150Gb macOS partition and a 350Gb Windows partition.

Once you’ve partitioned the disk, install Windows as usual. You should end up with a regular Boot Camp system, that you can boot into macOS or Windows by holding down the alt key during system startup or by using the Boot Camp options in Windows. Now here’s the fun part. Once you’ve installed Windows, but before you install any other apps or files, you’re going to use the Windows disk utility to shrink that Windows partition down to 150Gb and then create a shared exFAT partition in the remaining space. Right-click the BOOTCAMP (C:) partition, click Shrink Volume...

There’s some limit of the NTFS filesystem that means “you cannot shrink a volume beyond the point where any unmovable files are located”. Now I don’t know how much variance there is between Windows installations, but I couldn’t shrink my C: drive any smaller than 163Gb, despite the fact that the fresh Windows install only took up about 20Gb. For what I needed, this was close enough – if not, I’d have been looking for something that would let me defrag and reorganize the NTFS partition. Or reading up on how an SSD gets fragmented in the first place – I thought they were just big boxes of non-volatile solid-state memory?

Anyway. Next step was to format the new partition as exFAT, called it Shared, and then reboot into macOS and make sure it was visible. I poked around a bit, created some text documents and other files, rebooted back into Windows… After an hour or so this all seemed to be working quite nicely. Next step was to see how Dropbox coped with it. Rather than throw it at my 40Gb Dropbox account, I set up a new Dropbox account, installed Dropbox on both macOS and Windows and signed into to both of them using the new account.

This is where things got a bit weird. Here’s my best hypothesis as to what’s happening.

macOS is storing some additional metadata for every file. On a regular ExtFS filesystem, the metadata is stored somewhere in the filesystem itself. On an exFAT partition, it’s stored in a tiny hidden file (apparently known as an AppleDouble) next to the original. Alongside foo.txt there’s now a 4k file called ._foo.txt. I don’t think this is a Dropbox thing, it just appears to be caused by running macOS on a non-ExtFS drive.
The Dropbox agent on macOS sees these extra files, thinks that they’re important, and uploads them to Dropbox.
The dot-underscore ._ files are then synced across all your other Dropbox devices – I can see them in the web interface as well as on my Windows partition.

If I wasn’t using Dropbox, the ._ files would still be there, but macOS marks them as hidden and Windows appears to respect this – doing a dir /a:h in one of my exFAT folders will show them but by default they don’t appear in Windows Explorer. I figured this wasn’t a big deal – I could live with a bunch of hidden 4K files – so I removed the temporary Dropbox account and installed the real one. This may have been a bit overly optimistic. Dropbox happily synced 40Gb of files from Windows, but when I rebooted into macOS, it created the ._ files for all of them… and then when I rebooted into Windows again, it started trying to sync all the hidden ._ files. After twelve hours of this, I concluded that something wasn’t working quite as I’d hoped, so I modified the plan slightly.

I realized that I don’t actually need the Dropbox agent running on both OSes – just having the shared Dropbox folder is enough – so I uninstalled the Dropbox agent from macOS, and things suddenly got a lot more straightforward. The dot-underscore files are still there. On the exFAT shared partition they’re hidden by default, but Dropbox helpfully syncs all the ._ files into all my other Windows systems and so I’ve created a batch script in my Dropbox folder that finds any of these and sets them to hidden so they don’t show up in Windows Explorer. It’s a one-liner:

for /r %1 in (._*) do attrib +h "%1"

The last thing I wanted to try before doing it for real was to see how VMWare Fusion coped with the shared partition. As well as dual-booting between macOS and Windows, I’ll also occasionally use VMWare to run the Windows partition as a VM inside macOS, which works pretty well. Here’s what I discovered:

As soon as you boot the Windows VM, both the BOOTCAMP partion and the Shared partition disappear from macOS.
The BOOTCAMP partition is mounted in VMWare as drive C: (obviously), and the Shared partition appears in the VM as drive D:
The Dropbox agent will immediately pick up and sync any files created from macOS
When I shut down the VM, the BOOTCAMP and Shared partitions both show up in macOS again and everything’s back to normal.

So, here’s the final setup I ended up with. Windows 10 is running the Dropbox agent, and syncing files with my Dropbox cloud storage as intended. The Dropbox folder is on the shared exFAT partition, so macOS can read and write my local Dropbox folder, but macOS doesn’t sync anything. If I add files to Dropbox when running macOS, when I reboot into Windows again, the sync agent will pick up these new files and sync them to the cloud. Finally, if I’m in macOS and find I need a file that’s been added to Dropbox from another device and hasn’t synced yet, I can either reboot into Windows and sync it, fire up the VMWare machine for a few moments, or just download it via the web interface.

I've been running it for a couple of weeks, including a week of wrangling PowerPoint slides and shared files at NDC London, and it's working really quite nicely. And it's really rather nice to have nearly 200Gb of free disk space after installing both operating systems, all my apps and utilities, and 40Gb worth of Dropbox files.

My Life with the Microsoft Natural Keyboard

2017-01-03T21:53:00+00:00

A moment ago I was catching up on Twitter and I saw this:

I'm sitting here trying NOT to buy this keyboard. It looks amazing. https://t.co/Es70Ed7tU1
— Scott Hanselman (@shanselman) January 3, 2017

Now Scott has written some great posts about keyboards – and mice, and workstation ergonomics in general – over the years, so I immediately clicked the link to see what was so exciting… and wow. This is the new Microsoft Surface Ergonomic Keyboard, and I, too, am now sitting here trying not to buy it. Actually, I’m waiting for a UK layout, and then I suspect that trying not to buy it will rapidly become unmanageable. I may even fail to not buy two of them so I have one at work and one at home.

Now, there’s a couple of things about this keyboard that are interesting only because Microsoft have consistently got them right, and then messed them up, and then got them right again, and then messed them up again - over many, many years. Like an even-numbered Star Trek movie or an odd-numbered version of CorelDraw, this latest one is a good one – and as somebody who’s used every incarnation of the Microsoft Natural keyboard, this seems like a nice opportunity to take a little wander down memory lane.
OK, cue the Wayne’s World dream-sequence time-travel effects… and here’s a keyboard. A 102-key IBM PS/2 keyboard, which was the standard layout for PC keyboards for about a decade, back in the days when keyboards were beige and the cool kids knew all sorts of tricks that would let you load HIMEM.SYS and your mouse driver and still have enough main memory left to run Wing Commander 2.
(IBM PS/2 keyboard by Raymangold22 via Wikipedia | CC0 | Link)
Around the time of Windows 95, Microsoft proposed the subtle addition of a two new keys – a Windows key (actually two of them) in those handy gaps next to the Ctrl keys, and a key that I’ve just this second learned is called the menu key, which is basically a right-click key. Other than shortening the space bar slightly, these new keys didn’t really move things around much, which was great, because that hard-earned muscle memory that let you hit triple-key combinations without taking your eyes off the screen worked just fine.
Oh, and they also started making them in colours that weren’t beige, which was a big improvement if you actually had to share your living space with them.

It’s around this time that I left school and got my first IT job, which meant I was typing for a good chunk of every day, and within a year I started getting unpleasant pain in my wrists and forearms. During a chance conversation at work, someone mentioned that a former employee there had had the same problem and switched to using an ergonomic keyboard – which was still knocking around in a cupboard somewhere. I tried it out and was instantly smitten.
This was the original Microsoft Natural keyboard, released in 1994.

Photo by DeanW77 via Wikipedia – own work, CC BY-SA 4.0, Link

I absolutely loved it. I literally wore it out – I used it until some of the keys no longer worked, and then went shopping for a replacement… and discovered you couldn’t get the original Natural keyboard anymore; it had been replaced by the Natural Keyboard Elite, released in 1998.

https://www.engadget.com/products/microsoft/natural-keyboard/elite/

Looks close enough, right? Except if you look closely, you’ll see that instead of the familiar inverted-T cursor shape and two rows of navigation keys, the cursor keys are in a sort of weird diamond formation and the navigation keys are in two columns.

This was horrible. All those years of muscle memory suddenly gone – every time you’d try to hit Ctrl-PgUp or Shift-End you’d get it wrong. And when you’re in the zone, that’s a horrible, jarring experience – every time it happens it interrupts your flow, wrenches you back to reality and makes you want to throw the damn thing across the room. Imagine driving a car where the brake pedal is above the accelerator instead of alongside it – it was a truly unpleasant experience.

Fortunately, it didn’t take long for them to realise this one was a bit of a mistake. A year later in 1999, they came out with the Natural Keyboard Pro, and it was fantastic.

By ----PCStuff via Wikipedia, CC BY-SA 2.5, Link

All the keys were in the right places, and it included a two-port USB hub which was great for plugging in your mouse. It added a bunch of “multimedia” buttons (which I never really used) and a dedicated button for launching the Windows Calculator (which actually proved to be surprisingly useful.) I loved this keyboard dearly. You can guess what happened next… yep, I used it until it wore out, went shopping for a new one, and… yeah. No more Natural Keyboard Pro. Instead, we had this delightful triumph of form over function:

photo via Engadget

The Microsoft Natural Multimedia Keyboard, released in 2004. OK, it’s got the classic inverted-T cursor keys (yay, but - look at that! Not only are the navigation keys all wrong, but the Insert key isn’t even there; we’ve got a massive double-height Delete key instead. If memory serves, Insert was relegated to some sort of funky combo involving the PrtSc key. Oh, and this was also the keyboard where you had to keep F-Lock switched on all the time because the function keys were actually some sort of weird keyboard shortcuts that nobody ever used ever. You know. Like a dedicated key for “Send”, because Ctrl-Enter is just too complicated.

Again, it didn’t take long for them to come out with something better… and boy, did they ever get it right with the next one. The Microsoft Natural Ergonomic Keyboard 4000.

photo via microsoft.com

This was my main keyboard for most of the last decade. I loved this keyboard so much I actually stockpiled it – one at work, one at home, and a few spares standing by in case they wore out. And wear out they did, one by one, until late last year I decided it was time to replenish the stockpile… which is when my long-suffering colleagues half-jokingly asked if, maybe, I’d be prepared to try something quieter. See, the 4000 is lovely, but it’s noisy. The keys have plenty of travel, with a nice satisfying thump at the bottom of each keystroke, and the keyboard’s casing – which is big – makes a rather effective sounding-box. Until the bottom falls out of the London real estate market and we can justify private offices for our developers, FogCreek style, the open plan office is an unfortunate fact of life for most of us… and so, in the spirit of workplace harmony, I ordered a Microsoft Sculpt keyboard.

photo via Microsoft

And you know what? It’s pretty close to perfect. It’s comfortable. It’s quiet. It takes up about 60% of the desk space that the old Ergonomic 4000 did. The numeric keypad is actually separate, which I’m completely undecided about… when you actually want to type on it, it’s annoying not having it fixed in place, but being able to pick it up and use it like an old-school calculator is actually surprisingly useful. Except – yep – the navigation key layout is all screwed up. Again. I hit Insert by mistake all the time on this thing, and frequently land on the left cursor when I’m reaching for Ctrl.

But with the release of the Surface keyboard at the top of the post, it looks like, yet again, there’s a truly great keyboard hot on the heels of the not-so-great one. I’m just really curious as to why they keep bringing out models that use non-standard keyboard layouts.

photo via Microsoft

When it hits the market here in the UK, I’ll pick one up and let you know how I get on. Now, if they’d only release one that was completely black – with no key markings – then we’d really be onto something. I wonder if you can spray-paint it...

IdentityServer, OpenID Connect and Microsoft CRM Portals

2016-11-14T18:17:00+00:00

As readers of this blog will know, here at Spotlight we’re in the process of moving nine decades’ worth of legacy business process onto Microsoft Dynamics CRM, aka CRM Online, which I gather is now called Dynamics 365 (because hey, it’s not like naming things was hard enough already, right?)

We’re also investigating a couple of options for building customer-facing systems that integrate with Dynamics. Until last year, there were really three options for this – a product called Adxstudio, a free Microsoft component called the CRM Portal Accelerator, or rolling your own solution using the CRM SDK. Around this time last year, Microsoft quietly retired the Portal Accelerator component and acquired Adxstudio, and since then, they’ve been in the process of assimilating it into the Dynamics product family – which has meant it’s been something of a moving target, both in terms of the supported features and in terms of the quality of documentation and examples.

I’ve previously blogged about one way to integrate Adxstudio with your existing authentication system, but that approach relied completely on running Adxstudio on-premise so you could run your own code as part of the request lifecycle – and as you may have noticed, there’s a bit of a trend in IT at the moment away from running your own servers and towards using hosted managed services, so that patching and backups are somebody else’s problem. Since Microsoft acquired Adxstudio, there’s been a lot of churn around what’s supported and what’s not – I’m guessing that behind the scenes they’re going through the Adxstudio codebase feature-by-feature and making sure it lines up with their plans for the Dynamics 365 platform, but that’s just guesswork on my part.

One of the main integration points I’ve been waiting for is the ability for a Microsoft-hosted Portal solution to use a third-party OpenID Connect endpoint to authenticate users, and it appears in the latest update this is finally supported – albeit with a couple of bumps along the way. Here’s what I’ve had to do to get a proof-of-concept up and running.

Setting up Dynamics CRM Portals

First, you’ll need to set up a Dynamics Portal trial. You can get a 30-day hosted trial of Dynamics CRM Online by signing up here – this actually gives you a full Office 365 organization including things like hosted Active Directory, as well as the Dynamics CRM Online instance we’re using in this example. Next, you’ll need to ask nicely for a trial of the portal add-on – which you can do by filling out the form at crmmanagedtrials.dynamics.com.

Setting up IdentityServer and configuring an ngrok tunnel

Whilst you’re waiting for the nice Microsoft people to send you your trial license, get up and running with IdentityServer. For this prototype, I’m using the MVC Authentication example from the IdentityServer3.Samples project – clone it to your workstation, open the MVC Authentication solution, hit F5, verify you can get up and running on localhost.

Next – in order for Dynamics CRM Online to talk to your IdentityServer instance, you’ll need to make your IdentityServer endpoints visible to the internet. You could do this by deploying your IdentityServer sample to Azure or AWS, but for experiments like this, I like to use a tool called ngrok, which will create temporary, secure tunnels from the internet to your workstation. Download ngrok, unzip it somewhere sensible.

Pick a tunnel name. I’m using authdemo in this example but any valid DNS host name will do. Next, create a local IIS application pointing to the EmbeddedMvc folder in your samples directory, and set the host name to <your tunnel name>.ngrok.io

Now run ngrok.exe to create a tunnel from the internet to your new IIS application:

C:\tools\ngrok> ngrok.exe http –subdomain=authdemo 80

ngrok by @inconshreveable

Session Status        online
Version               2.1.18
Region                United States (us)
Web Interface         http://127.0.0.1:4040
Forwarding            http://authdemo.ngrok.io -> localhost:80
Forwarding            https://authdemo.ngrok.io -> localhost:80

Connections ttl opn rt1 rt5 p50 p90
0 0 0.00 0.00 0.00 0.00

All being

If that’s worked, you should be able to fire up a browser, go to http://authdemo.ngrok.io/ – replacing ‘authdemo’ with your own tunnel name - and see the IdentityServer3 sample landing page:

Configuring IdentityServer

Right. Next thing we need to do is to make a couple of changes to the IdentityServer configuration, so that it’ll run happily on authdemo.ngrok.io instead of on localhost

First, enable logging. Just do it. Use the package manager console to install the Serilog.Sinks.Trace package. Then add this to the top of your Configuration() method inside Startup:

Log.Logger = new LoggerConfiguration()
                .MinimumLevel.Debug()
                .WriteTo.Trace()
                .CreateLogger();

and add this to your web.config, specifying a path that’s writable by the application pool:

<system.diagnostics>
  <trace autoflush="true"
         indentsize="4">
    <listeners>
      <add name="myListener"
           type="System.Diagnostics.TextWriterTraceListener"
           initializeData="C:\logfiles\identityserver.log" />
      <remove name="Default" />
    </listeners>
  </trace>
</system.diagnostics>

Next, do a global search and replace, replacing any occurrence of localhost:44319 with authdemo.ngrok.io – again, substituting your own tunnel name as required.

Next, add a new client to the static EmbeddedMvc.IdentityServer.Clients class the IdentityServer sample project – changing the highlighted values to your own client ID, client secret, and portal instance URL:


new Client {
    ClientName = "Dynamics CRM Online",
    ClientId = "crm",
    Flow = Flows.Hybrid,
    ClientSecrets = new List() { new Secret("secret01".Sha256()) },
    RedirectUris = new List { 
      "https://my-portal-instance.microsoftcrmportals.com" 
    },
    PostLogoutRedirectUris = new List { 
      "https://my-portal-instance.microsoftcrmportals.com" 
    },
    AllowedScopes = new List { "openid" }
},

Adding IdentityServer as an endpoint in CRM Portals

Finally, you need to add your new IdentityServer as an identity provider. CRM Portals uses the Dynamics CRM platform for all its configuration and data storage, so to add new settings you’ll need to log into your Dynamics CRM Online instance, go into Portals > Site Settings, and add the following values:

Name	Value	Website
Authentication/OpenIdConnect/AuthDemo/Authority	http://authdemo.ngrok.io/identity/	Customer Self-Service
Authentication/OpenIdConnect/AuthDemo/Caption	IdentityServer OpenID Connect Demo	Customer Self-Service
Authentication/OpenIdConnect/AuthDemo/ClientId	crm	Customer Self-Service
Authentication/OpenIdConnect/AuthDemo/ClientSecret	secret01	Customer Self-Service
Authentication/OpenIdConnect/AuthDemo/MetadataAddress	http://authdemo.ngrok.io/identity/.well-known/openid-configuration	Customer Self-Service

Finally, it looks like you’ll need to restart the portal instance to get it to pick up the updated values – which you can do by logging into the Office 365 Admin Center, Admin Centers, CRM, Applications, Portal Add-On, clicking ‘MANAGE’, and pressing the nice big RESTART button on the Portal Actions page:

And – assuming everything lines up exactly right – you should now see an additional login button on your CRM Portals instance:

Clicking on it will bounce you across to your ngrok-tunnelled IdentityServer MVC app running on localhost:

…and when you hit ‘Yes, Allow’, you’ll be redirected back to the CRM Portals instance, which will create a new CRM Contact linked to your OpenID Connect identity, and log you in to the portal.

Conclusions

Of course, in the real world there’s a lot more to it than this – there is a huge difference between a proof of concept like this and a production system. These sorts of user journeys form such a key part of delivering great user experience, and integrating multiple systems into your login and authentication/authorization journeys only makes this harder. But it did work, and it wasn’t actually all that complicated to get it up and running. It’s also interesting to see how something like OpenID Connect can be used to integrate a powerful open-source solution like IdentityServer with a heavyweight hosted platform service like CRM Portals.

Whether we end up adopting a hosted solution like CRM Portals – as opposed to just building our own apps that connect to CRM via the SDK or the new OData API – remains to be seen, but it’s nice to see solutions from two radically different sources playing nicely together thanks to the joy of open protocols like OpenID Connect. Long may it continue.

The Laws of Distributed Systems

2016-10-31T17:10:00+00:00

I’ve spent a lot of time over the last year reading, thinking, and speaking at conferences about distributed systems, organisational structures, and the eponymous laws of software development. Over the course of many conversations and countless blog posts and articles, something has crystallised from thinking about three laws in particular, which – if it’s right - could have substantial implications for all of us as software developers, and for the people who use the systems we build.

TL;DR: if we keep having meetings, the internet will stop working.

(There – that got your attention, didn’t it?)

Moore’s Law

So, let’s recap. Gordon Moore was the co-founder of Intel and Fairchild Semiconductor. Back in 1965, Moore wrote a paper predicting that the number of components per integrated circuit would double every year. In 1975, he revised his forecast to doubling every two years. His predictions have proved accurate for several decades, and will probably continue to do so until the 2020s, but what’s really interesting is what’s happened since 2000.

Here’s the average total transistor count of CPUs against their year of introduction since 1965. Plotted on a a logarithmic axis, it’s pretty close to a straight line.

[Data source: Wikipedia]

Now here’s the same graph, but showing transistors per core.

[Data source: Wikipedia]

See how around 2005 the two series suddenly diverge sharply? That’s because in the early 2000s, we began hitting the physical limits of how many transistors could be integrated into a single CPU. Somewhere around the 4Ghz mark, we hit a wall in terms of raw clock speed, and so the semiconductor industry hit upon the bright idea of multicore CPUs – basically putting more than one CPU into the same physical package.

In the same time frame, we’ve seen an industry-wide shift away from monolithic powerhouse servers towards distributed systems. Modern web apps – which are really just big multiuser systems – run across clusters of dozens or hundreds of ephemeral worker nodes; a radical contrast to the timesharing mainframe systems of the 1970s and 1980s.

The amount of computing power available at a particular price point is still increasing exponentially, but we’re no longer scaling up, we’re scaling out. And the reason we’re scaling is that the load on our systems – our websites, APIs and servers – is also increasing. More people are getting online, people are using more devices and connected services, and those devices are delivering increasingly rich user experiences – which means more data, more power and more bandwidth. Here’s Business Insider’s analysis and forecast of the number of connected devices from 2015 to 2020:

To cope with this ever-increasing level of expectation, we need to build systems that will scale out to cope with demand. Our code needs to parallelize. We need to decompose our problems into small, autonomous units of work that can be distributed across as many cores or nodes as we have available, and combine the outputs of those operations to deliver the results our users are expecting.

Amdahl’s Law

This brings us to Amdahl’s Law. Gene Amdahl started out designing mainframe systems for IBM. He was the chief architect of the IBM System/360, and he first presented his eponymous law back in 1967. Amdahl’s Law controls the theoretical performance improvements we can expect by parallelizing some given workload.

Amdahl’s Law is actually S_latency(s) = 1/((1-p) + (p/s)) – but the gist of it is nicely explained by thinking about Christmas dinner. Or Thanksgiving, if that’s your thing. If you’ve got one person with one cooker working alone, it’ll take a good 20 hours to prepare all the trimmings for a Christmas dinner. By adding more people and more cookers, you can parallelise this and so complete it faster – but you reach a point where everything is done and everyone’s stood around waiting for Jeff to finish roasting the turkey, because you can’t roast a turkey in under four hours, no matter how many chefs and ovens you’ve got.

If you need to parallelize like this, eliminate the turkey. Have steaks instead – because you can cook steaks in parallel. If you suddenly get another 20 guests showing up half an hour before lunch, no problem – you don’t need to wait four more hours to roast another turkey; just get 20 more chefs to cook 20 more steaks and you’ll still be done on time. And because you’re using cloud infrastructure, you can spin up more chefs and griddles instantly to cope with the increased demand.

See, by designing systems to eliminate those non-parallelisable workloads, we create systems that scale smoothly with the available resources. The beauty of that is that, like all the best solutions in software, it turns “how fast is our website” into a pure business decision. You want faster pages? Pay for more servers. No need to rewrite your algorithms; just throw more power at it.

Conway’s Law

Finally, there’s Conway’s Law. First published in 1964, Conway’s Law is the observation that ‘any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.’ As with Moore’s Law, it’s an observation that has proved remarkably prescient over the intervening decades. I’ve spoken at length about how Conway’s Law has affected the teams and projects I’ve worked on personally, but there’s also some interesting examples in the software industry at large. The relationship between the Linux kernel and the various distributions based on it has interesting parallels with Linus Torvalds’ role in the Linux ecosystem. Id Software’s genre-defining Doom and Quake games – tight, cohesive, focused engines, created by a bunch of coders camped out in a beach house with soda, pizza and no distractions, with less tightly-coupled elements like music and level design handled by less close-knit development efforts. High-profile open source projects like Chromium – the underlying rendering engine, the browser itself, and the ecosystem of plugins and extensions closely reflecting the tight-knit Webkit project, the loosely-organised contributors and pull requests that shape the development of the browser application, and the community of plugin and extension developers who don’t engage with the project directly but rely on published contracts and protocols just as their plugins and extensions do.

And then there’s the really obvious examples, like this one:

Putting it all together…

OK, so let’s look at what happens when we interpret those three laws together. Moore’s Law has informed half a century of user expectations about technology. More people do more stuff on more devices, and they expect those experiences to keep on getting better, faster and more responsive. As we’ve seen above, the increase in raw computing power that’s going to deliver those improvements isn’t about clock speed any more, it’s about parallelism. Amdahl’s Law tells us whether systems will benefit from that parallelism or not – and that systems based around blocking, non-parallelisable, long-running operations will benefit the least from the next decade of computing innovation. And Conway’s Law says that if we don’t want our systems to contain these kinds of blocking, non-parallelisable operations, then we should be looking to eliminate them from our organisations.

Which brings us to the crux of the thing: what’s the organizational equivalent of a long-running non-parallelizable operation?

How about sitting around reading Hacker News because the person who’s asked you to build a “Summary Dashboard” hasn’t told you where to find the data, or what the dashboard should look like, and they’re out of the office right now, they didn’t leave any notes, and you can’t do anything until they get back?

How about a two-hour project update meeting where a series of people sit around telling each other things they could have emailed, or written down in a ticket or on a wiki page?

How about sitting on a train for an hour to get to the office in Canary Wharf where you’re expected to be at 09:00 every day, despite the fact that your source code is hosted in the US, your data centre is in Ireland, your issue tracking system is hosted in Frankfurt and your customers are online 24/7 all over the world?

One of the underlying principles of the agile manifesto is that ‘the most efficient and effective method of conveying information to and within a development team is face-to-face conversation’. I think that’s correct, but I think it might be optimising for the wrong metric. Sure, a conversation is a high-bandwidth, high-interaction discussion medium, and I find face-to-face great for bouncing ideas around and solving problems - but conversations are ephemeral. They’re not captured anywhere, nobody outside the conversation knows what was said, and there’s always the risk the people you’re talking to assure you they get it when they actually haven’t understood a single word you said. Perhaps we should be optimising our communication patterns for discoverability instead of raw bandwidth; trading a little temporary velocity for some long-term efficiency.

This isn’t just about cancelling a couple of meetings and letting people work from home on Fridays. It’s about changing the way we think about collaboration, so that the interaction patterns we want to see in our systems can emerge organically from the interaction patterns used by the people who created them. It’s about taking established architectural patterns and practises used in asynchronous distributed systems, and working out if we can apply those patterns to our teams and our projects. What if you applied event sourcing to your project backlogs, so you don’t keep having to ask people about the context behind a particular decision? Maybe you’re even doing this already – I know a lot of open-source projects that do an excellent job of capturing this history as part of their open issues and tasks so anybody who wants to pick up a particular ticket can see the complete history, the discussion, the arguments and hopefully the eventual consensus. What if you treat your documentation - wiki pages, GitHub pages, READMEs - like the query stores in a CQRS system? Rapid retrieval, read-only, optimised for consumption, and updated as necessary when processing commands (i.e. making changes) that affect the underlying systems that are being documented?

What I find remarkable is that Moore, Amdahl and Conway all published their eponymous laws almost exactly fifty years ago – Conway’s Law was published in 1964, Moore was 1965, Amdahl was 1967. Their observations hail from a decade of astonishing engineering achievements – Apollo, the Boeing 747, the Lockheed SR-71, the geodesic dome – in an era when computers were still highly specialist devices. Sure, you could argue that people working on timesharing systems in the 1960s couldn’t possibly have foreseen the long-term social implications of distributed systems engineering – but remember, this is the generation that landed on the moon using slide rules and No. 2 pencils. Do you really want to bet on them being wrong?

The Mystery of the Chinese Junk

2016-10-25T21:33:00+00:00

You know it’s going to be one of those days when, just as you’re about to put on your headphones and get into ‘the zone’, you overhear somebody saying the fateful words ‘ok, then maybe we’ll need to get Dylan to look at it.’

See, amongst the many hats I wear in the course of a given week, there’s one that’s probably labelled ‘dungeon master’ I’m the one who remembers where all the bodies are buried, because – for all sorts of reasons that made very good sense at the time – I probably helped bury most of them. And on this particular day, the source of so much excitement was our venerable Microsoft Dynamics CRM v4 server. It started out with a sort of general grumbling on the support channel about CRM4 being slow… but by the time it was handed over to me to look into, it was beautifully summarised as ‘dude… there’s Chinese in the Windows event log’

And, sure enough, there is – complete with the lovely Courier typeface that Windows Event Viewer kicks into when you get errors so weird that good old Microsoft Sans Serif can’t even display them:

Now, whilst it’s been a while since I’ve done any serious work on our old CRM system, I’m pretty sure it’s not supposed to do that – so we start investigating. Working theory #1: some sort of vulnerability has resulted in attackers injecting Chinese characters into our database – whilst CRM4 is generally pretty well insulated from any public-facing code, there’s one or two places where signup forms would generate CRM Leads, that sort of thing. So we start grepping the entire database for one of the Chinese strings we’ve found in the event log.

Whilst this is going on – and trust me, it takes a while – I decide to share my excitement via the wonder of social media. This turns out to be a Really Good idea, because... well, here's what happened...

"The incoming tabular data stream TDS RPC protocol stream is incorrect. Parameter ("䐀攀氀攀琀椀漀渀匀琀愀琀攀..." Oh. It's gonna be one of THOSE days.

— Dylan Beattie (@dylanbeattie) October 18, 2016

@dwm @dylanbeattie The low bits are all null, so this probably UTF-16LE being mistaken for UTF-16BE (or vice versa...?). pic.twitter.com/D0IO4px9YE

— Fake Unicode ⁰ ⁧ (@FakeUnicode) October 18, 2016

You see in @FakeUnicode's screenshot there, the words ‘DeletionState’ appear quite clearly at the bottom of the message?

Whilst this is going on, our database search comes back reporting that there’s no mysterious Chinese characters in any of our CRM database tables. Which is good, since it means we probably haven’t been compromised. So, next step is to work through that Unicode lead, see if that gets us anywhere. Because .NET has a built-in encoding for big-endian Unicode, this is pretty simple:

var source = "䐀攀氀攀琀椀漀渀匀琀愀琀攀";
var bytes = Encoding.BigEndianUnicode.GetBytes(source);
var result = Encoding.Unicode.GetString(bytes);
Console.WriteLine(result);

Turns out – just as in FakeUnicode’s screenshot – that’s the text “DeletionState” with the byte order flipped. We grabbed a few examples of the ‘Chinese’ text from the event log, ran it through this – sure enough, in every single case it’s a valid CRM database query that’s somehow been flipped into wrong-endian Unicode. At this point we start suspecting some sort of latent bug – this is old software, running on an old operating system,talking to an old database server, and sure enough, a bit of googling turns up a couple of likely-looking issues, most of which are addressed in various updates to SQL Server 2008. We take a VM snapshot in case everything goes horribly wrong, and one of the Ops gang volunteers to work late to get the server patched.

Next morning, turns out the server hasn’t been patched – because every single download of the relevant service pack has been corrupted. At which point all bets are off, because chances are the problem is actually network-related – which also explains where the ‘Chinese’ is coming from.

OK, let’s capture a stream of bytes from somewhere. Like, say, from the TDS data stream used by the MSCRMAsyncService

What does that say? If you think you know the answer, you’re wrong. Pop off and read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) – done? Awesome. NOW what do you think it says?

See, we have no idea. It’s a stream of bytes. Without some indication of how we’re supposed to interpret those bytes, it’s meaningless. OK, I’ll give you a clue – it’s UTF-16. Now can you tell what it says? No, you can’t – because (1) you don’t know whether it’s big-endian or little-endian, and (2) you don’t know where it started.

If we assume it’s big-endian, then the first byte pair – 00 48 – would encode the character ‘H’, the second byte pair – 00 65 – would encode ‘e’, and so on. If we assume it’s little-endian, then the first byte pair – 00 48 – encodes the character 䠀 – and suddenly the mysterious Chinese characters in the event log start to make sense.

Of course, the data stream between the MSCRMAsyncService and the SQL server hasn’t actually flipped from little-endian UTF-16 to big-endian – what’s happened is that the network connection between them is dropping bytes. And if you drop a single byte – or any odd number of bytes – from a little-endian Unicode stream, you get a sort of off-by-one error right along the rest of the data stream, resulting in all sorts of weirdness – including Chinese in the event logs.

Turns out there was a problem with the virtual network interface on the SQL Server box – which was causing poor performance, timeouts, bizarre query syntax errors, Chinese in the event logs, and corrupted service pack downloads. Fortunately the databases themselves were intact, so we offlined them, cloned the virtual disk they were sitting on, attached that to a different server and brought them back online.

Every once in a while, you get a weird problem like this. I’ve seen maybe half-a-dozen problems in my entire career that made absolutely no sense until they turned out to be a faulty network connection, at which point generally you not only solve the problem, but explain a whole load of other weirdness that you hadn’t got round to investigating yet. The only thing more fun than dodgy networks is dodgy memory – but that’s a post for another day.

Oh, and if you’re wondering about the title of this post, you clearly haven’t studied the classics.

Upcoming Conferences and User Group Talks

2016-09-15T17:49:00+00:00

I’m appearing at some conferences and user groups in October and November, talking about ReST, the history of the web, and how we can apply the scientific method to software development. Though not all at the same time.

October 19th I'll be speaking at FullStack Bytes at SkillsMatter here in London - one of a series of monthly events around the same themes and ideas as the annual FullStack conference - "JavaScript, NodeJS and the Internet of Things". On October 25th I’m heading up to Telford to give my Real World REST talk at the Shropshire Devs User Group.

November 5th, I’m one of the keynote speakers at Tampere Goes Agile, a free one-day conference in Tampere, Finland all about agile software development. The theme of the conference this year is ‘experimentation’, and I’ll be talking about the scientific method, the history of experimentation and how we can apply scientific principles to our own software projects and agile processes. It’ll be my first time visiting Finland – new country, new conference, and a new topic – and I’m looking forward to it immensely.

I’ll be speaking at Oredev in Malmo, Sweden on the 8/9/10th of November, where as well as a couple of technical talks, Rob Conery has signed me up for a ‘head to head’ session with Jimmy Bogard, which should be entertaining. Thanks, Rob... :)

Finally, I’ll be rounding out 2016 with a couple of talks at BuildStuff - in Vilnius on Nov 16-18 and then in Kyiv on the 21-22. If you’ve ever been, you’ll know that BuildStuff is an excellent conference with great people and great content – and if you haven’t, get yourself a ticket, come along and see for yourself.

Affordances, Signifiers, and Cartographobia

2016-07-27T17:38:00+00:00

One of the teams here is putting the finishing touches on a new online version of Spotlight Contacts, our venerable and much-loved industry guide that started life as a printed handbook way back in 1947. Along the way, we’ve learned some very interesting things about data, and how people perceive that their data is being used.

One of the features of the new online version is that every listing includes a location map – a little embedded Google Map showing the business’ location. When we rolled this feature out as part of a recent beta, we got some very unhappy advertisers asking us to please remove the map from their listing immediately. Now, most of these were freelancers who work from home – so you can understand their concerns. But what’s really interesting is that in most cases, they were quite happy for their full street address to stay on the page – it was just the map that they were worried about.

Of course, this immediately resulted in a quite a lot of “what? they want to keep the address and remove the map? ha ha! that’s daft!” from developers – who, as you well know, are prone to occasional outbursts of apoplectic indignation when they have to let go of their abstractions and engage with reality for any length of time – but when you think about it, it actually makes quite a lot of sense.

See, street addresses are used for lots of things. They’re used on contracts and invoices, they’re used to post letters and deliver packages. Yes, you can also use somebody’s address to go and pay them a visit, but there are many, many reasons why you might need to know somebody’s address that have nothing to do with you turning up on their doorstep. In UX parlance, we’d say that the address affords all of these interactions – the presence of a street address enables us to post a letter, write a contract or plan a trip.

A map, on the other hand, only affords one kind of interaction; it tells you how to actually visit somewhere. But because of this, a map is also a signifier. It sends a message saying “come and visit us” – because if you weren’t actually planning to visit us, why would you need to know that Spotlight’s office at 7 Leicester Place is actually in between the cinema and the church, down one of the little alleys that run between Leicester Square and Chinatown? For posting a letter or writing a contract, you don’t care – the street address is enough. But by including a map, you’re sending a message that says “hey – stop round next time you’re in the neighbourhood”, and it’s easy to see why that’s not really something you want if you’re a freelancer working from your home.

It’s important to consider this distinction between affordances and signifiers when you’re designing your user interactions. Don’t just think about what your system can do – think about all the subtle and not-so-subtle messages that your UI is sending.

Here’s the classic Far Side cartoon “Midvale School for the Gifted”, which provides us with some great examples of affordances and signifiers. The fact you can pull the door is an affordance. The sign saying PULL is a signifier – but the handle is both. Looking at it gives you a clue “hey – I could probably pull that!” – and when you do, voila, the door swings open. If you’ve ever found a door where you have to grasp the handle and push,, then you’ve found a false affordance – a handle that’s sat there saying ‘pull me…’ and when you do, nothing happens. And, in software as in the Far Side, there’s going to be times when all the affordances and signifiers in the world are no match for your users’ astonishing capacity to ignore them all and persist in doing it wrong.

(Far Side © Gary Larson)

ASP.NET Authentication with Adxstudio

2016-07-27T11:53:00+00:00

I’m looking into options for integrating our shiny new CRM system with our website, so we can provide all sorts of neat self-service capabilities and features. One of the applications I’m investigating is a thing called Adxstudio – now owned by Microsoft - which claims to “transform Dynamics CRM into powerful application platform with dozens of apps and starter portals.”

This is one of those situations where we really are dealing with ‘solved problems.’ Email campaigns. Customers updating their own contact details, potentially things like forums, helpdesk/ticketing systems – lots of things which are nice-to-have but really aren’t strategic differentiators, and so there’s a compelling argument to find an off-the-shelf solution and just plug it in. We already have a federated authentication system here at Spotlight – something we built a few years ago that provides basic identity and authentication capabilities on top of OAuth2. At the time we built it, OpenID Connect didn’t exist yet, so we’ve got a system that does basically the same thing but isn’t actually compatible with OpenID Connect – and consequently doesn’t work out-of-the-box with Adxstudio. So I’ve been poking around, trying to work out the best way to plug Adxstudio into our infrastructure so we can evaluate it as a solution.

One of the options on the table was to replace our existing authentication system with IdentityServer; another was to implement OpenID Connect support on top of our existing authentication system – both quite elegant solutions, but both of which involve quite a lot more work than is actually required for what we’re trying to do.

The core requirement here is:

We already have a CRM Contact record for every user of our system
We can look up a user’s CRM Contact GUID during authentication
We want to set up the Adxstudio MasterPortal demo so that our customers are seamlessly authenticated and can use Adxstudio features as though they had registered via the Adxstudio registration facility.

Now, one of the nice things about Adxstudio is that it’s built as OWIN middleware, and uses the ASP.NET Identity framework to handle authentication – so what we need to do is work out how to translate the CRM Contact GUID into an IPrincipal/IIdentity instance that we can assign to the HttpContext.Current.User property, and hope that Adxstudio then does the right thing once the HttpContext User is set correctly.

Adxstudio provides an implementation of ApplicationUserManager that’s already registered with the OWIN model, which accepts a CRM Contact GUID (as a string) and returns an instance of ApplicationUser that we can use to spin up a new ClaimsIdentity. So the simplest possible approach here is this snippet of code:

Doing this with a Contact that’s been created via the Adxstudio registration thing works just fine – but trying to do it with a ‘vanilla’ contact blows up:

As you can see from that stack trace, deep down buried under several layers of Adxstudio and ASP.NET Identity code, something is trying to construct a System.Security.Claims.Claim() instance and it’s blowing up because we’re passing in a null value for something that’s not allowed to be null. Unfortunately for us, because we don’t have the source for the thing that’s actually blowing up, we can’t see what the actual parameter values are that are causing the exception… so it’s time for a bit of hunch-driven development. :)

I’d already noticed that when you install Adxstudio into your CRM system, it adds a bunch of custom attributes to the Contact entity in Dynamics CRM; here’s a dump of those attributes for a working contact:

Attribute Key	Value
adx_changepasswordatnextlogon	False
adx_identity_emailaddress1confirmed	False
adx_identity_lockoutenabled	True
adx_identity_logonenabled	True
adx_identity_mobilephoneconfirmed	False
adx_identity_passwordhash	(omitted for security reasons)
adx_identity_securitystamp	ca49a664-0385-4eb4-90c0-6283c9e704ea
adx_identity_twofactorenabled	False
adx_identity_username	ali_baba
adx_lockedout	False
adx_logonenabled	False
adx_profilealert	False
adx_profileisanonymous	False
adx_profilemodifiedon	2016-07-20 09:18:43

The ASP.NET Identity model is generally pretty flexible, but I have a hunch that the username and the security stamp are both required fields because they’re fundamental to the way authentication works. So, let’s try inserting some code into the AuthenticateRequest handler that will check these fields exist, and update them directly in CRM if they don’t:

protected void Application_AuthenticateRequest(object sender, EventArgs e) {
Guid userGuid;
var cookie = Request.Cookies["crm_contact_guid"];
if (cookie != null && Guid.TryParse(cookie.Value, out userGuid)) {
    var http = HttpContext.Current;
    var owin = http.GetOwinContext();
    var userManager = owin.Get<ApplicationUserManager>();
    var user = userManager.FindById(userGuid.ToString());
    if (String.IsNullOrEmpty(user.UserName)
        ||
        String.IsNullOrEmpty(user.SecurityStamp)
     ) {
       // "Xrm" is the connection string name from web.config
       using (var crm = new OrganizationService("Xrm")) {
         var entity = crm.Retrieve("contact", userGuid, new ColumnSet(true));
         entity.Attributes["adx_identity_securitystamp"] =
           Guid.NewGuid().ToString();
         entity.Attributes["adx_identity_username"] =
           Guid.NewGuid().ToString().Substring(0, 8);
         crm.Update(entity);
      }
    }
    var identity = user.GenerateUserIdentityAsync(userManager).Result;
    HttpContext.Current.User = new RolePrincipal(identity);
}
}

(For the sake of this demo, all we care about is making sure those values are no longer null. In reality, make sure you understand the significance of the username and security stamp fields in the identity model, and populate them with suitable values.)

OK, this now works sometimes – but only following an IISRESET. Turns out that Adxstudio is actually caching data from CRM locally, so although that new chunk of code is updating the Contact entity into a valid identity, the Adxstudio local cache doesn’t see those changes because it’s looking at an out-of-date copy of the Contact entity. So… time to configure some cache invalidation.

You can read about Adxstudio’s web notifications feature here. Adxstudio includes some code that will call a cache invalidation handler on your own site every time an entity is updated. Which works just fine IF CRM Online can see your Adxstudio portal site. And right now I’m running CRM Online as a 30-day trial and I’m running Adxstudio on localhost, and my workstation isn’t on the internet, so CRM Online can’t see it.

Time to fire up my favourite toolchain – Runscope and Ngrok. First, I’ve set up ngrok so that any requests to mytunnel.ngrok.io will be forwarded to my local machine – you’ll need a paid ngrok license to use custom tunnel names, but if you’re using the free version try this:

D:\tools\ngrok>ngrok http –host-header=adx.local 80

Now, as long as that NGrok process is running, you can hit that URL – http://ef736c25.ngrok.io/ – from anywhere on the internet, and it’ll be tunneled to localhost on port 80 and have the host-header rewritten to be adx.local. This neatly solves the problem of CRM Online not being able to connect to my local Adxstudio instance.

Next, just to give us a bit of insight into what’s going on, I’m going to set up a Runscope bucket for that. Remember – we need to route requests to /cache.axd on our local Adxstudio portal instance, via ngrok, so here’s how to get the Runscope URL you’ll need:

So, last step – you see that big URL in the middle? We need to tell the Adxstudio Web Notifications plugin to notify that URL every time something changes. The option is under CRM > Settings > Web Notification URLs.

Note that the Adxstudio documentation refers to a Configuration screen accessible from the Solutions > Adxstudio Portals Base. It appears this screen doesn’t exist any more – I certainly couldn’t find any trace of it in my CRM Online instance – but it also appears it isn’t necessary, because as soon as I’d created and active Web Notification URL things started happening.

So, now we have something that works – but it still fails on the first Portal request for a particular Contact, probably because the Adxstudio cache isn’t picking up those two new fields fast enough for the login to succeed. To work around this, I’ve put in a thread sleep and then an HTTP redirect, so the first time a user lands on the portal they’ll get a slight delay whilst we populate their Adxstudio attributes, and then they’ll get their personalised screen:

protected void Application_AuthenticateRequest(object sender, EventArgs e) {
Guid userGuid;
var cookie = Request.Cookies["crm_contact_guid"];
if (cookie != null && Guid.TryParse(cookie.Value, out userGuid)) {
    var http = HttpContext.Current;
    var owin = http.GetOwinContext();
    var userManager = owin.Get<ApplicationUserManager>();
    var user = userManager.FindById(userGuid.ToString());
   if (String.IsNullOrEmpty(user.UserName)
        ||
        String.IsNullOrEmpty(user.SecurityStamp)
     ) {
       using (var crm = new OrganizationService("Xrm")) {
         var entity = crm.Retrieve("contact", userGuid, new ColumnSet(true));
         entity.Attributes["adx_identity_securitystamp"] =
           Guid.NewGuid().ToString();
         entity.Attributes["adx_identity_username"] =
           Guid.NewGuid().ToString().Substring(0, 8);
         crm.Update(entity);
         Thread.Sleep(TimeSpan.FromSeconds(5));
         // Redirect back to the same page, so that Adxstudio will
         // retrieve a fresh copy of the cached data.
         http.Response.Redirect(http.Request.RawUrl);
      }
    }
    var identity = user.GenerateUserIdentityAsync(userManager).Result;
    HttpContext.Current.User = new RolePrincipal(identity);
}
}

And it works. The final step for me was to spin up a separate web app that lists all the Contacts in the CRM system, with a login handler that puts the CRM Contact GUID into a cookie and redirects the browser to http://adx.local/ – and it works. No registration, no login, and any user with a valid CRM Contact GUID can now log directly into the Adxstudio MasterPortal example.

ASP.NET Core 1.0 High Performance

2016-07-20T14:28:00+00:00

I have exciting news. My friend and erstwhile colleague James Singleton has just published his first book, ASP.NET Core 1.0 High Performance. James was gracious enough to invite me to contribute the foreword – and since the whole point of a foreword is to tell you all why the book is worth buying, I figured I’d just post the whole thing. Read the foreword, then read the book (or better still, buy it then read it.)

TL;DR: it’s a really good book aimed at .NET developers who want to improve application performance, it’s out now, and you can buy your copy direct from packtpub.com.

And that foreword in full, in case you’re not convinced:

"The most amazing achievement of the computer software industry is its continuing cancellation of the steady and staggering gains made by the computer hardware industry."
- Henry Petroski

We live in the age of distributed systems. Computers have shrunk from room-sized industrial mainframes to embedded devices smaller than a thumbnail. However, at the same time, the software applications that we build, maintain and use every day have grown beyond measure. We create distributed applications that run on clusters of virtual machines scattered all over the world, and billions of people rely on these systems, such as email, chat, social networks, productivity applications and banking, every day. We're online 24 hours a day, 7 days a week, and we're hooked on instant gratification. A generation ago we'd happily wait until after the weekend for a cheque to clear, or allow 28 days for delivery; today, we expect instant feedback, and why shouldn't we? The modern web is real-time, immediate, on-demand, built on packets of data flashing round the world at the speed of light, and when it isn't, we notice. We've all had that sinking feeling... you know, when you've just put your credit card number into a page to buy some expensive concert tickets, and the site takes just a little too long to respond. Performance and responsiveness are a fundamental part of delivering great user experience in the distributed age. However, for a working developer trying to ship your next feature on time, performance is often one of the most challenging requirements. How do you find the bottlenecks in your application performance? How do you measure the impact of those problems? How do you analyse them, design and test solutions and workarounds, and monitor them in production so you can be confident they won’t happen again?

This book has the answers. Inside, James Singleton presents a pragmatic, in-depth and balanced discussion of modern performance optimization techniques, and how to apply them to your .NET and web applications. Starting from the premise that we should treat performance as a core feature of our systems, James shows how you can use profiling tools like Glimpse, MiniProfiler, Fiddler and Wireshark to track down the bottlenecks and bugs that are causing your performance problems. He addresses the scientific principles behind effective performance tuning - monitoring, instrumentation, and the importance of using accurate and repeatable measurements when you’re making changes to a running system to try and improve performance.

The book goes on to discuss almost every aspect of modern application development - database tuning, hardware optimisations, compression algorithms, network protocols, object-relational mappers. For each topic, James describes the symptoms of common performance problems, identifies the underlying causes of those symptoms, and then describes the patterns and tools you can use to measure and fix those underlying causes in your own applications. There’s in-depth discussion of high-performance software patterns like asynchronous methods and message queues, accompanied by real-world examples showing how to implement these patterns in the latest versions of the .NET framework. Finally, James shows how you can not only load test your applications as part of your release pipeline, but can continuously monitor and measure your systems in production, letting you find and fix potential problems long before they start upsetting your end users.

When I worked with James here at Spotlight, he consistently demonstrated a remarkable breadth of knowledge, from ASP.NET to Arduinos, from Resharper to resistors. One day he’d be building reactive front-end interfaces in ASP.NET and JavaScript, the next he’d be creating build monitors by wiring microcontrollers into Star Wars toys, or working out how to connect the bathroom door lock to the intranet so that our bicycling employees could see from their desks when the office shower was free. Since James moved on from Spotlight, I’ve been following his work with Cleanweb and Computing 4 Kids Education. He’s one of those rare developers who really understands the social and environmental implications of technology - that whether it’s delivering great user interactions or just saving electricity, improving your systems’ performance is a great way to delight your users. With this book, James has distilled years of hands-on lessons and experience into a truly excellent all-round reference for .NET developers who want to understand how to build responsive, scalable applications. It’s a great resource for new developers who want to develop a holistic understanding of application performance, but the coverage of cutting-edge techniques and patterns means it’s also ideal for more experienced developers who want to make sure they’re not getting left behind. Buy it, read it, share it with your team, and let’s make the web a better place.

Check it out. The chapter on caching & message queueing is particularly good :)

Why I'm Voting Remain: Social Media Redux

2016-06-15T17:48:00+00:00

So, I wrote a thing about why I'm voting to remain in the EU, and lots of people liked it, but pointed out it was quite long. And they're right. It was. So here's a really nice short manifesto-style version that I think captures the gist of the thing.

If you want to share it, please go ahead

Twitter version
Facebook post
You can even download a printable PDF version of it here.

Regular readers – don't go away. Normal tech blogging will resume on the 24th, one way or another. But this is a once-in-a-generation event, and it's important.

Why I’m Voting to Remain in the European Union.

2016-06-14T23:47:00+00:00

There's a referendum coming up. I'm not going to talk about why it’s happening. I'm not going to share any “facts” about politics or the economy, beyond a few historical details that are the basis for my opinion on this. But I am going to tell you why I'm voting for the UK to remain in the European Union.

I live in London. I’m British, though I grew up in Africa. I’ve never been much of a patriot, although I’ll confess to the odd surge of pride when it comes to Pink Floyd, the BBC and Thrust SSC. I’ve spent most of my adult life working in software development, and I'm lucky enough to work with an outstanding team of people – some of the brightest, smartest people you'll ever meet. More than half of our development team are EU citizens who are living and working in London thanks to the UK being part of the EU. I love working with these people, and there's numerous studies supporting my own anecdotal experience that working with people you like is a massive factor in how happy you are. And, like many Londoners, my friends are a motley assortment of people from all over the world, and a lot of them are here as citizens of the EU. On a very immediate level, it would really suck if they all had to either leave the UK or go through the merry hell of applying for permanent leave to remain, and potentially have to choose between their native citizenship and becoming British citizens. But that's both a very personal and a very temporary thing. I’m sure we'd all get over it eventually.

In the last year, I've been lucky enough to visit Lithuania, Denmark, France, Belgium, Norway and Ukraine. France and Belgium were founder members of the EEC, the predecessor to the European Union. Denmark and the UK joined in 1973. Lithuania, part of the USSR until 1991, became a member of the EU in 2004. Norway is a bit special, but it is a part of the European Economic Area and the Schengen visa area. Ukraine has gone, within my lifetime, from being part of the Soviet Union to winning the Eurovision Song Contest. Ukraine borders four EU member states (Romania, Hungary, Poland, and Slovakia), and whilst I don't think it will be admitted as a member of the European Union any time soon, the prospect of closer integration with the EU is a huge factor in Ukrainian politics. Whilst the UK was gearing up for a referendum about whether to leave the EU, two hundred thousand Ukrainians were marching through Independence Square in protest at their government's failure to sign a trade agreement that would have represented one small step towards eventual EU membership. I've visited all these countries without applying for a visa. I've spent time in these remarkable places with friends and colleagues from all over the world.

In Oslo, there is a museum dedicated to the great Norwegian explorer Thor Heyerdahl, who dedicated his life to the study of human migration, researching the astonishing stories of people who crossed Earth's vast oceans on flimsy wooden boats in search of a better life. This wanderlust, the urge to venture beyond our comfort zone, is as old as humanity. Along one wall is a quote from Heyerdahl:

“Borders - I have never seen one. But I have heard they exist in the minds of some people.”

He's right. Borders aren't physical things. Sure, some of them run along coastlines and rivers, but that's not the point. Borders are where, once upon a time, we drew a line in the sand and agreed that as long as your lot stayed on your side and our lot stayed on our side, it would be OK if maybe we stopped killing each other for a bit. And, slowly, as we learned to put down the pointed sticks and talk to each other instead, the world has changed. We can cross those borders now. We can live, work and study abroad. We can make friends, fall in love, start families, start businesses. Fifty years ago, someone from Birmingham falling for someone from Bucharest would have been a Pulitzer Prize-winning novel. Today it's just one more data point on Tinder (Or Grindr, depending who's playing.)

And that's why this matters to me. The impact of next week’s referendum will last for generations. It isn't about the petty power struggle between David Cameron and Boris Johnson. It isn't about Jeremy Corbyn or UKIP or immigration or the "war on terror" or economic subsidies. It's about peace, and stability, and the opportunities that affords all of us. The author Charlie Stross captured perfectly my thoughts about this when he wrote:

"The EU is the current incarnation of an institution established in 1947 to ensure that never again would the nations of western Europe go to war with one another."

The European Union is the legacy of a group of nations who, still reeling from the horror and the bloodshed of World War II, agreed that whatever the challenges the future might hold, we would be stronger facing those challenges together. They believed, as I do, that a unified Europe represents a significant step on the road towards a better world for humankind. Next week, the UK goes to the polls to decide whether we want to step off that road or not. Whether we want to close our borders, lock our doors and go it alone. To me, this is tantamount to a decision about whether we, the citizens of the United Kingdom, believe that we deserve to prosper while others are suffering, and there's really only one way I can answer that question and still sleep at night.

Neither side is presenting a terribly cohesive argument. Speculation, scaremongering and misinformation are absolutely rife on both sides. Nobody knows what'll happen if we leave. Nobody really knows what'll happen if we stay. I'm not quite sure what "vote remain" are campaigning on, but from the leaflets I’ve been getting through my front door, it looks to me very much like the "vote leave" camp is campaigning on fear. Fear of immigration and poverty and hardship. Fear of terrorism. Fear of things which are a fact of life for billions of people around the world. I am not scared of these things, because I believe that if we work together, we can solve them, just as we brought lasting peace to the countries of western Europe after centuries of frequent, bloody conflict. Twenty years ago, most British people’s idea of a terrorist was an Irishman in a ski mask. Things can change, and they will.

That doesn’t mean I’m not scared. I'm scared that fear and misinformation are going to rule the day. I'm scared for my friends and colleagues for whom a "leave" result on the 23rd could have a real, immediate impact on their lives and livelihoods. I'm scared that, a century from now, some kid is going ask grandad how the war started, and he’ll tell them a story that starts with Brexit. If you don't think that's a possibility, consider that it's less than a hundred years since the Battle of the Somme - nearly a million British, French and German soldiers killed and wounded on the fields of northern France. No Islamic State, no al-Qaeda, no "war on terror" - just honest God-fearing white guys killing one another for the hell of it. Within thirty years, they'd do it all over again – nearly a thousand veterans of that conflict are still alive today.

Since the 1950s, the EEC, and later the EU, has grown from six countries to twenty-eight, and there has never been an armed conflict between two EU member states. And if you think that EU membership is expensive, remember that World War I nearly bankrupted the UK, and World War II saw us borrow so much money from the United States that we didn't finish paying them back until 2006. Peace is small potatoes compared to the cost of war.

Different people care about different things, and I respect that. Ask yourself what matters to you, research your position, and vote accordingly. Maybe even draw up a list of things you hope are going to happen as a result, so you can validate your assumptions a few years from now. But these are the things that matter to me, and I haven’t heard a single plausible argument that leaving the EU will improve any of them.

I believe that peace is more important than politics. I believe that the desire to travel in search of new ideas and new experiences is fundamental to what makes us human. I believe that open borders and common markets give us unprecedented freedom to explore the world we live in. I believe that membership of the European Union embodies all of these principles, and I’m voting to remain.

Join us at the Progressive.NET Tutorials 2016!

2016-06-13T12:32:00+00:00

Next week sees the return of the Progressive.NET Tutorials at SkillsMatter here in London.

Progressive.NET is a unique event. It goes beyond the high-level slides & code demos of most conferences, and offers a series of hands-on, in-depth workshops with some of the best speakers and experts in the .NET world. This year we’ve got a great lineup of speakers and workshops. On Wednesday, Glenn Block will be showing you how to run C# outside your IDE using ScriptCS, Ian Cooper will be demonstrating high-availability patterns for distributed systems, Toby Henderson will be showing you how to get up and running with .NET Core, and I’ll be running a new workshop about asynchronous programming patterns, async/await and how you can use them to deliver more responsive apps.

Thursday, we’ve got Ashic Mahtab showing you how to process millions of messages a second using Apache Kafka, we’ve got Mark Gray talking about machine learning in F#. After lunch Ben Hall will be showing you how to get up and running with .NET and Docker, and Phil Trelford will be building a compiler with F#, starting with an abstract syntax tree and a parser, and ending up with a compiler that generates .NET IL code or JavaScript. Thursday wraps up with the Progressive.NET Party – food, drinks, and a chance to hang out with the Progressive.NET speakers and developers in the Space Bar at SkillsMatter’s amazing CodeNode building.

On Friday, we round out the event with a day of talks from some of the best speakers in .NET

Rachel Reese on event-driven microservices
Charalampos Karypidis on isomorphic JavaScript applications and .NEt
Sebastien Lambla on API versioning (and why it’s evil)
Rajpal Singh Wilkhu on OpenID Connect and Identity Server
David Whitney on metaprogramming
Harry Cummings on Node.js for .NET developers
Barbara Fusinska talking about predicting the future with Azure machine learning
Sam Elamin talking about metrics-driven development
Liam Westley on App 2.0 and why the web lost
Evelina Gabasova on how to spice up your website with machine learning

Progressive.NET has been running since 2009. It’s an excellent event, and it’s also a great way to keep up to date with everything that’s new in the .NET world. If you’ve been hearing about things like .NET Core, Docker, Identity Server, async and await, machine learning – and you’re not quite sure what they are or how they apply to you – come along to Progressive.NET. By the end of the week, you’ll have a head full of new ideas, a laptop full of working code you can refer back to, and a load of new friends who can help out when you get stuck.

The event is next week, 22-24th June, at the SkillsMatter CodeNode here in London. Tickets are normally £875+VAT, but you can use the code SPECIAL_LDNUG_PROGNET to get 40% off - £525 instead of £875 – so what are you waiting for? Get your ticket now and we’ll see you there!

The Next Big(int) Thing

2016-05-13T14:15:00+00:00

One of our systems here uses a bigint identity column as a database primary key – because we knew when we built it, back in 2010, that we were going to end up with more than 2,147,483,647 records.

Well, that happened at 12:02 today, and a couple of systems promptly failed – because, despite the underlying database being designed to handle 2^63 records, the POCOs that were being mapped to those classes were using a regular C# int to store the record ID, and so as soon as they got an ID from the database that's bigger than Int32.MaxValue, they blew up. Thanks to the underlying DB schema already supporting 64-bit IDs, the fix was pretty simple – just change int to long in a few carefully-selected places and redeploy the applications – but it's still annoying that something we knew about, and planned for, still came back to bite us. So I started thinking – how could we stop this happening?

The problem is that, despite being a bigint column, we just accepted SQL Server's default identity setting of (1,1) – i.e. start counting at 1, and increment by 1 each time. Which means that until you hit 2-billion-and-something records, it doesn't actually make any difference - and that takes a while. In our case, it took 5 years, 8 months and 26 days. During that time we've made hundreds of changes to our code, and in a handful of those cases, we've mapped that bigint ID onto a regular C# Int32 – and so inadvertently planted a little time-bomb in our production code. Tick, tick, tick…

So here's a nice neat solution, that I wish I'd thought of five years ago. Anytime you create a bigint identity, seed it with (2147483648, 1) – so that right from day one, it's already too big to fit in an Int32. Any system that tries to store an ID in a regular int variable will fail immediately, not in five years when someone creates that magic 2.14-billion-and-somethingth record. Even though you've effectively thrown away 2^32 possible values, you have another (2^64 - 2^32) values to play with, so you've lost a tiny, tiny fraction of the available keyspace in exchange for immediate feedback if any of your client apps can't cope with 64-bit ID values.

We're Gonna Build a Framework!

2016-05-05T17:32:00+00:00

Check it out! I made a thing. A musical thing. Well, Billy Joel made the musical thing (which is copyright © 1989 Columbia Records, btw) and I wrote and recorded a new set of lyrics for it, all about software frameworks and how lovely it is to have so many to choose from.

And, for anyone who doesn't believe that every single one of those really is a software development framework, here's the Googlified lyrics.

Handlebars, Hibernate, Solar, Activate,
Phalcon, Flask and Silverstripe and TYPO3 Flow,
Agavi, Pixie, Hazaar MVC,
CodeIgniter, Lithium and PRADO.

Raphael, Bobo, Bottle and Tornado,
Django, CherryPy and WSGI
Glashammer, WebSphere, RedBean, TurboGears,
Albatross, Aquarium, Selenium, web.py

CHORUS:

We’re gonna build a framework,
‘cos we wanna use one, but don’t wanna choose one,
We’re gonna build a framework,
we didn’t like the others, so we’ll write another…

SiteCore, Tapestry, Maverick and JSP,
Barracuda, Ay Caramba, Groovy on Grails,
Intercal on Interstates, Cascade, NHibernate,
JDBC, Ruby on Rails
Jasmine, Doctrine, Java Forms Engine,
Active Record, D3, Dapper and Velocity,
Thymeleaf, TopLink, Pyramid, Rethync,
Aura, Rico, Midori and Mojito,

Sitemesh, Cymbeline, Enterprise Java Beans,
Hug, Grok, Boost, Click, anything by Telerik,
Rango, Dojo, LLBLGen Pro,
Carbonado, Seaside, Pylons and Pyroxide

Fusebox, Flight, Flex, ServiceStack, Silex,
Carbon, Cocoa, Ample and Giotto,
Banshee, Symfony, Laravel, Fat Free,
Mocha, Pecker, Hobo, Cuba and Rialto

OpenRasta, Nancy, ASP net MVC,
Kendo, Zend, ODBC and Tempo,
Java Server Faces, Entity Spaces,
Cappuccino, SpecFlow, Polymer, JDO,
Google Web Toolkit, no need to learn Javascript,
JDK, code away, what else do I have to say?

Prototype, Boilerplate, jQuery, animate,
Mustache, Wijmo, Ionic and Allegro,
Bootstrap, Backbone, running on an iPhone;
Mobile apps, going live, built in HTML5
Angular, Scriptaculous, react, redux,
Knockout, ember, does anyone remember,
It’s just like in the browser wars,
IExplore and Netscape 4,
lodash vs underscore,
and I can’t take it any more!

(yes, I know ODBC and JDBC aren't strictly speaking frameworks.)

(yes, I also know the recorded version uses Tempo twice. Relax. It's OK.)

London.NET User Group – A Progressive.NET Special!

2016-05-04T23:30:00+00:00

This July, the Progressive.NET Tutorials returns to SkillsMatter – for the eighth year running, we’re bringing together some of the best speakers and experts in the .NET community for three days of in-depth workshops and hands-on tutorials. The team at SkillsMatter are working hard to finalize what promises to be a great programme, covering all the latest platforms, patterns and ideas in the world of .NET development.

To coincide with this, we’re running a special London.NET User Group meetup at SkillsMatter the night before the conference – Tuesday 21st June.

We’re going to be hosting a series of short talks and demos, around a theme of “zero footprint” development tools. Over the years, we’ve had a lot of really excellent talks covering languages, platforms, patterns, architectural styles – but quite often, we’ll be in the pub afterwards chatting about the talk we’ve just seen, and you’ll hear people saying “it looks really great; maybe I can try it out on my next project” or “I love the idea but I can’t see the rest of the team going for it.”

So, for our Progressive.NET special, we want to showcase the very best tools, utilities and ideas that you can start using right away – they won’t modify your code, they won’t break your build pipeline, they won’t interfere with what the rest of your team are doing, but they might make your day just a little bit more pleasant – and we want your help!

Come along and give us a 10-minute demo of something you think is awesome. Maybe it’s an open source tool, maybe it’s one of your company’s products, maybe it’s just something you use daily and can’t live without – but remember: zero footprint. Don’t break the build, don’t don’t move the cheese.

In the spirit of things like Scott Hanselman’s 2014 Ultimate Developer and Power Users Tool List, we’re looking for talks about:

Test runners like NCrunch and TestDriven.NET
Git clients – like GitHub Desktop and GitKraken
Static analysis tools, like NDepend and StyleCop, that help you visualise the complexities of the code you’re working on.
Diagnostic and monitoring tools like ngrok, RunScope and ANTS Profiler
Productivity tools for developers, like Powercmd and Instant Eyedropper

Whether you’re a first-timer speaker or a fulltime product evangelist, we’d love to have you on board, so have a think, get in touch – by email or find us @LondonDotNet on Twitter - and get involved!

Hintjens

2016-04-27T01:30:00+00:00

I’m at BuildStuff Vilnius, in November 2015. It’s Thursday night. Mark Rendle and I are doing our comedy panel game quiz thing. We found out about ten minutes ago that we’re doing our show in a nightclub, with no wi-fi, hardly any microphones and… basically it’s a bit of a train crash. And we’re hustling for volunteers to help us make the train crash funny. Between Mark’s friends and mine, we rope in half-a-dozen people. One of them is this weird tall Belgian guy. He’s good; they’re all good. He gets it. He’s funny and engaging and genuinely interesting. We somehow walk away with our collective dignity intact, and people even tell us afterwards that they loved it. Mark and I swear to each other we’re never doing that particular show in a nightclub ever again.

We’re still in Vilnius, it’s Saturday, and I’ve had the day off. I’ve been sightseeing. I’m tired and hungry, and I don’t want to just head back to the hotel, so I find a café with wi-fi, I look up Vilnius on MetalTravelGuide.com, and I find this bar – Bix Baras. I go in, I chat to the staff. Their English, like their beer, is excellent; my Lithuanan barely covers “hello” and “thank you” – but I eat lunch and have a few beers, and then head back to the hotel. When I get there, the tall Belgian guy from the quiz is playing the piano – one of those wonderful grand pianos that adorn hotel bars the world over without anyone ever really playing them. I ask if I can join him; he moves over, I sit down, we play for a while – he’s doing most of the work, I’m just bouncing along on the white notes, picking out pentatonic minor melodies that fit with what he’s doing. It’s fun. It’s nice, and it feels somehow conspiratorial – for starters, we’re playing one of those pianos that you probably walk past every day of your life and assume that it’s for Other People to play, and not for you, and yet here we are.

I go upstairs, change my shirt, ask Seb if he fancies heading downtown for a beer. We need a night off from the whole conference crowd, but on a whim I ask the Belgian guy on the piano if he wants to join us. He says yes, and introduces himself as Pieter. We’ve officially met. We jump in a cab and head downtown.

Bix Baras has good beer, and great snacks, and we talk – myself and Seb and Pieter. We drink, we eat hard cheese and pigs’ ears and Lithuanian dark bread. And it’s remarkable, because there’s no small talk. We talk about ideas, and we share experiences. The conversation is disarmingly easy. I’m not used to this. Most people at conferences talk about tech – about .NET or NodeJS or Docker. We talk about life. We talk about what we do, and why we care. We talk about friendship, and failed relationships, and psychopaths, and adventures. We walk up the road to the pool hall where some of the other BuildStuff gang are having drinks. Pieter and I get talking about speaking. Within the hour, he’s challenged pretty much every idea I’ve ever had about speaking and giving talks, but it doesn’t feel adversarial – there’s something genuinely inspirational about it. We finish our drinks and wander back to the hotel, but the conversation resonates.

Sunday, we fly to Kyiv – a whole crowd of us. I’m walking next to Pieter on the tarmac as we head out to our plane, and he’s talking about how much he’s enjoying the experience – “For the first time ever I feel like I’m on the road with my gang” – and I know exactly what he means. The sense of camaraderie is wonderful – 30-odd hardcore geeks heading out to Ukraine together – yet it somehow didn’t really click until Pieter pointed it out.

In Kyiv, we hang out. We chat. We talk about code, about community, about psychology. I watch his talk about building open source communities. From where I’m sitting, he appears to give a 50-minute talk with no notes and no slides, and solve a Rubik’s cube while he’s doing it. He confides in me afterwards that the cube was a bit of a stunt – shuffle it a couple of turns, memorise them, play them backwards on stage – but that almost doesn’t matter; the talk is brilliant, the audience are involved and engaged, and I’m sat there wondering how much of my life I’ve spent making Powerpoint slides, and why…

Eventually, Pieter turns our conversation in that bar in Vilnius into a blog post – Ten Steps to Better Public Speaking – which is simultaneously gratifying and terrifying. Gratifying that he thinks our conversation is interesting enough to warrant an entire blog post. Terrifying, because when you’re name-checked in a post like that, the only thing you can really do is rise to the challenge, and that means I’m gonna need to REALLY work hard on… well, on every talk I ever give again.

Months pass. I think often of our conversation in that pool hall in Vilnius and the blog post that followed. One day, I email Pieter – “Hey, remember that chat in Vilnius? Do you fancy doing a joint talk at NDC Oslo?” He says yes, I write something up, I send it over, and start worrying about the fact I’ll be sharing a stage with the great Pieter Hintjens – and about the fact I’ve signed up to give a talk that’s gonna drag me out of my comfort zone in almost every way.

By chance, I’m in Brussels in March, en route to a long weekend in Leuven with my girlfriend. I email Pieter, we arrange to meet for lunch: we talk about ideas. He’s riffing on ideas – about opening an office in Brussels for people who need a place to hack; about using mesh networking to build “smart chairs” that tell the pavement café when they need replacing; about speaking and software and people and life. He talks about his father, about euthanasia, about family. We talk briefly about our joint talk and NDC, but not too much; after all, too much rehearsal would undermine the vulnerability. And we part with a hug, and a promise to see each other in Oslo.

I watch Pieter and @jesslynnrose joking on Twitter about gender-swapped TV shows. Pieter posts this: prescient, or just meditative? Then on March 26th, following a whole lot of the kind of fallout that just doesn’t fit into 140 characters, Pieter announces he’s leaving Twitter. I’m sad to see him go, but have no doubt I’ll have many more evenings hanging out and having my preconceptions challenged by this remarkable individual.

Then I get an email. The subject just says “NDC” It reads:

“Hi Dylan,
Seems my cancer has come back... still waiting for detailed prognosis and next steps. Looks pretty bad atm. In any case, no travel for me for the next months.
You're going to have to do the talk by yourself. Stick to the ten rules, watch my Serbian video a couple of times and you'll do fine. :)
Sorry about this.”

I don’t care about the talk. I’m worried about my friend – this sounds bad. I email him back. He replies. Time passes. He rejoins Twitter, because it’s a good way to connect with a lot of people who want to know what’s going on. And then he posts this:

We will try chemotherapy. It's palliative, there is no cure for this. So, time to start saying goodbye.

And then he posts “A Protocol for Dying”, and it’s pretty clear that this is it. One way or another, it won’t be long before Pieter’s not around any more. And people start talking, and posting, and tweeting… and before long, a common thread emerges. It seems you really didn’t need to spend very much time with Pieter for him to leave a lasting impression.

I spent five days with Pieter late last year, and had lunch with him once, a few months ago. I’ve never visited his house, never met his family, never collaborated with him – but the time I’ve spent with him and the conversations we’ve shared have been some of the most profoundly challenging and inspiring interactions I’ve had in a very long time. And it’s not just me. There are countless comments on Pieter’s most recent blog posts from people who met him once or twice – or not at all, in the case of the people who know Pieter through email and through his code – but whom nevertheless believe that knowing him has had a profound impact on their life.

I was in a restaurant earlier tonight, with my girlfriend, Clare, and some of my cow-orkers. We ended up talking about Pieter. Clare met Pieter briefly, for about five minutes, in Bruxelles-Midi railway station back in March. At the time, Clare was feeling completely freaked out at being in an unfamiliar country where she didn’t speak the language or know how things worked, and my meeting up with this weird guy who “looked really stern” didn’t help at all. Pieter warned us (a pair of hardcore Londoners) about the risks and dangers of hanging out in the station, and then helped Clare find her train to Leuven.

When I got that first email from Pieter, I told Clare. When I saw his Twitter post, and when he posted “A Protocol for Dying”, I told Clare – and she’d already read it. And then she said to me tonight “I want to email Pieter. I don’t know him, but I know what’s happening, and I just want him to know that I’ll remember him next time – probably every time? – that I go through Brussels, and I hope one day I’ll be a bit more badass – just like he is.”

So here’s to Pieter, and here’s hoping that long after he’s stopped coding and tweeting and blogging, he’ll still be inspiring all of us to open up, to embrace our vulnerability and “to be a bit more badass”.

Coming Soon To A City Near You…

2016-04-19T10:47:00+00:00

Following on from "The Rest of REST" talk that I've given at several conferences over the last year, I'll be talking about real-world REST at several user groups over the next few months. In this talk, I'll be exploring some REST architectural patterns in more depth, and doing some hands-on demos showing how you can implement these patterns in your own applications.

For this demo, I'll be using NancyFX and the HAL+JSON hypermedia language – there's a beautiful synergy between the dynamic type model in C#, the dynamic model used in NancyFX and the way HAL exploits the dynamic features of JavaScript to extend JSON into a powerful hypermedia format, and I'll be showing how you can wire all these bits together to build flexible, RESTful HTTP APIs.

May 9th I'll be at Smart Devs User Group in Hereford.

May 18th I'll be at The Dev Bakery in Altrincham.

May 26th, I'll be at the Copenhagen .NET User Group in Denmark – and many thanks to the team over at Siteimprove for hosting and sponsoring the event.

As always - if your user group would be interested in a session on this or one of my of other talks, get in touch!

I'll also be appearing at these conferences throughout 2016:

8/9/10 June I’ll be at NDC { Oslo } in Norway, and I’ll be sticking around for PubConf at NDC { Oslo } on 11th June – the PubConf event in London here in January was an absolute blast, so if you’re going to be in Oslo for NDC, book yourself a hotel for the weekend, see some of the city, and come to PubConf.

22-24 June I’ll be at the Progressive.NET Tutorials at SkillsMatter here in London, running a deep-dive hands-on workshop which will probably be on async/await and asynchronous programming in C# – watch this space for further announcements as we finalise the programme.

8-9 July I’ll be at BuildStuff Odessa – following the successful and hugely enjoyable BuildStuff conference in Kyiv last year, I’m really excited to be going back to Ukraine for a weekend of code, games and sunshine by the sea.

13-15 July I’ll be at FullStack 2016 in London, the conference on “JavaScript, NodeJS and the Internet of Things” – I’ve been helping the team at SkillsMatter finalise the programme for this one, we’ve got some absolutely excellent speakers and sessions lined up, and it promises to be a great event.

3/4/5 August I’m going to be at NDC { Sydney }, joining an amazing programme of speakers to bring the NDC experience to the southern hemisphere.

See you on the road!

The Axosoft GitKraken Songwriting Battle!

2016-04-15T01:14:00+00:00

A few weeks ago, I stumbled across GitKraken – a cross-platform GUI for Git that looks, well, lovely. ‘Cos git is a wonderful, amazing, powerful system, but it’s about time somebody added a little, well, beauty around the place.

So I’m using GitKraken daily now, and loving it, and then I find out that they’re having a contest. A song-writing contest. About revision control systems. Now, I love music. I play instruments. I write songs. I write parody songs about tech stuff. There is absolutely no way I can see a songwriting contest about revision control and not enter it… so I did. And here’s my entry. A completely original song, written and recorded for the contest, and called, simply, Git Kraken.

Exupérianism: Improving Things by Removing Things

2016-04-08T10:53:00+00:00

Last night this popped up on Twitter:

There should be a term for when you add a new feature with net negative lines of code. pic.twitter.com/yZB0uuq3H5
— John Sheehan (@johnsheehan) April 7, 2016

Last year, as part of migrating our main web stack to AWS, we created a set of conventions for things like connection strings and API endpoint addresses across our various environments, and then updated all of our legacy systems to use these conventions instead of fragile per-environment configuration. This meant deleting quite a lot of code, and reviewing pull requests with a lot more red lines than green lines in them – I once reviewed a PR which removed 600 lines of code across fifteen different files, and added nothing. No new files, no new lines - no green at all – and yet that change made one of our most complicated applications completely environment-agnostic. It was absolutely delightful.

When I saw John's tweet, what instantly came to mind was a quote from Antoine de Saint-Exupéry:

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away."

So how about we adopt the term "Exupérian" for any change which improves something by making it smaller or simpler? The commit that removes 600 lines of unnecessary configuration. The copy-edit that turns fifteen thousand words of unstructured waffle into ten thousand words of focused, elegant writing. Maybe even that one weekend you spent going through all the clutter in your garage and finally getting rid of your unwanted lamps and old VHS tapes.

Saint-Exupéry was talking about designing aircraft, but I think the principle is equally applicable to software, to writing, to music, to architecture – in fact, to just about any creative process. I was submitting papers to a couple of conferences last week, and discovered that Øredev has a 1,000-character limit for session descriptions. Turns out my session descriptions all end up around 2,000-3,000 characters, and editing those down to 1,000 characters is really hard. But - it made them better. You look at every single word, you think 'does it still work if I remove this?', and it's surprising how often the answer is 'yes'.

Go on, give it a try. Do something #exuperian today. Edit that email before you send it. Remove those two classes that you're sure aren't used any more but you're scared to delete in case they break something. Throw out the dead batteries and expired coupons from your desk drawer. Remove a pointless feature nobody really wants.

Maybe you even have an EU cookie banner you can get rid of? :)

How to really break the internet.

2016-03-23T12:08:00+00:00

Yesterday someone broke the internet, and this time it had nothing to with Kim Kardashian. According to The Register and various other sources, developer Azer Koçulu removed over 250 of his own modules from NPM, the package management platform used by almost all open-source JavaScript projects. Among the projects he removed was left-pad, a tiny library that performs string padding. Turns out that literally thousand of open-source projects depend on left-pad – over 2 million downloads last month – and with left-pad removed from NPM, suddenly none of these projects would build any more. Oops.

It's worth remembering this is JavaScript, and one of the many things that makes JavaScript so entertaining is that it doesn't have a standard runtime library. In every other language I have ever used, string padding is either built-in, or it's part of a standard runtime that's available locally on every workstation and build server. But not in JavaScript – if you want to pad a string in JS, you either write your own function, you copy & paste one from StackOverflow, or you import a package to do it for you, and based on the fallout from yesterday it looks like a lot of people went for the package option.

Now, package management and dependency management is hard. Projects like NPM and RubyGems and NuGet appear to have made it a lot easier, but in most cases what they're actually doing is asking us to relinquish control of elements of our own projects in exchange for convenience. And they are convenient - as long as you're online, and everything's working, and everybody's being friendly, it's great – but I don't think we, as an industry, have done nearly enough to understand what happens if those assumptions turn out to be false.

Say you're about to get on a long-haul flight; just before they call your gate, you git clone a couple of projects onto your laptop and then board the plane. Would you be able to build the project once in flight? I suspect in many cases the answer is "no", because standard practice these days is to download all the required libraries as the first stage of the build process – and that means you can't build your project without being online.

Ok, that one's easy. This time round you shrug and watch the inflight movies instead, and next time you remember to do a full build before you go offline. Not a big deal. But what would happen if nuget.org was offline? Online services go dark for all sorts of reasons - infrastructure problems, legal action, DoS attacks. Most of the time, they come back, but not always – in 2014, Code Spaces was completely wiped out by an attacker who gained access to their AWS account. Does your build pipeline quietly rely on the fact that nuget.org isn't going to go away? And if it did, how long would it take to get things back up and running?

View post on imgur.com

But NuGet or NPM going down isn't even the worst case scenario. According to NuGet.org, the most download packages over the last six weeks are:

NewtonSoft.Json (2.4M downloads)
jQuery (1.2M downloads)
Microsoft.AspNet.Mvc (993K downloads)
Microsoft.AspNet.Razor (987K downloads)
EntityFramework (974K downloads)

Now, let's imagine a Nefarious Super Villain breaks into James Newton-King's house one night and forces him at gunpoint to deploy a new point release of NewtonSoft.Json. A release that works perfectly, only every time you call JsonConvert.Serialize(), it sneakily posts a copy of the JSON output to an IP address in Russia. How long before you ended up running this malicious code on your own production systems? How long before you noticed something was amiss – assuming you ever noticed at all? OK, somebody would notice eventually – that's the beauty of open-source, after all – but what about closed-source libraries? If you're using EntityFramework, I'd wager good money that you're using a single set of database credentials that has read/write/delete access to all your data – and trusting that the code isn't going to do anything unpleasant.

EDIT: Demis Bellot pointed out on Twitter after I first posted this article that source code is just one of many attack vectors that exploit our faith in package repositories. Unless you're comparing checksums or building your own reference binaries, you're blindly trusting that the source you're reading on Github is the same source that was used to built the binaries that are now running on your production servers.

There's ways around some of this. At the very least, cache the packages used by your recent builds in case you need to run them again. Most package repositories run over HTTP and use stable URLs, so all you really need is a caching proxy between your build pipeline and your package repo servers. Here at Spotlight, we run a NuGet server on our LAN that hosts our own packages, and we've also set up TeamCity so that following a successful build, all the .nupkg files (including the dependencies used by the build) are published to our local NuGet server. We actually started doing this by mistake – we set up a build to publish our own .nupkg files to our server and then realised afterwards it had also picked up all the package dependencies – but it actually works pretty well, and it means that if nuget.org was to go dark for a while, we could still build & deploy software as long as we didn't update any package dependencies.

As for package dependencies as an attack vector for malicious code? That one's a lot harder. We often do some ad-hoc traffic inspection as part of our review process – fire up the application with SQL Monitor and Fiddler running, take a look at the network traffic, database activity, debug logs and so on just to make sure we haven't done something stupid – but it's interesting to think how this could be turned into something more rigorous.

But as with so much else, it boils down to a choice – trust people, or do everything yourself. One is a calculated risk, one is a guaranteed expense, and it's managing that balance between risk and cost that makes IT – and business – so endlessly fascinating.

The Mysterious Case of the Latin General: Collations in SQL Server

2016-02-29T15:08:00+00:00

Computers deal in absolutes. True, false, yes, no. So when you ask a computer if two things are the same, it can’t get away with “well, sort of… it depends, really; why are you asking?”.

But we use computers to model reality, and reality is a horrible mess of edge cases, quirks and idiosyncracies. Today is Leap Year Day – February 29th. Which means you can do this:

var today = DateTime.Now.AddYears(1).AddYears(-1);
var yesterday = DateTime.Now.AddDays(-1);
if (today == yesterday) {
    Console.WriteLine("In your face, Einstein!");
}

And if you think that's messy, wait until you start looking at string comparisons. Which, by happy coincidence, was the thing on the top of my stack when I got into work this morning.

OK, pop quiz time! Are these statements true or false?

"foo" == "foo"
"FOO" == "foo"
"café" == "cafe"
"œnophile" == "oenophile"
"strasse" = "straße"
"SIGURÐAR" == "SIGURDAR"

Well? How did you do? See, the answer in almost every case is that lovely old chestnut "it depends". And when you're dealing with text columns in Microsoft SQL Server, the thing it depends on is called a collation. A collation is a set of rules that defines whether or not two strings should be considered to be equal, and what 'alphabetical order' means for ordering a given set of strings. For years, my team has run most of our databases systems with a collation called SQL_Latin1_General_CP1_CI_AI – and the important part there is the CI_AI at the end, which indicates that this is a case-insensitive (CI), accent-insensitive (AI) collation, which means that if you search our database for 'CAFE', you'll find records containing 'café', because the case-insensitive rule says that "C" and "c" are the same, and the accent-insensitive part says "ignore the accent – pretend that é is the same as e"

Now, in recent versions of SQL Server, Microsoft has introduced a new set of collations to choose from – which means that even though we've already decided we want a Latin alphabet, with case-insensitive and accent-insensitive string comparisons, we still have a choice of six:

SELECT * FROM fn_helpcollations() WHERE name LIKE '%Latin1_General%CI_AI'

Latin1_General_CI_AI	Latin1-General, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Latin1_General_100_CI_AI	Latin1-General-100, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
SQL_Latin1_General_CP1_CI_AI	Latin1-General, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive for Unicode Data, SQL Server Sort Order 54 on Code Page 1252 for non-Unicode Data
SQL_Latin1_General_CP1253_CI_AI	Latin1-General, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive for Unicode Data, SQL Server Sort Order 124 on Code Page 1253 for non-Unicode Data
SQL_Latin1_General_CP437_CI_AI	Latin1-General, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive for Unicode Data, SQL Server Sort Order 34 on Code Page 437 for non-Unicode Data
SQL_Latin1_General_CP850_CI_AI	Latin1-General, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive for Unicode Data, SQL Server Sort Order 44 on Code Page 850 for non-Unicode Data

What I'm specifically interested in here – because it's my job as architect to decide this kind of thing - is why we might choose to use the new Latin1_General_CI_AI collation instead of the tried'n'tested SQL_Latin1_General_CP1_CI_AI collation. So I did a little Googling, and dug up this excellent article, which explains a bit about how these collations differ when it comes to Unicode character expansions. See, there are quite a lot of 'letters' which can be written as either one or two characters, depending on your cultural conventions (and what kind of keyboard you've got). Typesetting purists will refer to archæologists and œnophiles, whilst the rest of us have to settle for archaeologists and oenophiles because we don't know how to type "æ" and "œ". And then there's the German "sharp S", which is written as either 'ß' or 'ss' depending on who's writing it.

Now here's the real-world scenario where this matters. Let's say we've got a database of rock music clubs, and someone's added Roadrunner's Paradise in Berlin, and – because they're German, typing in German on a German keyboard – they're put the address in as "Saarbrücker Straße 24, 10405 Berlin". Now one of our British users is on holiday, and they're in Berlin, and they're trying to find out if there are any rock clubs on Saarbrücker Straße. Except because they've got an English guidebook and an English keyboard, they search for "saarbrucker strasse", because they don't know how to type the umlaut or the sharp-S 'ß' character.

Whether their search works or not can potentially depend on several things

The character type (varchar or nvarchar) of our StreetAddress column
The collation of our StreetAddress column
The character type specified in the query – SQL will assume string literals are ANSI strings unless you prefix them with N, so LIKE 'saarbrucker strasse' is different to LIKE N'saarbrucker strasse'
Whether or not we explicitly specify a collation in our WHERE clause.

OK, so we have two different column types (varchar vs nvarchar), we have two different collations (Latin1 vs SQL_Latin1), we have two different ways of specifying our search text (N'foo' vs 'foo'), and we have three different ways to specify collation in the query (explicit Latin1, explicit SQL_Latin1, or implicit) - meaning there are 2*2*2*3 = 24 different ways to store and query the data supporting this particular scenario.

So let's try them all. For SCIENCE. Code to create and run these examples is at https://gist.github.com/dylanbeattie/c9705dc5c4efd82f60b8, and when you run it, it works like this:

Column Type	Colum collation	Search text	Query collation	Did it work?
nvarchar	Latin1	'%saarbrucker strasse%'	(implicit)	Yes
nvarchar	Latin1	'%saarbrucker strasse%'	Latin1	Yes
nvarchar	Latin1	'%saarbrucker strasse%'	SQL_Latin1	Yes
nvarchar	Latin1	N'%saarbrucker strasse%'	(implicit)	Yes
nvarchar	Latin1	N'%saarbrucker strasse%'	Latin1	Yes
nvarchar	Latin1	N'%saarbrucker strasse%'	SQL_Latin1	Yes
nvarchar	SQL_Latin1	'%saarbrucker strasse%'	(implicit)	Yes
nvarchar	SQL_Latin1	'%saarbrucker strasse%'	Latin1	Yes
nvarchar	SQL_Latin1	'%saarbrucker strasse%'	SQL_Latin1	Yes
nvarchar	SQL_Latin1	N'%saarbrucker strasse%'	(implicit)	Yes
nvarchar	SQL_Latin1	N'%saarbrucker strasse%'	Latin1	Yes
nvarchar	SQL_Latin1	N'%saarbrucker strasse%'	SQL_Latin1	Yes
varchar	Latin1	'%saarbrucker strasse%'	(implicit)	Yes
varchar	Latin1	'%saarbrucker strasse%'	Latin1	Yes
varchar	Latin1	'%saarbrucker strasse%'	SQL_Latin1	No
varchar	Latin1	N'%saarbrucker strasse%'	(implicit)	Yes
varchar	Latin1	N'%saarbrucker strasse%'	Latin1	Yes
varchar	Latin1	N'%saarbrucker strasse%'	SQL_Latin1	Yes
varchar	SQL_Latin1	'%saarbrucker strasse%'	(implicit)	No
varchar	SQL_Latin1	'%saarbrucker strasse%'	Latin1	Yes
varchar	SQL_Latin1	'%saarbrucker strasse%'	SQL_Latin1	No
varchar	SQL_Latin1	N'%saarbrucker strasse%'	(implicit)	Yes
varchar	SQL_Latin1	N'%saarbrucker strasse%'	Latin1	Yes
varchar	SQL_Latin1	N'%saarbrucker strasse%'	SQL_Latin1	Yes

So there's the difference. If you're dealing with varchar columns (as opposed to nvarchars), and you specify your query text as an ANSI string (as opposed to a Unicode string), then you need to use the Latin1_General_CI_AI collation if you want "strasse" to match "straße"; in all other scenarios, 'ss' is equal to 'ß' (and apparently this is applies to all Latin collations regardless of accent sensitivity).

Of course, this works both ways – by using Latin1_General_CI_AI, you create a scenario where somebody searching for '%aße%' will find English words like 'assessment' and 'molasses', because as far as SQL Server is concerned, 'aße' and 'asse' are the same string. If that's a problem for your particular scenario, you'll need to do something like restrict your search inputs to the basic ASCII character set.

As with almost everything in software development, modelling the real world involves making compromises. It's our job to make sure we're aware of those compromises, and that we understand how they translate back into real-world behaviour, and it's up to us to explain them to users who think our software sucks because it doesn't know what day it is and it says there's no rock bars in Saarbrucker Strasse.

Happy Leap Year Day!

Better Hypermedia Through Obfuscation

2016-02-23T12:59:00+00:00

Here’s a fun and completely twisted idea that came to me on the train this morning. One of the constraints of ReSTful architectures is that they use hypermedia as the engine of application state – in other words, clients shouldn’t maintain their own information about where they “are” and what resources and actions are available to them right now; they should rely on hypermedia embedded in the current resource representation they’ve retrieved from the server. A common example of this is pagination – navigating big lists of resources, using a representation something like this example:



  GET /people

  "_embedded": {
    "people": [ /* array of 10 people */ ]
  },
  "_links": {
    "first": { "href": "/people?page=0" },
    "last": { "href": "/people?page=12" },
    "next": { "href": "/people?page=1" },
    "self": { "href": "/people?page=0" }
  },
  "count": 10,
  "total": 115
}

Now, with an API like this, it’s all too easy for the client – or rather, the person building the client – to go “ok, page number is a zero-based integer; let’s cut a few corners here” and just program something like

for(var i = 0; i < 12; i++) {
    http.get(“https://api.foo.com/people?page=”+page);
}

Now, I’m a big fan of something we call the "pit of success" – the idea being that we "build platforms such that […] developers just fall into doing the 'right thing'", and more generally, the idea that the easiest way to achieve something is also the ‘correct’ way to achieve that thing. So what if we intentionally obfuscate our APIs so that hypermedia navigation becomes easier than building a for() loop? By, for example, requiring that our page number is written out longform, instead of numerically? And, just for fun, we’ll require that it’s in a language that isn’t in the regular ASCII character set. Like Ukrainian:

{
    "first": { "href": "/people?page=один" },
    "last": { "href": "/people?page=дванадцять" },
    "prev": { "href": "/people?page=два" },
    "next": { "href": "/people?page=чотири" },
    "self": { "href": "/people?page=три" }
}

Suddenly, your ‘shortcut’ isn’t a short cut any more. For starters, you’ll probably need to install a special keyboard in order to type those characters – not to mention suddenly your source code files will need to be in UTF8, and I’ll wager that somewhere in your build pipeline there’s a tool that can’t handle UTF8 source code. And you’ll need a routine which can translate integers into Ukrainian string representations… No, the easiest thing to do now is to retrieve the first resource, parse the next.href property, retrieve that resource, and so on until you hit a resource with no next link. Which, of course, is exactly how hypermedia is supposed to work in the first place.

“How Can Software Be So Hard?”

2016-02-10T12:47:00+00:00

Last night, I went to a Gresham College lecture at the Museum of London, “How Can Software Be So Hard?” presented by Professor Martyn Thomas CBE. The lecture itself was great – good content supported by solid examples. I must say there wasn’t a great deal there that I haven’t heard before – software engineering is an immature discipline; we rely too heavily on “testing” to validate and verify the systems we create; validation normally happens far too late to do anything about the problems it uncover; we’re overly reliant on modules and components that we can’t actually trust to work properly… all interesting and valid observations, but nothing particularly revolutionary.

Personally, I think the question posed in the lecture title is, to some extent, built on a false premise. Towards the end, he makes the observation – whilst talking about teenagers getting rich by writing iPhone games – that we only focus on the success stories, and the countless failures don’t get any attention. Which is true – but I think with software, we routinely take so much success for granted that it’s the failures which stand out. Sure, they happen – but if you think software is hard, try building a mechanical or an analog system that can send high-fidelity music from Boston to Singapore, or show the same colour photograph to a million people five minutes after it was taken. So many software projects are instigated to pursue efficiencies – to take some business process or system that used to require tens of thousands of people, and hire a few dozen programmers to replace it with a system that basically runs itself; it is any wonder it doesn’t always go smoothly? There’s a lot of things which are easy in software and close to impossible in any other domain, and I think to lose sight of that would be disadvantageous in an age where we’re trying to inspire more kids to study STEM subjects and pursue careers in science and engineering.

The thing that really stuck in my head, though, was a comment Professor Thomas made in response to a question after the lecture. Somebody asked him what he thought about self-driving cars, and amongst the points he raised in response, he said something like:

What about pedestrians? Why would you find a crossing, press the button and wait for the green man if you know all the cars on the road are programmed not to run you over?

Over the last few years, I’ve spent a lot of time studying the way smart devices are affecting the way we interact with the world around us – something I’ve covered at length in my talk “Are Smart Systems Making Us Stupid?”, which I presented at BuildStuff last year. I’ve looked into all sorts of models of human/machine interaction, but I’d never considered that particular angle before – and it’s fascinating.

Our basic instinct for self-preservation is reinforced from an early age – is there anybody here who DOESN’T remember being taught how to cross the street? So what happens to those behaviour patterns in a world where kids work out pretty early on that they can jump in front of Google Cars and get away with it? Do we program the cars to hit a kid once in a while – not to kill them, just give ‘em a nasty bump to teach them a lesson? How much use is a self-driving car when your journey takes longer than walking because your car slams the brakes on every time a pedestrian walks in front of it? Maybe you’re better off dumping your luggage, or your shopping, in the car, and taking the train instead?

It’s also an interesting perspective on a discussion that, until now, has been framed very much from the perspective of the driver – “will my self-driving car kill me to save a dozen schoolkids?” – and raises even more questions around the social implications of technology like driverless cars.

The next talk in the series is “Computers, People and the Real World” on April 5th, and if you’re interested in how our increasing dependency on machines and software is affecting the world we live in and the lives we live, I’d heartily recommend it.

(PS: if anyone from Gresham College is reading this – get somebody to introduce your speakers. They’ve worked hard to prepare the material they’re presenting; help them make the best possible first impression by taking the stage to a round of applause from a crowd who already know who they are. It’s not that hard, and it makes a huge difference.)

Let’s Talk About Feedback

2016-01-27T13:00:00+00:00

There’s been some really interesting blog posts about speaking at conferences recently – from Todd Motto’s “So you want to talk at conferences?” piece, to Basia Fusinska’s “Conference talks, where did we go wrong?” and Pieter Hintjens “Ten Steps to Better Public Speaking”, to Seb’s recent “What’s good feedback from a talk?” post. There’s a lot of very useful general advice in those posts, but I absolutely believe the best way to improve as a speaker is to ask your audience what you could be doing better, and that’s hard.

After most talks you’ll have a couple of people come up to you at the coffee machine or in the bar and say “hey, great talk!” – and that sort of positive feedback is great for your confidence but it doesn’t actually give you much scope to improve unless you take the initiative. Don’t turn it into an interrogation, but it’s easy to spin this into conversation – “hey, really glad you enjoyed it. Anything you think I could have done differently? Could you read the slides?” If there’s something in your talk which you’re not sure about, this is a great chance to get an anecdotal second opinion from somebody who actually saw it.

Twitter is another great way to get anecdotal feedback. Include your Twitter handle and hashtags in your slides – at the beginning AND at the end – and invite people to tweet at you if they have any comments or questions. In my experience, Twitter feedback tends to be positive (or neutral – I’ve been mentioned in lots of “@dylanbeattie talking about yada yada yada at #conference”-type tweets) – and again, whilst it’s good for giving your confidence a boost, it can also be a great opportunity to engage with your audience and try to get some more detailed feedback.

And then there’s the official feedback loops – the colored cards, the voting systems and the feedback app. I really like how BuildStuff does this. They gather feedback on each talk through a combination of coloured-card voting and an online feedback app. Attendees who give feedback online go into a prize draw, which is a nice incentive to do so – and it makes it an easy sell for the speakers: “Thank you – and please remember to leave me feedback; you might win an iPad!” The other great thing BuildStuff does is to send you your feedback as a JPEG, which looks good and makes it really easy for speakers to share feedback and swap notes afterwards. Here’s mine from Vilnius and Kyiv last year:

Now, I’m pretty sure there were more than five people in my talk in Kyiv – so I think something might have got lost in the transcription here – but the real value here is in the anecdotal comments.

Some conferences also run a live “leaderboard”, showing which speakers are getting the highest ratings. I’m generally not a big fan of this – I think it perpetuates a culture of celebrity that runs contrary to the approachability and openness of most conference speakers – but if you are going to do it, then make sure it works. Don’t run a live leaderboard that excludes all the speakers from Room 7 because the voting machine in Room 7 was broken.

Finally, two pieces of feedback I had from my talk about ReST at NDC London this year. The official talk rating, which I’m quite happy with but doesn’t really offer any scope for improvement:

And then there’s this, thanks to Seb, who not only came along to my talk but sat in the front row scribbling away on his iPad Pro and then sent me his notes afterwards. There’s some real substance here, some good points to follow up on and things I know I could improve:

This also goes to highlight one of the pitfalls of doing in-depth technical talks – your audience probably aren’t in a position to judge whether there’s flaws in your content or not, so your feedback is more likely to reflect the quality of your slides and your presentation style than the substance of your content. In other words – just because you got 45 green tickets doesn’t mean you can’t improve. Find a subject matter expert and ask if they can watch your talk and give you notes on it. Share your slides and talks online and ask the wider community for their feedback. And don’t get lazy. Even if you’ve given a talk a dozen times before, we’re in a constantly-changing industry and every audience is different.

“The Face of Things to Come” from PubConf

2016-01-24T17:58:00+00:00

A version of my talk from PubConf London, “The Face of Things to Come”, is now online. This isn’t a recording of the actual talk – the audio has been recorded specially, one slide has been replaced for copyright reasons, and a couple of things have been fixed in the edit – but it’s close enough.

The Face of Things to Come from Dylan Beattie on Vimeo.

As always, the only way to improve as a speaker is to listen to your audience, so I would love to hear your comments or feedback – leave a comment, email me or ping me on Twitter.

Would you like to speak at London .NET User Group in 2016?

2016-01-19T18:42:00+00:00

The London .NET User Group, aka LDNUG – founded and run by Ian Cooper, with help from Liam Westley, Toby Henderson and me – is now accepting speaker submissions for 2016.

We aim to run at least one meetup a month during 2016, with at least two speakers at each meetup. Meetups are always on weekday evenings in central London, are free, and we want to have at least two speakers at each of our meetups this year. We’re particularly keen to welcome some new faces and new ideas to the London .NET community, so if you’ve ever been at a talk or a conference and thought “hey – maybe I could do that!” – this is your chance.

We’re going to try and introduce some variation on the format this year, so we’re inviting submissions for 45-minute talks, 15-minute short talks and 5-minute lightning talks, on any topic that’s associated with .NET, software development and the developer community. Come along and tell us about your cool new open source library, or that really big project your team’s just shipped. Tell us something we didn’t know about asynchronous programming, or distributed systems architecture. We welcome submissions from subject matter experts but we’re also keen to hear your first-hand stories and experience. Never mind what the documentation said – what worked for you and your team? Why did it work? What would you do differently?

If you’re a new speaker and you’d like some help and support, let us know. We’d be happy to discuss your ideas for potential talks, help you write your summary, rehearse your talk, improve your slide deck. Drop me an email, ping me on Twitter or come and find me after the next meetup (I’m the one in the hat!) and I’ll be happy to help.

So what are you waiting for? Send us your ideas, come along to our meetups, and let’s make 2016 a great year for London.NET.

Yes! I want to speak at London.NET in 2016!

Conway’s Law and the Mythical 17:00 Split

2016-01-19T15:06:00+00:00

I was at PubConf on Saturday. It was an absolutely terrific event – fast-paced, irreverent, thought-provoking and hugely enjoyable. Huge thanks to Todd Gardner for making it happen, to Liam Westley for advanced detail-wrangling, and to NDC London, Track:js, Red Gate and Zopa for their generous sponsorship.

Seb delivered a great talk about the Mythical 17:00 Split, which he has now written up on his blog. His talk resonated a lot with me, because I also find a lot of things about workplace culture very strange. I’m lucky enough to work somewhere where I seldom encounter these issues directly, but I know people whose managers genuinely believe that the best way to write software is to do it wearing a suit and tie, at eight’o’clock in the morning, whilst sitting in a crowded office full of hedge fund traders.

But Seb’s write-up post said something that really struck a chord.

“Take working in teams. The best teams are made of people that like working together, and the worst teams I’ve had was when a developer had specific issues with me, to the point of causing a lot of tension”

Now, I’m a big fan of Conway’s Law – over the course of my career, I’ve seen (and built) dozens of software systems that turned out to reflect the communication structures of the organizations that created them. I’ve even given a conference talk at BuildStuff 2015 about Conway’s Law with Mel Conway in the audience – which was great fun, if a little nerve-wracking.

In a nutshell, Conway’s Law says of Seb’s observation regarding teams that if you take a bunch of people who are fundamentally incompatible, and force them to work together, you’ll end up with a system which is a bunch of incompatible components that are being forced to work together. If you want to know whether – and how – your systems are going to fail in production, look at the team dynamic of the people who are building them. If watching those people leaves you feeling calm, reassured and relaxed, then you’re gonna end up with something good. If one person is absolutely dominating the conversations, one component is going to end up dominating the architecture. If there’s two people on the team who won’t speak to each other and end up mediating all their communication through somebody else – guess what? You’re going to end up with a broker in your system that has to manage communication between two components that won’t communicate directly.

If your team hate each other, your product will suck – and in the entire history of humankind, there are only two documented exceptions to this rule. One is Guns’n’Roses “Appetite for Destruction” and the other is “Rumours” by Fleetwood Mac.

\m/

The Rest of ReST at NDC London

2016-01-14T17:58:00+00:00

A huge thank you to everyone who came along to my ReST talk here at NDC London. Links to a couple of resources you might find useful:

The slide deck from my presentation is on Github (PDF)
Roy Fielding’s dissertation, ‘Architectural Styles and the Design of Network-based Software Architectures’, in which he outlines the constraints of representational state transfer (ReST). Highly recommended and remarkably readable.
The HAL – Hypertext Application Language site and draft specification
A great article from Stormpath on linking and resource expansion in REST APIs.
The OAuth2.0 site and specification

Thanks again for coming – and any comments, questions or feedback, you’ll find me on Twitter as @dylanbeattie.

Confession Time. I Implemented the EU Cookie Banner

2016-01-07T17:47:00+00:00

Troy Hunt kicked off 2016 with a great post about poor user experiences online – a catalogue of common UX antipatterns that “make online life much more painful than it needs to be”.

One of the things he picks up on is EU Cookie Warnings – “this is just plain stupid.” And yeah, it is. Absolutely everybody I know who added an EU cookie warning to their website agrees – this is just plain stupid. But for folks outside the European Union, it might be insightful to learn just why these things started appearing all over the place.

First, a VERY brief primer on how the European Union works. There’s currently 28 countries in the EU. The United Kingdom, where I live and work, is one of them. One of the aims of the EU is to create a consistent legal framework that covers all citizens of all its member states. Overseeing all this is the European Parliament. They make laws. It’s then up to the governments of the individual member states to interpret and enforce those laws within their own countries.

So, in 2009, the European Parliament issued a directive called 2009/136/EC – OpenRightsGroup has some good coverage of this. The kicker here is Article 5(3), which says

“The storing of information or the gaining of access to information already stored in the user’s equipment is only allowed on the condition that the subscriber or user concerned has given their consent, having been provided with clear and comprehensive information in accordance with Directive 95/46/EC, inter alia, about the purposes of the processing. This shall not prevent any technical storage or access for the sole purpose of carrying out the transmission of a communication over an electronic communications network, or as strictly necessary in order for the provider of an information society service explicitly requested by the subscriber or user to provide the service.”

In a nutshell, this means you can’t store anything (such as a cookie) on a user’s device, unless

You’ve told them what you’re doing and they’ve given their explicit consent, OR
It’s absolutely necessary to provide the service they’ve asked for.

Directive 2009/136 goes on to state (my emphasis):

“Under the added Article 15a, Member States are obliged to laydown rules on penalties, including criminal sanctions where applicable to infringements of the national provisions, which have been adopted to implement this Directive. The Member States shall also take “all measures necessary” to ensure that these are implemented. The new article further states that “the penalties provided for must be effective, proportionate and dissuasive and may be applied to cover the period of any breach, even where the breach has subsequently been rectified”.

Golly! Criminal sanctions? Retrospectively applied, even for something that we already fixed? That sounds pretty ominous.

Anyway. Here’s what happens next. Directive 2009/136 means that is is now THE LAW that you don’t store cookies without consent, and the various member states swing into action and try to work out what this means and how to enforce it. In the UK, Parliament interpreted this via something called the Privacy and Electronic Communications (EC Directive) (Amendment) Regulations 2011, which would come into effect in 2012.

My team and I found out in late 2011 that, when the new regulations came into force on 26 May 2012, we would be breaking the law if we put cookies on our user’s machines without their explicit consent. And nobody had the faintest idea what that actually meant, because nobody had ever broken this law yet, so nobody knew what the penalties for non-compliance would be. The arm of the UK government that deals with this kind of thing is the Information Commissioner’s Office (ICO), who have a reputation for taking data protection very seriously, and the power to exact fines up to £500,000 for non-compliance. The ICO also usually publish quite clear and reasonable guidelines on how to comply with various elements of the law – but that takes time, so in late 2011 we found ourselves with a tangle of bureacracy, a hard deadline, the possibility of severe penalties, and absolutely no guidance to work from.

So… we implemented it. Despite it being a pointless, stupid, ridiculous endeavour that would waste our time and piss off our users, we did it - because we didn’t want to end up in court and nobody could assure us that we wouldn’t.

We built a nice self-contained JavaScript library to handle displaying the banner across our various sites and pages.

Instead of just plastering something on every page saying “We use cookies. Deal with it”, the approach taken by most sites - we actually split our cookies into the essential ones required to make our site work, and the non-essential ones used by Boomerang, Google Analytics and other stats and analytics tools. And we allowed users to opt-out of the non-essential ones. We went live with this on 10th May 2012. Around 30% of our users chose to opt-out of non-essential cookies – meaning they became invisible to Google Analytics and our other tracking software. Here’s our web traffic graph for April – June 2012 – see how the peaks after May 10th are suddenly a lot lower?

On 25th May 2012, ONE DAY before the new regulations became law, the ICO issued some new guidance, which significantly relaxed the requirements around ‘consent’. “Implied consent” was suddenly OK – i.e. if your users hadn’t disabled cookies in their browser, you could interpret that as meaning they had consented to receive cookies from your site.

They also announced that any enforcement would be in response to user complaints about a specific site:

“The end of the safe period "doesn't mean the ICO is going to launch a torrent of enforcement action" said the deputy commissioner and it would take serious breaches of data protection that caused "significant distress" to attract the maximum £0.5m non-compliance fine.” (via The Register)

So there you have it. Go to http://www.spotlight.com/ and, just once, you’ll see a nice friendly banner asking if you mind us tracking your session using cookies. And if you opt out, that’s absolutely fine – our site still works and you won’t show up in any of our analytics. Couple of weeks of effort, a nice, clean, technically sound implementation… did it make the slightest bit of difference? Nah. Except now we multiply all our Analytics numbers by 1.5. And yes, we periodically review the latest guidance to see whether the EU has finally admitted the whole thing was a bit silly and maybe isn’t actually helping, but so far nada – and in the absence of any hard evidence to the contrary, it’s hard to make a business case for doing work that would make us technically non-compliant, even if the odds of any enforcement action are minimal.

Now, if the European Parliament really wanted to make the internet a better place, how about they read Troy’s post and ban popover adverts, unnecessary pagination, linkbait headlines and restrictions on passwords? Now that’s the kind of legislation I could really get behind.

Restival Part 6: Who Am I, Revisited

2016-01-07T10:09:00+00:00

Note: code for this instalment is in https://github.com/dylanbeattie/Restival/tree/v0.0.6

In the last instalment, we looked at adding HTTP Basic authentication to a simple HTTP endpoint - GET /whoami - that returns information about the authenticated user.

Well... I didn't like it. Both the OpenRasta and the WebAPI implementations felt really over-engineered, so I kept digging and discovered a few things that made everything much, much cleaner.

Basic auth in OpenRasta - Take 2

There's an HTTP Basic authentication feature baked into OpenRasta 2.5.x, but all of the classes are marked as deprecated so in my first implementation I avoided using it. After talking with Seb, the creator of OpenRasta, I understand a lot more about the rationale behind deprecating these classes - they're earmarked for migration into a standalone module, not for outright deletion, and they'll definitely remain part of the OpenRasta 2.x codebase for the foreseeable future.

Armed with that knowledge, and the magical compiler directive #pragma warning disable 618 that stops Visual Studio complaining about you using deprecated code, I switched Restival back to running on the OpenRasta NuGet package instead of my forked build, and reimplemented the authentication feature - and yes, it's much, much nicer.

There's a RestivalAuthenticator which implements OpenRasta's IBasicAuthenticator interface - as with the other frameworks, this ends up being a really simple wrapper around the IDataStore:

public class RestivalAuthenticator : IBasicAuthenticator {
private readonly IDataStore db;

public RestivalAuthenticator(IDataStore db) {
    this.db = db;
}

public AuthenticationResult Authenticate(BasicAuthRequestHeader header) {
    var user = db.FindUserByUsername(header.Username);
    if (user != null && user.Password == header.Password) return (new AuthenticationResult.Success(user.Username, new string[] { }));
    return (new AuthenticationResult.Failed());
}

public string Realm { get { return ("Restival.OpenRasta"); } }
}

and then there's the configuration code to initialise the authentication provider.

ResourceSpace.Uses.PipelineContributor<AuthenticationContributor>();
ResourceSpace.Uses.PipelineContributor<AuthenticationChallengerContributor>();
ResourceSpace.Uses.CustomDependency<IAuthenticationScheme, BasicAuthenticationScheme>(DependencyLifetime.Singleton);
ResourceSpace.Uses.CustomDependency<IBasicAuthenticator, RestivalAuthenticator>(DependencyLifetime.Transient);

This one stumped me for a while, until I realised that - unlike, say, Nancy, which just does everything by magic, you need to explicitly register both the AuthenticationContributor and the AuthenticationChallengerContributor. These are the OpenRasta components that handle the HTTP header parsing, decoding and the WWW-Authenticate challenge response, but if you don't explicitly wire them into your pipeline, your custom auth classes will never get called.

Basic auth in WebAPI - Take 2

As part of the last instalment, I wired up the LightInject IoC container to my WebAPI implementation. I love LightInject, but something I had never previously realised is that LightInject can do property injection on your custom attributes. This is a game-changer, because previously I'd been following a pattern of using purely decorative attributes - i.e. with no behaviour - and then implementing a separate ActionFilter that would check for the presence of the corresponding attribute before running some custom code - and all this just so I could inject dependencies into my filter instances.

Well, with LightInject up and running, you don't need to do any of that - you can just put a public IService { get; set; } onto your MyCustomAttribute class, and LightInject will resolve IService at runtime and inject an instance of MyAwesomeService : IService into your attribute code. Which means the half-a-dozen classes worth of custom filters and responses from the last implementation can be ripped out in favour of a single RequireHttpBasicAuthorizationAttribute - which overrides WebAPI's built-in AuthorizeAttribute class to provide authorization header parsing, WWW-Authenticate challenge response, and hook the authentication up to our IDataStore interface.

I'm much happier now with all four implementations, so it raises the interesting question of how much development time is really worth. Based on the code provided here, I suspect a good developer could implement HTTP Basic auth on any of these frameworks in about fifteen minutes - but something that takes fifteen minutes to implement doesn't really count if it takes you two days to work out how to do that fifteen-minute implementation.

In forthcoming instalments, we're going to be adding hypermedia, HAL+JSON and resource expansion - as we move further away from basic HTTP capabilities and into more advanced REST/serialization/content negotiation, it'll be interesting to see how our four frameworks measure up.

Restival Part 5: Who Am I?

2016-01-07T10:08:00+00:00

NOTE: Code for this article is at https://github.com/dylanbeattie/Restival/tree/v0.0.5

UPDATE: The WebAPI and OpenRasta implementations described here are... inelegant. After this release, I spent a little more time and came up with something much cleaner - which you can read all about in the follow-up post http://dylanbeattie.blogspot.co.uk/2015/12/restival-part-6-who-am-i-revisited.html

Welcome back. It's been a very busy summer that's turned into a very busy autumn - including a fantastic couple of weeks in Eastern Europe, speaking at BuildStuff in Lithuania and Ukraine, where I met a lot of very interesting people, discussed a lot of very interesting ideas, tried a lot of very interesting food, and generally got a nice hard kick out of my comfort zone. Which is good.

Anyway. As London starts getting frosty and festive, it's time to pick up where we left off earlier in the year with my ongoing Restival project - implementing the same API in four different .NET API frameworks (ServiceStack, NancyFX, OpenRasta and WebAPI).

In this instalment, we're going to add two very straightforward capabilities to our API:

HTTP Basic authentication. Clients can include a username/password in an HTTP Authorization header, and the API will verify their credentials against a simple user store. (If you're doing this in production, you'd enforce HTTPS/TLS so that credentials can't be sniffed in transit, but since this is a demonstration project and I'm optimising for readability, TLS is out of scope for now.)
A /whoami endpoint, where authenticated users can GET details of their own user record.

And that's it. Sounds simple, right? OK, first things first - let's thrash out our requirements into something a little more detailed:

GET /whoami with valid credentials should return the current users' details (id, name, username)

But remember - we're building an authentication system, so we should also be testing the negative responses:

GET /whoami with invalid credentials returns 401 Unauthorized
- "invalid" means an unsupported scheme, an unsupported header format, invalid credential encoding, or a correctly formatted header containing an unrecognised username and/or password.
GET /whoami without any credentials returns a 401 Unauthorized
GET /whoami without any credentials returns a WWW-Authenticate header indicating that we support HTTP Basic authentication.

The WWW-Authenticate thing is partly a node to HATEOAS, and partly there to make it easier to test things from a normal web browser. I tend to use Postman for building and testing any serious HTTP services, but when it comes to quick'n'dirty API discovery and troubleshooting, I can't overstate the convenience of being able to paste something into a normal web browser and get a readable response.

The test code is in WhoAmITestBase.cs. Notice that in this implementation, our users are stored in a fake data store - FakeDataStore.cs - and we're actually using this class as a TestCaseSource in our tests so we maintain parity between the test data and the test coverage.

The WhoAmI endpoint was trivial - find the current user name, look them up in the user store, and convert their user data to a WhoAmIResponse. The fun part here was the authentication.

A Note on Authentication vs Authorization

NOTE: Authentication and authorization are, strictly speaking, different things. Authentication is "who are you?", authorization is "are you allowed to do this thing?" Someone trying to enter the United States with a Libyan passport is authenticated - the US border service know exactly who they are - but they're not authorized because Libyan citizens can't enter the US without a valid visa.

In one respect, HTTP gets this exactly right - a 401 Unauthorized means "we don't know who you are", a 403 Forbidden means "we know who you are, but you're not allowed to do this." In another respect, it gets this badly wrong, because the Authentication header in the HTTP specification is called Authorization.

Authentication - The Best Case Scenario

OK, to build our secure /whoami endpoint, we need a handful of extension points in our framework. We need to:

Validate a supplied username and password against our own credential store
Indicate that specific resources require authorization, so that unauthorized requests for those resources will be rejected
Determine the identity of the authenticated user when responding to an authorized request

The rest - WWW-Authenticate negotiation, decoding base64-encoded Authorization headers, returning 401 Unauthorized responses - is completely generic, so in an ideal world our framework will do all this for us; all we need to do is implement the three extension points above.

Let's look at Nancy first, because alphabetical is as good an order as any.

NancyFX

Implementing HTTP Basic auth in NancyFX proved fairly straightforward. The first head-scratching moment I hit is that - unlike the other frameworks in this project - Nancy is so lightweight that I didn't actually have any kind of bootstrapper or initialization code anywhere, so it wasn't immediately obvious where to configure the additional pipeline steps. A bit of Googling and poking around the NancyFX source led me to the Nancy.Demo.Authentication.Basic project, which made things a lot clearer. From this point, implementation involved:

Add an AuthenticationBootstrapper - which Just WorkedTM without any explicit registration. I'm guessing it's invoked by ~~magic~~ sufficiently advanced technology, because it overrides DefaultNancyBootstrapper.
Implement IUserValidator to connect Nancy to my custom user store - my implementation is just a really simple wrapper around my user data store. My UserValidator depends on my IDataStore interface - and, thanks to Nancy's auto-configuration, I didn't have to explicitly register either of these as dependencies.
In this bootstrapper, call pipelines.EnableBasicAuthentication() and pass in a basic authentication configuration.

protected override void ApplicationStartup(TinyIoCContainer container, IPipelines pipelines) {
    base.ApplicationStartup(container, pipelines);
    var authConfig = new BasicAuthenticationConfiguration(container.Resolve<IUserValidator>(), "Restival.Nancy");
    pipelines.EnableBasicAuthentication(authConfig);
}

Finally, the WhoAmIModule needs a call to this.RequiresAuthentication(), and that's it. Clean, straightforward, no real showstoppers.

UPDATE: The NancyFX documentation has now been updated with a detailed walkthrough on enabling HTTP Basic authentication

OpenRasta

Adding HTTP Basic auth to OpenRasta was a lot more involved - and the reasons why provide some interesting insight into the underlying development practises of the frameworks we're comparing. In 2013, there were some major changes to the request processing pipeline used by OpenRasta. As part of these changes, the basic authentication features of OpenRasta (dating back to 2008) were marked as deprecated - and until now, nobody's contributed an up-to-date implementation, I'm guessing because none of the people using OpenRasta have needed one. Which left me with a choice - do I use the deprecated approach, contribute my own implementation, or disqualify OpenRasta? Well, disqualification would be no fun, and deprecated code means you get Resharper yelling at you all over the place, so I ended up implementing a pipeline-based HTTP Basic authorization module to the OpenRasta codebase.

Whilst writing this article, I had a long and very enlightening chat with Sebastien Lambla, the creator of OpenRasta, about the current state of the project and specifically the authentication capabilities. It turns out the features marked as deprecated were intended to be migrated into a separate OpenRasta.Security module, thus decoupling authentication concerns from the core pipeline model - but this hasn't happened yet, and so the code is still in the OpenRasta core package. It's up to you, the implementer, whether to use the existing ('deprecated') HTTP authentication provider, or to roll your own based on the new pipeline approach.

The original pull request for this is #89, which is based on the Digest authentication example included in the OpenRasta core codebase - but shortly after it was accepted, I found a bug with the way both the Digest and my new Basic implementation handled access to non-secured resources. The fix for this is in #91, which at the time of writing is still being reviewed, so Restival's currently building against my fork of OpenRasta in order to get the authentication and WhoAmI tests passing properly.

Following those changes to the framework itself, the implementation was pretty easy.

Provide an implementation of IAuthenticationProvider based on my IDataStore interface.
Register the authentication provider and pipeline contributor in the Configuration class:

ResourceSpace.Uses.CustomDependency<IDataStore, FakeDataStore>(DependencyLifetime.Singleton);
ResourceSpace.Uses.CustomDependency<IAuthenticationProvider, AuthenticationProvider>(DependencyLifetime.Singleton);
ResourceSpace.Uses.PipelineContributor<BasicAuthorizerContributor>();
Decorate the WhoAmIHandler with [RequiresBasicAuthentication(realm)]

Total research and Google time took the best part of a day - including implementing the missing pieces of the framework. Implementation time once those were in place was around an hour, although I suspect even if the necessary authorization components already existed I'd have needed a couple of hours Google time to work out exactly how to plug into the pipeline.

ServiceStack

Service was really straightforward, mainly because the extension points for overriding built-in authentication behaviour are logical and clearly documented. Implementation involved creating my own AuthProvider that extends the built-in BasicAuthProvider, injecting an IDataStore into this auth provider, and then registering my provider with ServiceStack in AppHost.Configure().

The only gotcha I encountered here was that the first implementation would response with an HTTP redirect to a login page if the request wasn't authorized - but Googling "servicestack authentication redirecting to /login" brought up this StackOverflow post, which explained that (a) this only happens for HTML content-type requests (my fault for using a web browser to test my HTTP API, I guess!), and that you can disable it by specifying HtmlRedirect = null when you initialize the auth feature:

public class AppHost : AppHostBase {
public AppHost() : base("Restival", typeof(HelloService).Assembly) { }
public override void Configure(Container container) {
    var db = new FakeDataStore();
    container.Register<IDataStore>(c => db).ReusedWithin(ReuseScope.Container);
    var auth = new AuthFeature(() => new AuthUserSession(), new IAuthProvider[] { new RestivalAuthProvider(db) }) {
    HtmlRedirect = null
    };
    Plugins.Add(auth);
}
}

Total implementation time was about half an hour here - but bear in mind I've worked with ServiceStack a lot, so I'm familiar with the plugin architecture and the pipeline.

WebAPI

It took a surprisingly long time to come up with an HTTP Basic auth implementation on WebAPI. The first stumbling block here was the sheer number of posts, articles and examples demonstrating how to do it. There's a lot of excellent resources online about adding HTTP Basic authentication support to WebAPI, most of which are subtle variations on a common theme:

Rick Strahl's WestwindToolkit
Authentication Filters in ASP.NET Web API 2 on www.asp.net
Robert Muehsig's Using Basic Authentication in ASP.NET WebAPI (and the associated code on GitHub)
Jamie Kurtz' BasicAuthForWebAPI (based on the ASP.NET membership provider)
"Custom Authorization in ASP.NET WebAPI - What A Mess?" on StackOverflow

Now, the optimal number of references in a scenario like this is one. A single article, ideally written by the framework authors, saying "this is how you implement this very simple thing", and identifying the available integration points. Any framework where 3+ people have written in-depth articles on how to add something as simple as HTTP Basic authentication is insufficiently opinionated. Then there's the realization that WebAPI follows the ASP.NET MVC convention of decorating controllers and actions with attributes to support features like authentication - and injecting dependencies into attributes is fiddly, because of the way that attributes are instantiated by the runtime. So... fast-forward several hours of Googling and forking and compiling things and generally poking around, and I decided that Robert Muehsig's approach is the closest thing to what I'm after. And since it's on GitHub under an MIT license, I borrowed it. Most of the classes in Restival.Api.WebApi.Security are lifted directly from Robert's sample code. This produced a working implementation that passed all the tests, but I must admit I'm still not happy with it. It's definitely one I plan to revisit. It's notable that this implementation explicitly decouples the attribute itself from the filter implementation - this is so I can inject dependencies into the filter at runtime whilst still using the attribute declaratively, but it's rather inelegant and I can't say I'm terribly happy with it.

Total implementation time for this was at least a day, including research, reading examples, prototyping and digging into the WebAPI source to work out which bits to override. That's significantly more than I thought it would be, and I'm rather expecting someone to comment on this post "you're doing it wrong!" and link me to ANOTHER blog post or web page which demonstrates a different approach that "only takes five minutes". We shall see. :)

Conclusions

Here's how the four authentication implementations stack up. The implementation time here isn't how long it took, it's how long I think it would take to do it again now I'm familiar with the various frameworks authentication/plugin patterns.

Framework	Lines of code	New Classes	Research time	Implementation Time
WebAPI	~ 250	6 (here)	1 day	2-3 hours
ServiceStack	~ 20	1 (RestivalAuthProvider)	15 minutes	15 mins
NancyFX	~ 25	2 (UserIdentity, UserValidator)	30 minutes	15 mins
OpenRasta	~ 30 plus changes to framework	1 (AuthenticationProvider) plus two new framework classes	1 day	1-2 hours

HTTP Basic auth proved an interesting example, because whilst it's an obvious feature on any API feature 'checklist', any reasonably mature HTTP API will almost certainly have moved beyond basic authentication to something more sophisticated such as per-client API keys or OAuth2 tokens.

When it comes to ease of implementation, I'd say it's a dead heat between NancyFX and ServiceStack. I have more experience working with ServiceStack than I do with Nancy, and I suspect my familiarity with their plugin model made things a little easier, but I really don't think there's much in it - the integration points are sensible, the documentation is comprehensive (and accurate!), and I'm pretty confident the implementations presented here reflect the idioms of the corresponding platforms.

OpenRasta and WebAPI were much less straightforward, but for very different reasons, In a nutshell, I think WebAPI is insufficiently opinionated - rather than encouraging or enforcing a particular approach, it supports a wealth of different integration patterns and extension points, and it's really not clear why any one is better than another. OpenRasta, on the other hand, has a very strong authentication pattern that makes a lot of sense once you get your head around it - but a lack of examples and up-to-date documentation means it's quite hard to work out what the pattern is without digging into the framework source code and getting your hands dirty.

Tune in next time, when we're going to start adding hypermedia to our /whoami endpoint response.

Thanks to Jonathan Channon, Steven Robbins and Sebastien Lambla for their help and feedback on this instalment.

In London next Saturday? Come to PubConf!

2016-01-05T11:35:00+00:00

Next Saturday January 16th, some of the best speakers in the IT industry – and me – will be appearing at PubConf, a single-track, rapid-fire conference. A series of five-minute Ignite-style talks, curated around three themes that we hope are relevant and familiar to anybody who does “computering” – managing yourself; the future of computering, and ranting about broken things.

I’ll be talking about “The Face Of Things To Come” on the computering track – all about recent advances in computer vision, face detection and recognition, emotion analysis, and how I think this kind of technology is going to change the world. For real, this time.

It promises to be a really great event – and it’s completely absolutely free. Yep, free-as-in-beer free. Read more about it at pubconf.io, and get your tickets at EventBrite.

Restival Part 6: Who Am I, Revisited

2015-12-08T16:23:00+00:00

Note: code for this instalment is in https://github.com/dylanbeattie/Restival/tree/v0.0.6

In the last instalment, we looked at adding HTTP Basic authentication to a simple HTTP endpoint - GET /whoami - that returns information about the authenticated user.

Well... I didn't like it. Both the OpenRasta and the WebAPI implementations felt really over-engineered, so I kept digging and discovered a few things that made everything much, much cleaner.

Basic auth in OpenRasta - Take 2

There's a RestivalAuthenticator which implements OpenRasta's IBasicAuthenticator interface - as with the other frameworks, this ends up being a really simple wrapper around the IDataStore:

public class RestivalAuthenticator : IBasicAuthenticator {
private readonly IDataStore db;

public RestivalAuthenticator(IDataStore db) {
    this.db = db;
}

public AuthenticationResult Authenticate(BasicAuthRequestHeader header) {
    var user = db.FindUserByUsername(header.Username);
    if (user != null && user.Password == header.Password) return (new AuthenticationResult.Success(user.Username, new string[] { }));
    return (new AuthenticationResult.Failed());
}

public string Realm { get { return ("Restival.OpenRasta"); } }
}

and then there's the configuration code to initialise the authentication provider.

ResourceSpace.Uses.PipelineContributor<AuthenticationContributor>();
ResourceSpace.Uses.PipelineContributor<AuthenticationChallengerContributor>();
ResourceSpace.Uses.CustomDependency<IAuthenticationScheme, BasicAuthenticationScheme>(DependencyLifetime.Singleton);
ResourceSpace.Uses.CustomDependency<IBasicAuthenticator, RestivalAuthenticator>(DependencyLifetime.Transient);

Basic auth in WebAPI - Take 2

Restival Part 5: Who Am I?

2015-12-07T16:25:00+00:00

NOTE: Code for this article is at https://github.com/dylanbeattie/Restival/tree/v0.0.5

UPDATE: The WebAPI and OpenRasta implementations described here are... inelegant. After this release, I spent a little more time and came up with something much cleaner - which you can read all about in the follow-up post http://dylanbeattie.blogspot.co.uk/2015/12/restival-part-6-who-am-i-revisited.html

In this instalment, we're going to add two very straightforward capabilities to our API:

HTTP Basic authentication. Clients can include a username/password in an HTTP Authorization header, and the API will verify their credentials against a simple user store. (If you're doing this in production, you'd enforce HTTPS/TLS so that credentials can't be sniffed in transit, but since this is a demonstration project and I'm optimising for readability, TLS is out of scope for now.)
A /whoami endpoint, where authenticated users can GET details of their own user record.

And that's it. Sounds simple, right? OK, first things first - let's thrash out our requirements into something a little more detailed:

GET /whoami with valid credentials should return the current users' details (id, name, username)

But remember - we're building an authentication system, so we should also be testing the negative responses:

GET /whoami with invalid credentials returns 401 Unauthorized
- "invalid" means an unsupported scheme, an unsupported header format, invalid credential encoding, or a correctly formatted header containing an unrecognised username and/or password.
GET /whoami without any credentials returns a 401 Unauthorized
GET /whoami without any credentials returns a WWW-Authenticate header indicating that we support HTTP Basic authentication.

The WhoAmI endpoint was trivial - find the current user name, look them up in the user store, and convert their user data to a WhoAmIResponse. The fun part here was the authentication.

A Note on Authentication vs Authorization

Authentication - The Best Case Scenario

OK, to build our secure /whoami endpoint, we need a handful of extension points in our framework. We need to:

Validate a supplied username and password against our own credential store
Indicate that specific resources require authorization, so that unauthorized requests for those resources will be rejected
Determine the identity of the authenticated user when responding to an authorized request

Let's look at Nancy first, because alphabetical is as good an order as any.

NancyFX

Add an AuthenticationBootstrapper - which Just WorkedTM without any explicit registration. I'm guessing it's invoked by ~~magic~~ sufficiently advanced technology, because it overrides DefaultNancyBootstrapper.
Implement IUserValidator to connect Nancy to my custom user store - my implementation is just a really simple wrapper around my user data store. My UserValidator depends on my IDataStore interface - and, thanks to Nancy's auto-configuration, I didn't have to explicitly register either of these as dependencies.
In this bootstrapper, call pipelines.EnableBasicAuthentication() and pass in a basic authentication configuration.

protected override void ApplicationStartup(TinyIoCContainer container, IPipelines pipelines) {
    base.ApplicationStartup(container, pipelines);
    var authConfig = new BasicAuthenticationConfiguration(container.Resolve<IUserValidator>(), "Restival.Nancy");
    pipelines.EnableBasicAuthentication(authConfig);
}

Finally, the WhoAmIModule needs a call to this.RequiresAuthentication(), and that's it. Clean, straightforward, no real showstoppers.

UPDATE: The NancyFX documentation has now been updated with a detailed walkthrough on enabling HTTP Basic authentication

OpenRasta

Whilst writing this article, I had a long and very enlightening chat with Sebastien Lambla, the creator of OpenRasta, about the current state of the project and specifically the authentication capabilities. It turns out the features marked as deprecated were intended to be migrated into a separate OpenRasta.Security module, thus decoupling authentication concerns from the core pipeline model - but this hasn't happened yet, and so the code is still in the OpenRasta core package. It's up to you, the implementer, whether to use the existing ('deprecated') HTTP authentication provider, or to roll your own based on the new pipeline approach.

Following those changes to the framework itself, the implementation was pretty easy.

Provide an implementation of IAuthenticationProvider based on my IDataStore interface.
Register the authentication provider and pipeline contributor in the Configuration class:

ResourceSpace.Uses.CustomDependency<IDataStore, FakeDataStore>(DependencyLifetime.Singleton);
ResourceSpace.Uses.CustomDependency<IAuthenticationProvider, AuthenticationProvider>(DependencyLifetime.Singleton);
ResourceSpace.Uses.PipelineContributor<BasicAuthorizerContributor>();
Decorate the WhoAmIHandler with [RequiresBasicAuthentication(realm)]

ServiceStack

public class AppHost : AppHostBase {
public AppHost() : base("Restival", typeof(HelloService).Assembly) { }
public override void Configure(Container container) {
    var db = new FakeDataStore();
    container.Register<IDataStore>(c => db).ReusedWithin(ReuseScope.Container);
    var auth = new AuthFeature(() => new AuthUserSession(), new IAuthProvider[] { new RestivalAuthProvider(db) }) {
    HtmlRedirect = null
    };
    Plugins.Add(auth);
}
}

Total implementation time was about half an hour here - but bear in mind I've worked with ServiceStack a lot, so I'm familiar with the plugin architecture and the pipeline.

WebAPI

Rick Strahl's WestwindToolkit
Authentication Filters in ASP.NET Web API 2 on www.asp.net
Robert Muehsig's Using Basic Authentication in ASP.NET WebAPI (and the associated code on GitHub)
Jamie Kurtz' BasicAuthForWebAPI (based on the ASP.NET membership provider)
"Custom Authorization in ASP.NET WebAPI - What A Mess?" on StackOverflow

Conclusions

Framework	Lines of code	New Classes	Research time	Implementation Time
WebAPI	~ 250	6 (here)	1 day	2-3 hours
ServiceStack	~ 20	1 (RestivalAuthProvider)	15 minutes	15 mins
NancyFX	~ 25	2 (UserIdentity, UserValidator)	30 minutes	15 mins
OpenRasta	~ 30 plus changes to framework	1 (AuthenticationProvider) plus two new framework classes	1 day	1-2 hours

Tune in next time, when we're going to start adding hypermedia to our /whoami endpoint response.

Thanks to Jonathan Channon, Steven Robbins and Sebastien Lambla for their help and feedback on this instalment.

Membership and Dynamics CRM - How's It All Going, Then?

2015-09-29T15:02:00+00:00

A few months back, I blogged about how Spotlight has chosen Microsoft Dynamics CRM 2015 as the platform for delivering our new membership system, and our high-level plan for incorporating Dynamics into our existing infrastructure. Six months on, I thought it'd be worth revisiting to talk about how the project is progressing and the lessons we've learned so far. Particularly since people on Twitter keep asking me if I'm regretting it yet. :)

Now, I wear many hats. I'm a systems architect. I sit on the strategic board at Spotlight. I've been here long enough that I understand most of our business processes in excruciating detail. I'm interested in UX and interaction design. And, sometimes, when nothing else is going on, I write code and build stuff.

With my business hat on, Dynamics CRM is clearly the right solution for us. It works. It supports a business-to-consumer sales model, out of the box, which works really well for us (it surprised me how many of the other CRM systems we looked at assume that every customer is a company and every sale starts with a quote.) Dynamics CRM offers features like case management, service calendars, Outlook integration, multiple price lists and currencies, mobile device support, Active Directory integration - things that would probably never make it to the top of the backlog if we were building our own system. And that's fine - we are not in the CRM business. We build software to help people cast actors and make movies; everything else is overhead.

Strategically, it makes sense. Our initial customization and integration phase will end up being less than six months, after which membership and CRM will be very much a "solved problem" as far as our development team is concerned, freeing us up to focus on other things. Using an off-the-shelf product as the backbone of our membership and business activity gives us lots more options for making improvements in future - certainly more than if we'd built our own system.

So I still believe Dynamics CRM was, and remains, the best solution to our business requirements.

But... (you knew that was coming, didn't you?) as a software developer, it has frustrations, which can be broadly categorized as good, bad and ugly.

The Good...

It's a hugely complex product, and it has a learning curve. There's frequently several ways to achieve something - multiple email integration patterns, several different ways to implement single sign-on for user authentication. Almost anything is possible once you know how. This is frustrating when you don't know how, but once you get the hang of it, it works pretty well.

You can also write custom code in C# that runs directly within Dynamics, which - again - has a hell of a learning curve but is actually a really powerful technique. With a little ingenuity, you can even build C# components that implement your business logic and rules, and then decide later where that logic should be deployed. I like that sort of flexibility.

...the Bad...

Dynamics CRM is the heart of a software ecosystem that's still a long, long way from the software-as-craft ethos which is becoming fairly common elsewhere. Partly, this is a question of tooling and process. Things like revision control, continuous deployment and unit testing become really quite hard when your "project" is a ZIP file full of proprietary XML - no branching, no merging, only the most rudimentary support for rollbacks. As a developer, this is frustrating - but it's important to remember that systems like Dynamics blur the boundary between "developers" and "users" that's prevalent in most development projects. It's a platform. The people who use it every day for the next decade WILL be making changes to it - because the whole reason we're doing it is to empower The BusinessTM to manage their own membership system - and those people aren't software developers.

More worryingly, Dynamics CRM appears to be one of those products where you can make a very good living as an "expert" without actually knowing anything. The amount of misinformation and bad advice surrounding this product is astonishing. I've read blog posts promoting the most dreadful solutions. I've spoken to consultants - and interviewed contractors on quite generous day-rates - who are completely unaware of whole swathes of the core product's capabilities. That makes it really difficult to know who you can trust - and how much time to invest in validating your assumptions before committing to any kind of delivery.

This isn't a problem with Microsoft Dynamics per se - I think it's probably characteristic of any market sector where we see the dreaded "point and click - no development required!" shibboleth - but it's still a lot less fun than working on, say, Nancy or ServiceStack, where the code is open source and most of the community seem to know what they're doing.

...and the Ugly

And then there's the stuff that's just plain stupid. For example, there's an entity in CRM called a Contract. Once a contract has been "invoiced" - which you can do by clicking a single button in the UI - it is immutable. There is literally NOTHING you can do - even with all the Administrator permissions in the world - to change any detail on that contract. Ever. There's various workarounds for this. Microsoft's recommendation is that you need to "clone" the Contract as a draft, cancel the original and replace it with the clone. Our own solution was to write a custom plugin in C# that throws an exception if you ever try to invoice a contract - this works, but it's not pleasant having to work around such an arbitrary restriction.

They say there's only two hard problems in software development - cache invalidation and naming things. Well, Dynamics CRM will remind you of that on a daily basis. Almost everything in CRM needs to have a name otherwise you can't save the record. For people, companies and products, this makes sense... but, really, when you're creating an order to allow somebody to renew their membership, does it need a name? Really? So we're having to build our own code to automatically create these names all over the place, and it's a waste of time.

As for cache invalidation - there's a thing called the "portal toolkit", aka the Developer Extensions for Microsoft Dynamics CRM 2015. You can read all about it here. Pay particular attention to the sentence that says:

The Customer Portal and Partner Relationship Management (PRM) Portal solutions for Microsoft Dynamics CRM 2015 will be available from the Microsoft Dynamics Marketplace soon.

I don't know when "soon" is, but at the moment, if you start using the portal toolkit, you end up with a cache that you can't invalidate - the only supported invalidation mechanism involves deploying an (unreleased) solution to your CRM server, that calls an (undocumented) DLL that you're expected to deploy to your web server. Oh, and if you're running a server farm, you'll need to work out how to configure your load balancer to forward these cache invalidation calls to all your backend servers simultaneously.

So...

Like democracy, it's probably fair to say that Dynamics CRM is the worst option apart from all of the alternatives. It certainly isn't perfect, but it is pretty damn good, and I think the long-term advantages will significantly outweigh the short-term headaches we're having. And if they don't? Well, that's why we have loosely-coupled architecture, isn't it?

Dynamics CRM Online: Any SMTP Server You Like! (as long as it's Gmail or Yahoo)

2015-08-11T13:19:00+00:00

UPDATE: I take it all back. Following a really useful call from Dynamics CRM support, it turns out there are three different mail integration scenarios, only there's no reference to the third one in the CRM Online configuration screens.

Server-side sync using Exchange
Server-side sync using POP3/SMTP - this is the one that's restricted to Yahoo/Gmail, presumably because it IS doing something very clever.
Dynamics CRM Email Relay - this is a completely separate tool that you need to download and install, and use it to configure your email routing.

I'm going through the mail relay config as we speak - will let you know how it goes. Sorry, Microsoft. :)

We are doing a lot of work at the moment with Microsoft Dynamics CRM Online. It's generally very nice - as a business tool, it's excellent; the UI is great, it's a good fit for our requirements, and despite a couple of headaches caused by the restrictions of the CRM Online hosting environment, we've now got a pretty elegant two-way data sync system up and running using EasyNetQ and webhooks.

Now, one of our key requirements for CRM is being able to send email to our customers. Yeah, I know - audacious, right? CRM Online is our first foray into the cloudy world of Microsoft Office 365, and all the rest of our infrastructure (Exchange, Active Directory, etc.) is still hosted on-premise. For testing and staging systems, we use mailtrap.io - a really slick service that will accept incoming mail and store it so you can test things are relaying properly without actually sending any real emails. For production, we use Mandrill, which is a commercial mail relay service - high availability, reputation management, excellent delivery statistics. We send about a million emails a month through Mandrill, and it works beautifully.

So... this morning I log into CRM Online, go into Settings => Email Configuration => New POP3/SMTP Server Profile. Looks pretty straightforward, right? I enter some details, click "save" and get this:

Weird. I don't want to set up server-side synchronization - I just want to send email. So I start poking around, Googling, that kind of thing, and find this article, which says:

You may have read in the documentation that GMail and Yahoo are listed as supported pop3/smtp providers for Microsoft Dynamics CRM Online and

"Although other POP3/SMTP systems may work with Microsoft Dynamics CRM, those systems were not been tested by Microsoft and are not supported."

Let’s be clear about “not supported“. In this context it means precisely “you will not be able to go past server profile screen as we will reject any pop3/smtp provider that is not GMail or Yahoo.”

And that is exactly what the email relay screen does. If you enter any value other than "smtp.gmail.com" or "smtp.mail.yahoo.com" in the Outgoing Server Location field, you get the "unsupported email server" message. I've even tried modifying the configuration using the SDK instead of the UI, but get the same response - "the email server is unsupported".

There's two possibilities here:

Microsoft have worked closely with Yahoo and GMail to provide first-class business email support, with all sorts of clever features and proprietary extensions that aren't supported by any other SMTP mail servers. (UPDATE - yes, it appears this is more-or-less what they've done. See above.)
Somebody has arbitrarily decided to support GMail and Yahoo and exclude all other SMTP servers. Including, by the way, Hotmail (owned by Microsoft), Outlook.com (owned by Microsoft), and our on-premise SMTP relay (powered by Microsoft Exchange)

If I were a betting man, I know which one I'd be putting money on. I'm really disappointed about this. CRM Online isn't a toy. It's a seriously powerful platform, with a hefty price tag attached... and just when everything is going really nicely, something like this comes along and our beautiful product vision is completely hamstrung by the fact that if we want to email our customers, we need to do it via GMail or Yahoo - and there's absolutely no rational justification for this.

Anyone else encountered with this particular restriction? Anyone have any bright ideas as to how we can work around it?

As a Developer, I Want to Abolish User Stories, So That I Can Ship Great Products Faster.

2015-08-04T22:59:00+00:00

Once upon a time, when you programmed by the seat of your pants and the handlebars of your moustache, lots of people wrote specifications. And there was this lovely idea that you could get a Business Analyst to write you a specification, which laid out in very precise detail exactly how The Software was going to function, and then you would give the specification to a development team and they would build it for you, and Everything Would Be Lovely. Except, of course, it wasn't. Only one team ever shipped a massively successful project using this sort of specification-driven waterfall approach, and trust me - they were smarter than you, and they had a much bigger budget. So some bright folks started wondering why software always went wrong, and suggested a whole load of things that would probably help, and came up with things like scrum, and unit testing, and continuous integration.

One of the great ideas that came out this movement was the idea of user stories. See, specifications tended to be incredibly dry and formal, and often did a really bad job of communicating exactly why you were doing something. Joel Spolsky wrote a great article years ago on writing painless functional specifications, but user stories takes this idea even further. And, like a lot of the good ideas that came out of agile, there's a descriptive element to it and a prescriptive element to it. The descriptive part says "write short, simple stories, not detailed specifications" - and the prescriptive part suggests some 'templates' to help you get the hang of this. There's two formats that became popular for working with user stories - given-when-then and as-a-I-want-so-that.

As-a-I-want-so-that is pretty good for describing, at a very high level, what you are trying to do:

As a marketing coordinator, I want to send email to everyone on our mailing list, so that I can tell them about our big summer sale.

And then you'll add a couple of acceptance criteria to the story:

Given I am not subscribed to the mailing list, when the marketing coordinator sends an email about the summer sale, then I should not receive the email.

Given I have received a newsletter email, when I click the unsubscribe link, then I should be removed from the mailing list.

This sort of clarity makes it easy to build the feature, easy to test the feature, and easy to understand exactly what you're doing and why. See? Simple.

Right. Imagine we have a spec from the Olden Days, and Volume 6, Section 8, Subsection 14, Paragraph 9 says:

The handset will feature a Maplin RK82D component installed on the side of the unit. The RK82D will be positioned exactly 45mm from the top of the unit. When activated, the RK82D component will cause the internal speaker to emit a square wave signal at 16Hz, at a volume of not less than 90dBA as measured from a distance of 1 metre.

Now, let's take an old-school project manager and turn them into an agile product owner. Probably by sending them on a three-day course with a certificate and everything. When they get back, they'll dig out the old spec, and they'll laugh at how dry and opaque it all is. And they'll sit down, all excited, and start writing stories that look like this:

As a handset user, I want a Maplin N27QQ component installed on the side of the unit, so that when the component is activated the device will emit a square wave signal at 16Hz at a volume of not less than 90dBA measured from a distance of 1m

And then they'll add acceptance criteria:

Given I am a handset user, when I activate the N27QQ component, then the frequency of the square wave signal will be 16Hz

Given I am a handset user, when I activate the N27QQ component and measure the signal volume at a distance of 1m, then the volume of the square wave will be not less than 90dBA

Given I am a handset user, when I examine the handset, then the distance of the N27QQ component shall be 45mm from the top of the unit

and everything is AWESOME because we're being AGILE! The story will sit there in the icebox for a couple of weeks, until it gets bumped up the list and included in a backlog refinement meeting. At which point this conversation will happen:

Dev: "Er... why are we doing this? I don't understand the justification for including this feature."
PM: "Sorry, what?"
Dev: "Well... do you know what a Maplin N27QQ component is?"
PM: "It's the component specified in the specification"
Dev: "Yes... it's also a big round red plastic button the size of a softball"
PM: "Oh. Well, it's in the specification now."
Dev: "Right. Explain to me what happens when you press it"
PM: "Oh, easy. The unit emits a square wave signal at 16Hz, at a volume of..."
Dev: "Yeah, yeah. Do you know what that sounds like?"
PM: "Er... it's probably some sort of ring tone"
Dev: "No, it's a fart noise as loud as a motorcycle"
PM: "Oh. Well, can we estimate the story, please?"

At which point the developer will start flicking through LinkedIn on their phone and wondering how long it is until they can go to the pub.

You know what the story should have said? It should have said something like:

As Bongo the Clown, I want my new phone handset to feature a massive red fart button, so that I can make children laugh at parties when I answer it.

First of all, unless you happen to be in the circus supply business, someone's going to say "hang on - why are we making phone handsets for clowns? Is that a thing now?"

Second, anybody who reads that story can immediately picture what they're building, who wants it, and why. It leads to creative discussion and suggestions. Someone might have a brilliant idea about replacing the fart noise with a trombone, or making the button light up when you press it. One of your team might remember how Dad used to get mad every time Tony Blair came on the radio, but the volume knob on Dad's stereo fell off when he tried to turn the radio off, and how hilarious it was watching him chase it across the kitchen while cursing and muttering under his breath, and maybe we should make the fart button actually fall off when you press it so Bongo has to chase it around the room? Clear stories let you cut through the waffle and get straight to the important bit - what is this? How do we build it? How might we make it better?

Now, compare these two sentences:

As a handset user, I want a Maplin N27QQ component installed on the side of the unit, so that when the component is activated the device will emit a square wave signal at 16Hz at a volume of not less than 90dBA measured from a distance of 1m
We'll put a giant red fart button on the side of the phone, so that Bongo the Clown can use it to make kids laugh when he's doing children's parties.

Which one makes more sense to you, as a reader? Which one is going to lead to better decisions, better estimation and less time wasted in meetings?

As-a-I-want-so-that is not some sort of witchcraft. It doesn't magically translate your dry, meaningless specification into readable English. Like writing unit tests, it can help to keep you honest when you're breaking bad habits, and it's one more tool to keep handy when you're doing this stuff for real, but it is not why we're here. It's the story that matters, not the syntax. And if you can't tell me a short, simple story about what you want and why, then maybe you don't actually understand the thing you're trying to build, and no amount of syntactic convention is going to fix that.

The Mysterious Case of the Missing Milliseconds

2015-07-29T21:15:00+00:00

Strings are the lingua franca of many distributed systems, and Spotlight is no different. Earlier today, we hit a weird head-scratching bug in one of our services, and - surprise, surprise - turns out it's all do with strings. To work around limitations of an old line-of-business application, we have a database trigger (no, really) that captures changes made to a particular table, serializes them into an XML message, and pushes this into a SQL Service Broker queue; we then have a Windows service that pulls messages off the queue, parses the XML, breaks it into nicely manageable chunks and publishes them all to RabbitMQ using EasyNetQ. SImple. Except, once in a while, it blows up and starts complaining about FormatExceptions.

Now... within the database trigger, we're doing this:

SELECT @OccurredAtUtc = CONVERT(VARCHAR(128), GETUTCDATE(), 126)

which returns 2015-07-29T20:55:21.130 as you'd expect.

There's then a line of code in the Windows service that says:

var format = "yyyy-MM-ddTHH:mm:ss.fff";
DateTime.ParseExact(d, format, CultureInfo.InvariantCulture, DateTimeStyles.AdjustToUniversal | DateTimeStyles.AssumeUniversal);

Now, this is the code of somebody who knows that turning datetimes into strings and back again can get a bit tricky, and so has left absolutely nothing to chance - they've supplied an exact date format, they've specified a culture, they've even gone so far as to specify the DateTimeStyles. There's unit tests and integration tests, and everything looks absolutely lovely. And then it blows up. Very occasionally,

Except... SQL Server does something weird.

DECLARE @DateTime DATETIME
SELECT @DateTime = '2015-07-29 21:59:15:123'
SELECT CONVERT(VARCHAR(128), @DateTime, 126) -- returns 2015-07-29T21:59:15.123 (fine!)

SELECT @DateTime = '2015-07-29 21:59:15:000'
SELECT CONVERT(VARCHAR(128), @DateTime, 126) -- returns 2015-07-29T21:59:15

SELECT @DateTime = '2015-07-29 21:59:15:999'
SELECT CONVERT(VARCHAR(128), @DateTime, 126) -- returns 2015-07-29T21:59:16

SELECT @DateTime = '2015-07-29 21:59:15:001'
SELECT CONVERT(VARCHAR(128), @DateTime, 126) -- returns 2015-07-29T21:59:15

First, SQL Server doesn't have true millisecond precision - the milliseconds part will often get rounded by +- 0.001 seconds. Second - if the milliseconds part is zero, it'll be omitted from the string representation. Which means our incredibly specific and detailed date parsing routine will choke, because suddenly it has a date that doesn't match the format we've specified, and DateTime.ParseExact will throw a FormatException. Unit tests don't pick it up, because why would you mock such completely bizarre (and undocumented) behaviour, when you don't even know it exists?

What this means is that, since any changes done between .999 and .001 milliseconds will blow up, roughly 0.3% of all our transactions will be failing with a FormatException rather than getting synced to the rest of our systems. Which means fishing them out of the error queue and sorting them out manually - ah, the joy of distributed systems. This formatting weirdness happens on every version of SQL back as far as 2003, but there's no reference to it in the documentation until SQL Server 2012. It's been raised as a bug and closed as 'by design' because "the ISO 8601 spec leaves the conversion semantics for fractional seconds up to the implementation" - which I'm pretty sure didn't mean "go ahead and be internally inconsistent!" but as with so many other issues like this, fixing the bug would change behaviour that's been in place for years and could break things. I've no idea how - or why - anyone would build a system that genuinely relies on this bizarre idiosyncrasy, but I'll bet good money somebody out there has done it.

The beautiful irony, of course, is that if we'd used DateTime.Parse instead of ParseExact, we'd never have had a problem. :)

REST Workshop at Progressive.NET 2015 next week

2015-06-26T16:14:00+00:00

I'll be delivering a hands-on workshop at Progressive.NET 2015 at SkillsMatter here in London next week, where I'll be talking about advanced REST architectural patterns, and actually implementing some of those patterns in .NET using several of the frameworks available for building HTTP/REST APIs on ASP.NET

I've tried quite hard to avoid any esoteric requirements, so attendees should only need

A laptop running Visual Studio 2013, Powershell and IIS
A reasonable working knowledge of C# and HTTP
A test runner capable of running NUnit tests - personally I love NCrunch deeply but ReSharper or plain old NUnit will do just fine./
Some familiarity with Git and GitHub - if you know how to fork a repo and clone it to your workstation, you should be fine.

The repo we'll be working from is https://github.com/dylanbeattie/Restival - there shouldn't be a great deal of setup required, but if you want to clone the repository, check it compiles, and set up your local IIS endpoints by running deploy.ps1 ahead of time, it'll save a little time on the day.

During the workshop, we'll be discussing advanced HTTP/REST API patterns - hypermedia, pagination, resource expansion, HTTP PATCH, OAuth2 - and showing off some tools that can help us design and monitor our HTTP APIs. Alongside the discussion, we'll be implementing some of the techniques covered using your preferred choice of WebAPI, ServiceStack, OpenRasta or NancyFX - or even all four, if you're feeling productive - and then discussing the relative pros and cons of these frameworks for each of the patterns we're implementing.

See you there!

Slides and code from NDC Oslo 2015

2015-06-19T11:20:00+00:00

I’m here at the Oslo Spektrum in Norway at NDC 2015, where I’ve been talking about the machine code of the web, SASS, TypeScript, CoffeeScript, bundle transformations, web optimisation in ASP.NET, ReST, hypermedia, resource expansion, API versioning, OAuth2, Apiary, NGrok, RunScope – and that’s just the stuff I actually managed to cover in my two talks. It’s been a really great few days, and huge thanks to the organisers for going to such great lengths to make sure everything has gone so smoothly.

A couple of non-software highlights that I really liked:

The catering has been excellent, and having food available throughout the day is a great way to avoid the lunchtime rush. (And the free coffee at the Twilio stand is excellent!)
The overflow area – where you can tune into any of the 9 talks currently in progress via a wireless headset, or just sit and channel-surf – is a great idea. (But remember it’s there if you’re doing live demos with audience participation – I’m pretty sure the “winner” of my NGrok demo was one of the people in the overflow area!)
If you ever get the chance to see the Norwegian band LoveShack, do it. They played the conference after-party last night, and closed their set with a note-perfect 20-minute medley which went through (I think!) Jump, Celebrate, Girls! Girls! Girls!, Welcome to the Jungle, Paradise City, the theme from Baywatch, Livin’ on a Prayer, Radio Gaga and a half-dozen more before dropping back into Jump mid-guitar-solo without skipping a beat. They’re playing the John Dee bar in Oslo this evening, and I’m almost tempted to change my flight just to stick around and see them again…

Slides, Links and Code Samples

The slides and code samples for the talks I’ve given are up on GitHub: the repo is at https://github.com/dylanbeattie/ndc-oslo-2015 or if you want to download the slide decks directly, links are:

Front-End Fun with Sass and Coffee

Front-End Fun with Sass and Coffee – my slide deck from NDC Oslo 2015
The SASS Language site
TypeScript
CoffeeScript
Bundle Transformer – a “modular extension for System.Web.Optimization” that is also probably my favourite NuGet package of all time, for the sheer elegance of the way it’ll give you all the power of Ruby and Sass in a way that works so harmoniously with the ASP.NET runtime

The Rest of ReST

The Rest of REST – my slide deck from NDC Oslo 2015
“Architectural Styles and the Design of Network-based Software Architectures” – Roy Fielding’s doctorate thesis, in which he outlines the constraints that define ReSTful architecture.
“Your API Versioning is Wrong, which is why I decided to do it three different wrong ways” – Troy Hunt’s blog post about API versioning.
apiary.io – rapid prototyping for HTTP+JSON APIs. Which MIGHT be ReSTful if you read Roy Fielding’s thesis before you start playing with it.
NGrok – “secure tunnels to localhost”
RunScope – monitoring, logging and debugging for HTTP APIs and services.
HAL: Hypertext Application Language by Mike Kelly

I also want to follow up on one specific question somebody asked after my ReST talk this morning, which can be paraphrased as “are you comfortable recommending that people use HAL, seeing as it’s basically a dead specification?” An excellent question, and one that probably slightly more detailed answer than the one I gave on the spot. To put this in context, the HAL specification was submitted to the IETF as a draft-kelly-json-hal-06 in October 2013; that draft expired in 2014 and hasn’t been updated or ratified since, so I can see how you could argue that HAL is “dead”.

First – I’d disagree with that. Although the specification itself hasn’t changed in a while, the mailing list and community is still relatively active, and I’ve no doubt would still welcome engagement and contributions from anybody who wished to participate. Second – the spec still provides a perfectly valid approach. It’s a specification, not a tool or a framework, and in terms of delivering working software, if HAL helps you solve your problem then I say go go for it. Third – and I should have made this more obvious in this morning’s talk – HAL is just one of several approaches for delivering hypermedia over JSON. I used HAL in my examples because I think it’s the most readable, but that doesn’t mean it’s the best choice for your application. (Remember, one of my requirements for a hypermedia language in this context was “looks good on Powerpoint slides”.) If you’re interested, I would recommend also looking at JSONAPI, JSON-LD, Collection+JSON and SIREN. There is a great post by Kevin Sookocheff, which succintly summarises the difference between four of them – it doesn’t cover JsonAPI - and concludes “there is no clear winner. It depends on the constraints in place on your API”

Right. I’m going to watch Troy Hunt making hacking child’s play for an hour, and then head to the airport. Thank you, Oslo. It’s been a blast.

Restival Part 4: Deployment and Fun with Case Sensitivity

2015-06-12T16:12:00+00:00

Before we dive into the next phase of API development, I wanted to make it a little easier to install and run the Restival app on your own machine, so I've added a Powershell Deploy.ps1 script to the project which will:

Create a new local IIS website called Restival, bound to http://restival.local/
Create applications for each of our API implementations
Configure the whole thing to run under the ASP.NET v4.0 application pool.

One interesting quirk I discovered whilst doing this is that OpenRasta appears to be case-sensitive when it comes to request URLs. I'd initially created the applications like this:

The test project uses lowercase URLs - so http://restival.local/api.nancy/ - and for some strange reason, the OpenRasta implementation just doesn't work if the IIS application name differs in case from the URL in the unit test. I'll dig into this a little further but for now, I've just modified the deploy script to do a .ToLower() on the application name and everything's working. Code for this instalment is in https://github.com/dylanbeattie/Restival/tree/v0.0.4

Restival Part 2 Revisited: Attribute Routing in WebAPI

2015-05-19T21:55:00+00:00

(Code for this instalment is version 0.0.3 on GitHub if you're following along.)

Mike Thomas commented on my last post, asking "any reason why you are not looking at attribute routing in WebAPI"? To which my answer is "yes - I didn't know it existed", which I'd argue is a pretty good reason why I hadn't looked at it! But Mike's absolutely right to bring it up - if we're comparing frameworks, it makes a lot of sense to really explore the full capabilities of those frameworks. So I've been reading up on attribute routing, and have to say it looks rather nice - and will, I suspect, help out with a lot of the more advanced stuff that's coming up in future instalments.

According to the documentation on www.asp.net:

Web API 2 supports a new type of routing, called attribute routing. As the name implies, attribute routing uses attributes to define routes. Attribute routing gives you more control over the URIs in your web API. For example, you can easily create URIs that describe hierarchies of resources.

The earlier style of routing, called convention-based routing, is still fully supported. In fact, you can combine both techniques in the same project.

Sounds good, right? So what do we need to do to make Restival's WebAPI implementation run on attribute routing instead of convention-based routing?

As it turns out, not much. Well, apart from a little light yak-shaving. Attribute routing is in the Microsoft.AspNet.WebApi.WebHost package - so let's install it:

PM> Install-Package Microsoft.AspNet.WebApi.WebHost
Attempting to resolve dependency 'Microsoft.AspNet.WebApi.Core (≥ 5.2.3 && < 5.3.0)'.
Attempting to resolve dependency 'Microsoft.AspNet.WebApi.Client (≥ 5.2.3)'.

...

Install failed. Rolling back...
Install-Package : Could not install package 'Microsoft.AspNet.WebApi.Client 5.2.3'. You are trying to install this package
into a project that targets '.NETFramework,Version=v4.0', but the package does not contain any assembly references or content
files that are compatible with that framework. For more information, contact the package author.At line:1 char:1
+ Install-Package Microsoft.AspNet.WebApi.WebHost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Install-Package], InvalidOperationException
+ FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand

OK, no problem - we're currently targeting .NET 4.0 and it looks like WebApi.WebHost wants .NET 4.5. Right-click, properties, Target Framework to .NET Framework 4.5.1, done. Shift-Ctrl-B... what's this?

Oh. OK. Let's enable NuGet Package Restore so it'll reinstall packages when we compile the solution... oh dear:

Oh, joy. Right. This just stopped being fun, because now the solution has entered a sort of weird limbo-state where it's not restoring packages, but the option to enable package restore has disappeared. Time for a tried and tested troubleshooting routine:

Close Visual Studio. Completely. SHUT IT DOWN. Yes. And the other instance you've got open. In fact, reboot the machine. DO IT.
Whilst it reboots, get something to drink. Coffee if you're on the clock (did I mention Spotlight has a bean-to-cup espresso machine? We're hiring, you know...) - or something a little stronger if you're not.
Put on "Turn Up the Radio" by Autograph.
Take a deep breath.
Re-open Visual Studio, re-open your solution, try building it again.

This time, it builds. It gives a warning about assembly version conflicts, and then settles down to 0 errors and 1 warning:

Warning: Some NuGet packages were installed using a target framework different from the current target framework and may need to be reinstalled. Visit http://docs.nuget.org/docs/workflows/reinstalling-packages for more information. Packages affected: EntityFramework, Microsoft.Net.Http

Well, we're not using Entity Framework so I can just remove it. Except I can't, because Microsoft.AspNet.Providers.Core uses EntityFramework, and Microsoft.AspNet.Providers.LocalDB uses Providers.Core... but since I'm not using ANY of those, we can remove LocalDB, which removes Core, which removes EntityFramework, and we're down to a single warning about Microsoft.Net.Http, which we can fix with a NuGet package reinstall. Simple.

PM> Update-Package -reinstall Microsoft.Net.Http
Removing 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.WebApi.
Successfully removed 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.WebApi.
Removing 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.ServiceStack.
Successfully removed 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.ServiceStack.
Removing 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.OpenRasta.
Successfully removed 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.OpenRasta.
Uninstalling 'Microsoft.Net.Http 2.0.20710.0'.
Successfully uninstalled 'Microsoft.Net.Http 2.0.20710.0'.
Installing 'Microsoft.Net.Http 2.0.20710.0'.
You are downloading Microsoft.Net.Http from Microsoft, the license agreement to which is available at http://www.microsoft.com/web/webpi/eula/MVC_4_eula_ENU.htm. Check the package for additional dependencies, which may come with their own license agreement(s). Your use of the package and dependencies constitutes your acceptance of their license agreements. If you do not accept the license agreement(s), then delete the relevant components from your device.
Successfully installed 'Microsoft.Net.Http 2.0.20710.0'.
Adding 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.WebApi.
Install failed. Rolling back...
Update-Package : Unable to uninstall 'Microsoft.Net.Http 2.0.20710.0' because 'Microsoft.AspNet.WebApi.OData 4.0.30506'
depends on it.At line:1 char:1
+ Update-Package -reinstall Microsoft.Net.Http
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Update-Package], Exception
+ FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.UpdatePackageCommand

'Microsoft.Net.Http 2.0.20710.0' already installed.
Adding 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.ServiceStack.
Successfully added 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.ServiceStack.
'Microsoft.Net.Http 2.0.20710.0' already installed.
Adding 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.OpenRasta.
Successfully added 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.OpenRasta.

PM>

Hmm. I don't even know what WebApi.OData is, and I'm pretty sure I'm not using it... Let's remove it. Which, of course, means removing a bunch of other things, including Microsoft.Net.Http... which requires another Visual Studio restart. And this time, Update-Package fails because Microsoft.Net.Http is actually gone... so let's install it:

PM> Install-Package Microsoft.Net.Http

Zing! Done. Clean build, zero errors, zero warnings, it works. Now we can actually implement attribute routing.

First, we're going to remove our existing Hello route and enable attribute routing:

public static class WebApiConfig {
    public static void Register(HttpConfiguration config) {
        config.MapHttpAttributeRoutes();
        //config.Routes.MapHttpRoute(
        //    "Hello", // route name
        //    "hello/{name}", // route template
        //    new { Controller = "Hello", Name = "World" } // defaults
        //    );
    }
}

Next, we need to decorate our HelloController with the route attribute:

public class HelloController : ApiController {
    [Route("hello/{name}")]
    public Greeting Get(string name) {
        return (new Greeting(name));
    }
}

FInally, we need to change the way we register our configuration, because the old WebAPI 1.x convention isn't compatible with attribute routing:

protected void Application_Start() {
    // Old WebAPI 1.x syntax - not compatible with attribute routing:
    // WebApiConfig.Register(GlobalConfiguration.Configuration);

    // New WebAPI 2.x configuration via delegate instead of direct method call
    GlobalConfiguration.Configure(WebApiConfig.Register);
}

That works - well, everything works except our default "Hello, World" scenario - so let's add a default value to the {name} parameter in our route attribute:

public class HelloController : ApiController {
    [Route("hello/{name=World}")]
    public Greeting Get(string name) {
        return (new Greeting(name));
    }
}

And there you go. Attribute routing works, all tests are passing - and a yak so impeccably shaved you could use it to sell cologne. Not bad.

Restival Part 3: Hello, World!

2015-05-07T21:50:00+00:00

So far, we've got a /hello service up and running in four different frameworks (five if you include Andrew 'kolektiv' Cherry's F#/Freya implementation, which looks really interesting). Last time, we looked at how our frameworks handle routing; in this instalment, we're going to look at adding default values to our route parameters. Specifically, if you just call /hello, the service should return "Hello, World!"

GET /hello

should return

200 OK

{ "Message" : "Hello, World!" }

OK, so how we do this? There's three places we can potentially do it - the routing, the handlers, or the underlying Greeting object itself - but not all our frameworks support all three options.

Only WebAPI supports explicit defaults in the routing configuration, using this syntax:

config.Routes.MapHttpRoute(
    "Hello", // route name
    "hello/{name}", // route template
    new { Controller = "Hello", Name = "World" } // defaults
);

ServiceStack doesn't support routing defaults per se, but because the request is decoupled from the rest of our implementation, we can specify the default inside our request DTO - and thanks to the single responsibility principle, we know this isn't going to affect any other details of our implementation

[Route("/hello")]
[Route("/hello/{name}")]
public class Hello {
    private string name = "World";
    public string Name {
        get { return name; }
        set { name = value; }
    }
}

OpenRasta's routing maps our requests directly onto the Get method of our HelloHandler, which makes it really easy to use .NET's optional parameter syntax to supply a default value to the handler method itself:

public class HelloHandler {
    public object Get(string name = "World") {
        return (new Greeting(name));
    }
}

Finally, NancyFX has such sparse routing that there's only really one line of code we could modify - after all, our entire API's only two lines of code, and the other one's doing our /hello/{name} implementation.

public class HelloModule : NancyModule {
    public HelloModule() {
        Get["/hello"] = _ => new Greeting("World");
        Get["/hello/{name}"] = parameters => new Greeting(parameters.name);
    }
}

Astute readers will be going "but hang on - you've just defined the same default in four different places! That's not very SOLID! What if it changes?" - and yes, you're exactly right. Because this is a demo application, I'm favouring readability over abstraction; if you had to define this kind of default behaviour in four different places in a real app, you'd do it using constants, or implement it on the Greeting object itself so the code is reused between all four endpoints.

Code for this post is on GitHub as v0.0.2.

Now we've got a basic skeleton API up and running, it's probably time to share my to-do list - things we'll be implementing, in no particular order

List resources, and retrieving multiple objects
Support for PUT and DELETE (easy), POST (interesting) and PATCH (really interesting)
Hypermedia links using HAL
Pagination and resource expansion, as described in this excellent post on the Stormpath blog and this blog post from Etsy's Venkata Mahalingam
API versioning - because, sooner or later, we're going to break something and need to maintain backward compatibility for older clients. Check out Troy Hunt's great post on API versioning for a preview of how we'll be getting this wrong in three different ways.
OAuth2 and bearer authentication - this one's going to take some work, because before you can build an OAuth2 API, you need an OAuth2 authentication server, but I've wanted to build a really lightweight .NET OAuth2 server for a while so I'm quite looking forward to it.
API documentation using tools like Swagger, which generates documentation based on metadata exposed by your API itself.

See you next time.

Restival Part 2: All Aboard The Routemaster

2015-05-04T21:24:00+00:00

Hello, and welcome to the second instalment of Restival: The Great .NET ReST Showdown (part 1 is here if you missed it) So far, our API defines a single very simple method – “hello”. Making a GET request to /hello will return { "Message" : "Hello, World!" }, and making a GET request to /hello/chris will return { "Message" : "Hello, Chris!" }

The code we're discussing here is on GitHub as release v0.0.1. This release supports /hello/{name}, which demonstrates routing and parameter binding. I've deliberately not implemented "Hello, World" at /hello yet, because I want to do that by using the various frameworks' conventions for specifying default parameter values and that logically can't happen until you've defined your routes. Even at this incredibly early stage, there's some interesting design decisions going on.

Routing and Route Binding

Routing controls how your app will map incoming HTTP requests to methods - it's the set of rules that say "when you get a request that looks like X, run method Y on class Z"

Nancy has a really lightweight routing syntax inspired by Sinatra - by inheriting from NancyModule, you get access to a RouteBuilder, a sort of dictionary that maps routes to anonymous methods, for each supported HTTP verb (DELETE, GET, HEAD, OPTIONS, POST, PUT and PATCH) - to add a route, you supply the pattern to match and the method implementation:

public class HelloModule : NancyModule {
    public HelloModule() {
        Get["/hello/{name}"] = parameters => new Greeting(parameters.name);
    }
}

Note the Nancy convention whereby we use an underscore to indicate "we're not using this variable for anything" in handlers that don't actually use their parameter dictionary. It's also worth noting that Nancy's lightweight syntax won't stop you defining multiple handlers for the same route - but this can lead to non-deterministic behaviour, so don't do it :)

WebAPI uses an explicit routing table that's configured during application startup - in WebAPI, there's a call to WebApiConfig.Register(GlobalConfiguration.Configuration) in Application_Start, and routes are mapped by specifying the name, the URL pattern and the defaults to use for that route. (If you're familiar with routing in ASP.NET MVC, WebAPI uses a very similar routing configuration, but with the 'action' mapped to the HTTP verb instead of to a path segment.)

config.Routes.MapHttpRoute(
    "Hello",   // route name
    "hello/{name}", // route template
    new { Controller = "Hello" } // route defaults
);

OpenRasta and ServiceStack are both far more explicit about the relationship between resources, routes and handlers. OpenRasta uses a fluent configuration interface to declare your resources (i.e. the bits of data we're interested in), your URIs (routes), handlers (the bits of code that actually do the work), and contracts (which handle things like serialization and content types)

public class Configuration : IConfigurationSource {
    public void Configure() {
        using (OpenRastaConfiguration.Manual) {
            ResourceSpace.Has.ResourcesOfType<Greeting>()
                AtUri("/hello/{name}")
                .HandledBy<HelloHandler>()
                .AsJsonDataContract();
        }
    }
}

Finally, ServiceStack requires you to explicitly define requests (DTOs representing the incoming request data), services (analogous to handlers in our other frameworks) and responses. This is far more verbose than the other frameworks, but providing these abstraction layers between every aspect of your ReST API and your underlying codebase gives you more flexibility to evolve your API independently of the underlying business logic. You map your routes to your request DTOs using the Route attribute, and inherit from ServiceStack.Service when implementing your handlers. ServiceStack maps HTTP verbs onto service method names - HelloService.Get(Hello dto), HelloService.Post(Hello dto), etc. - but also supports a catch-all Any() method which will map incoming requests regardless of the request verb.

[Route("/hello")]
[Route("/hello/{name}")]
public class Hello {
    public string Name { get; set; }
}

public class HelloResponse {
    public string Message { get; set; }
}

public class HelloService : Service {
public HelloResponse Any(Hello dto) {
    var greeting = new Greeting(dto.Name);
    var response = new HelloResponse() { Message = greeting.Message };
    return (response);
}
}

So there you go. /hello/{name} takes one line in NancyFX, a couple of lines in OpenRasta and WebAPI, and three entire classes in ServiceStack. Before you draw any conclusions, though, try pointing a browser at the root URL of each API implementation.

Nancy gives you this rather splendid 404 page - complete with Tumbeast:

Running under IIS, WebAPI and OpenRasta both interpret GET / as a directory browse request, and give you the all-too-familiar IIS 7.5 HTTP error screen:

But the pay-off for the extra boilerplate required by ServiceStack is this rather nice API documentation page, describing the services and encoding formats supported by the API and providing WSDL files for adding our API as a service endpoint. Now, we're not actually using any of that yet... but as our API grows, it's going to be interesting to see how much extra work the other frameworks require to do things that ServiceStack provides for free. (Or for $800 per developer, depending on what you're doing with it.)

Now, it's important to remember that we're trying to reflect the conventions and idioms of our chosen frameworks here. You could, without too much difficulty, implement the request/service/response pattern favoured by ServiceStack on any of the other frameworks, or to get your ServiceStack services to return raw entities instead of mapping them into Response objects - but if you're trying to make framework A behave like framework B, you might as well just switch to framework B and be done with it.

In the next episode, we're going to make GET /hello return "Hello, World!", and in the process look at how to define default values for our route parameters in each of our frameworks. Until then, happy hacking!

One API, Four Frameworks: The Great .NET ReST Showdown

2015-04-27T23:26:00+00:00

There’s only two hard problems in software: cache invalidation, naming things, off-by-one errors, null-terminated lists, and choice overload. Back when I started building data-driven websites, classic ASP gave you Request, Response, Application, Session and Server, and you built the rest yourself - and it was uphill both ways! These days, we’re spoiled for choice. Whatever you’re trying to achieve, you can bet good money that somebody’s been there before you. If you’re lucky, they’ve shared some of their code – and if you’re really lucky, they’ve built a framework or a library you can use.

Since the heady days of classic ASP, I’ve built quite a few systems that have to expose data or services to other systems – from standalone HTTP endpoints that just return true or false, to full-featured ReST APIs. I’ve come across several frameworks that are designed to help developers create ReSTful APIs using Microsoft .NET, each of which doubtless has its own strengths and weaknesses – and now I find myself facing the aforementioned choice overload, because if I was starting a new API project right now, I honestly don’t know which framework I’d use. Whilst I’ve got hands-on experience with most of them, I’ve never had the opportunity to try implementing the same API spec in multiple frameworks to compare and contrast their capabilities on a like-for-like basis.

So here goes. Over the next few months, I’m going to be developing the same API in four different frameworks, side-by-side. The APIs have to use the same back-end code – same data access, same business logic – and have to pass the same integration test suite. I’m going to start out really simple = “hello, world!” simple – and gradually introduce features like resource expansion, pagination, OAuth2, content negotiation. The idea is that some of these features will actually break the interface, so I’ll also be looking at how to handle API versioning in each of my chosen frameworks. I’m also going to try and respect the idioms and conventions of each of the frameworks I’m working with – good frameworks tend to be opinionated, and if you don’t agree with their opinions you’re probably not going to find them particularly helpful.

The Frameworks

Microsoft ASP.NET WebAPI (http://www.asp.net/web-api)

Microsoft’s out-of-the-box solution for building HTTP-driven APIs. Superficially similar to ASP.NET MVC but I suspect there’s much more to it than that. I’ve built a couple of small standalone APIs in WebAPI but not used it for anything too substantial.

ServiceStack (https://servicestack.net/)

For a long while I was completely smitten with ServiceStack. Then it hit v4.0 and went from free-as-in-beer to expensive-as-in-$999-per-developer for any reasonable-sized project – there is a free usage tier, but it’s pretty restrictive. That said, it’s still a really powerful, flexible framework. Version 3 is still on NuGet, is still available under a BSD license, and there’s at least one active project based on a fork of the ServiceStack v3 codebase. I like ServiceStack’s conventions and idioms; working with it taught me a lot about ReST; it has great support for things like SwaggerUI, and I suspect as I start implementing various API features ServiceStack is going to deliver additional value and capabilities which the other frameworks can’t match. Will it add enough value to justify $999 per developer? We’ll see :)

OpenRasta (http://openrasta.org/)

I’ve played with OpenRasta on-and-off over the years, though I’ve never used it on anything substantial, but I know the folks over at Huddle are using it in a big way and having great results with it, so I’m really curious to see how it stacks up against the others. (I should probably disclose a slight bias here in that Sebastien Lambla, the creator of OpenRasta, is a friend of mine; it was Seb who first got me thinking about ReST via London .NET User Group and prolonged conversations in the pub afterwards.)

NancyFX (http://nancyfx.org/)

This one is completely new to me – until last week, I’d never even looked at it. But so far, it looks really nice – minimalist, elegant and expressive, and I’m looking forward to seeing what it can do.

Other Candidates

It would be interesting to get some F# into the mix – mainly because I’ve never used F# and I’m curious. I’ve heard interesting things about WebSharper and Freya – and, of course, if anyone wants to add an F# implementation and send me a pull request, go for it!

The API

I’m using Apiary.IO to design the API itself – you can check it out at http://docs.restival.apiary.io/

The Code

The code is on GitHub - https://github.com/dylanbeattie/Restival.

If you want to run it, you’ll need to set up IIS applications for each of the four framework implementations – I use a hosts file hack to point restival.local at 127.0.0.1, and I’ve set up http://restival.local/api.webapi, http://restival.local/api.nancy, http://restival.local/api.servicestack and http://restival.local/api.openrasta

The test suite is NUnit, uses RestSharp to make real live HTTP requests, and all the actual test logic is in the abstract base class. There’s four concrete implementations, and the only difference is the URL of the API endpoint under test.

The Backlog

Features I want to implement include, but are probably not restricted to…

Pagination. What’s a good way to handle huge result sets without causing timeouts and performance problems?
Resource expansion. How do you request a customer, and all their orders, and the invoices linked to those orders, in a single API call?
API versioning – using all three different ways of doing it wrong

Version numbers in the URL (api.foo.com/v1/)
Version numbers in a custom HTTP header (X-Restival-Version: 1.0)
Content negotation based on MIME types (Accept: application/vnd.restival.v1.0+json)

OAuth2 and bearer token authentication. (You’ll like this one because I’m not using DotNetOpenAuth)
API documentation – probably by seeing how easily I can add Swagger support to each implementation

In every case, this is stuff I’ve already implemented in at least one project over the last couple of years, so it’s going to be interesting seeing how readily those implementations translate across to the other frameworks I’m using.

Sound like fun? You bet it does. Tune in and see how it unfolds. Or come to NDC Oslo in June or Progressive.NET in London in July, where you not only get to listen to me talk about ReST, you get several days of talks and workshops from some of the best speakers in the industry.

Spotlight, Dynamics CRM, and the age-old question of “build vs buy”

2015-04-24T16:39:00+00:00

We’re in the early stages of creating a new membership system built on Microsoft Dynamics CRM 2015. This isn’t a decision we’ve taken lightly – the decision to buy vs. build is always complex where software is concerned, as Mike Hadlow explains in this excellent blog post. In our case, though, there’s two good reasons why we’ve decided to integrate an off-the-shelf solution.

First, we are willing to change our own business process to suit the software. We’ve been printing books since 1927, and our business model is tightly coupled to our publishing process. As part of this project, we’re going to decouple those two areas of activity. Publishing is a differentiator for us, and always will be, but membership is not – whilst it’s important that it works, it’s not hugely important how it works. If CRM can offer us a “pit of success”, we’ll happily do what’s necessary to fall in it.

Second – we really understand what it costs to write our own software. We’ve got a solid, mature agile process, which links in to our time-tracking system. One of the high-level metrics that we track is “cost per point” – how much, in pounds, does it cost us to get a single story point into production? It’s easy for businesses to think of off-the-shelf software as “expensive”, because it has a price tag, and bespoke software as ‘free’ because you’re already paying developers’ salaries, but that’s a pretty naive way to look at it. When you factor in the opportunity cost of all the things we could have been doing instead of reinventing the CRM wheel, the off-the-shelf option starts to look a lot more attractive even when it carries a hefty up-front price tag.

That’s two good reasons why we’re going with CRM2015. Now for two good reasons why CRM projects fail, and what we’re doing to mitigate them.

First – scope creep. CRM vendors will happily sell it CRM as the solution to all your problems, and then they’ll start showing off marketing campaigns and case management and Outlook integration and web portals and everybody’s eyes light up like this is the most amazing thing they’ve ever seen… and next thing you know you’ve got a 300-page “requirements” document and everyone’s got so carried away by what’s possible that they’ve forgotten what they were trying to fix in the first place. I’ve seen this happen first-hand, and it doesn’t work, and the reason it doesn’t work is that the project isn’t being driven by prioritised requirements, it’s being driven by wish-lists.

So… start with a problem. Any successful business probably has dozens of things that could be done better – so list them, analyze them, identify dependencies. Work out which one to solve first, and focus. CRM isn’t fundamentally different to any other software project. Identify your milestones. Be absolutely brutal about the MVP – what is the simplest possible thing that’s better than what we’ve got right now? Build that. Ship that. Get people using it. In our case, CRM’s first outing is going to be as a replacement for GroupMail, our email-merge tool, and that’s it. We’ll integrate just enough data that CRM can send personalised email to a specific group of customers, we’ll ship it, and we’ll use it – and then we’ll iterate based on feedback and lessons learned. We already have a pretty good idea what we’ll do after that, but we’re not going to worry about it until we’ve delivered that first MVP release.

I think the second reason CRM projects fail is over-extension. Dynamics CRM is a really powerful, flexible platform, and with enough ~~consultants~~ effort you can probably get it to do just about anything. But that doesn’t mean it’s the best solution. Sure, there’s going to be cases where it makes sense to customise CRM by adding a new field or some validation rules. Spotlight holds the same core “business” data as any other company – what’s your name? Where do you live? what’s your email address? Is your account up-to-date? Off-the-shelf CRM is very, very good at managing this sort of information – and once you’ve got this core information in CRM, there’s dozens of off-the-shelf marketing tools available to help you use it more effectively.

But Spotlight stores a lot more than that. We also store all the information that appears on your professional acting CV – height, weight, eye colour, hairstyle, skills, credits. We store details of almost all the productions being cast in the UK, we track tens of thousands of CV submissions every day, and millions of job notification emails each week. We manage terabytes of photography and video clips. You probably could get CRM to manage all this information. But hey, you can open a beer bottle with a dollar bill – doesn’t mean it’s a good idea, though.

The overlap – the green bit – is where CRM solves one of our problems. The blue bit is things CRM does that aren’t really relevant to us – not at the moment, anyway. The red bit is the stuff that we’re going to keep out of CRM. And that dark area on the border… that’s representation, which is a tremendously complicated tangle of operational data, business data and publishing data that we’re going to have to work out as part of our service roadmap. Fun.

Now, different people – and systems! – have different expectations about what “CRM” and “membership” mean.

To our customers, good CRM means it’s easy to join Spotlight, it’s easy to manage your account, it’s easy to talk to us and get answers to your questions. You know how you hate phoning the electric company because every time you get through you talk to a different person who has no idea what’s going on, and your “my account” page on their website says one thing and your bill says something else and the person on the phone doesn’t agree with either of them? Yeah. Imagine the complete opposite of that.
To our marketing team, good CRM is about accurate data, effective marketing campaigns, happy customers – and, yes, revenue. It’s about helping us work out what we’re doing right and what we’re doing wrong, giving us the intelligence we need to make decisions about new products and initiatives.
To our software team, good CRM means easy access to the data and processes you’ll need to build great products. Responsive systems, logical data structures, simple integration patterns and intuitive API endpoints. In other words, if you want to build an awesome online tool that’s only available to customers with a current, paid-up Spotlight Actors membership, it should be trivial to work out whether the current user can see it or not.

Same system, same data, three radically different use cases. So here’s how we’re proposing to make it work:

Astute readers will notice a slim black box marked “abstraction layer”. That’s how we’re going to fool the rest of our stack into thinking that Dynamics CRM 2015 is a ReSTful microservice, so tune in over the next couple of weeks to find out how it works, what sort of patterns and techniques we’re using, and how we’re going to test and monitor it.

(Astute readers may also notice a resemblance between Spotlight’s customer base and the cast of Game of Thrones… well, that’s because Spotlight’s customers are the cast of Game of Thrones. I told you working here was awesome.)

Spotlight, Dynamics CRM, and the age-old question of “build vs buy”

2015-04-24T16:28:00+00:00

We’re in the early stages of creating a new membership “engine” built on Microsoft Dynamics CRM 2015. This isn’t a decision we’ve taken lightly – the decision to buy vs. build is always complex where software is concerned, as Mike Hadlow explains in this excellent blog post. In our case, though, there’s two good reasons why we’ve decided to integrate an off-the-shelf solution.

That’s two good reasons why we’re going with CRM2015. Now for two good reasons why CRM projects fail, and what we’re doing to mitigate them.

Now, different people – and systems! – have different expectations about what “CRM” and “membership” mean.

To our customers, good CRM means it’s easy to join Spotlight, it’s easy to manage your account, it’s easy to talk to us and get answers to your questions. You know how you hate phoning the electric company because every time you get through you talk to a different person who has no idea what’s going on, and your “my account” page on their website says one thing and your bill says something else and the person on the phone doesn’t agree with either of them? Yeah. Imagine the complete opposite of that.
To our marketing team, good CRM is about accurate data, effective marketing campaigns, happy customers – and, yes, revenue. It’s about helping us work out what we’re doing right and what we’re doing wrong, giving us the intelligence we need to make decisions about new products and initiatives.
To our software team, good CRM means easy access to the data and processes you’ll need to build great products. Responsive systems, logical data structures, simple integration patterns and intuitive API endpoints. In other words, if you want to build an awesome online tool that’s only available to customers with a current, paid-up Spotlight Actors membership, it should be trivial to work out whether the current user can see it or not.

Same system, same data, three radically different use cases. So here’s how we’re proposing to make it work:

NSBCon 2014 – All About NServiceBus

2014-05-22T12:03:00+00:00

I'm excited to say I'll be speaking at NSBCon here in London in June. NServiceBus is the most popular open-source service bus for .NET platforms, and I'll be talking about how we use NServiceBus at Spotlight to deliver online photo and multimedia publishing systems aimed at professional actors, casting directors and production professionals.
I'll be talking about two distinct systems. The first is our proprietary system for managing online photo publication. Spotlight has been publishing photographs in one form or another since 1927, and so our current system is the latest in a long, long series of incremental developments - from acid-etched brass photograph plates, to lithographic bromide machines, to industrial film scanners and ImageMagick. Photography is an absolutely fundamental part of the creative casting process, and we reached a point a few years ago where our web servers were spending 60%-70% of their CPU time rendering photographs. Working within the limitations of our legacy systems, we developed a distributed thumbnailing and caching system that delivers the same results with a fraction of the processing overhead. I'll discuss how we used NServiceBus to work around those limitations, the architectural and operation challenges we faced building and running this system, and some of the lessons we learned when, a year after deploying it into production, we had to migrate this system from onsite hosting to a private cloud.
The second is an online audio/video upload and publishing system. Working with online video offers a unique set of challenges - dozens of incompatible file formats, massive file uploads, unpredictable analysis and encoding jobs. We built and deployed a system that satisfied these requirements whilst offering an intuitive, responsive user experience, and in this talk I'll cover the high-level architectural approach that enabled this. We'll look at how we used NServiceBus to decouple the 'heavy lifting' components of the application from the customer-facing user interface components, and some of the lessons we learned deploying a distributed NServiceBus application in an Amazon EC2 hosting environment.
NSBCon is at SkillsMatter here in London, on the 26th and 27th of June 2014. I'll be speaking alongside a great panel of .NET and distributed systems experts, including Udi Dahan, Ayende, Greg Young, Andreas Ohlund and many others. Follow @NSBCon on Twitter for more updates, and sign up at SkillsMatter if you're interested.

How to read FormsAuthentication.Timeout in .NET 3.5

2014-01-31T16:56:00+00:00

Forms authentication in .NET has a timeout property, controlled via the web.config section like so:

<system.web>
    <authentication>
        <forms name="foo" loginUrl="~/Account/Login" timeout="30" />
    </authentication>
</system.web>

This setting has existed since .NET 1.0, I believe, but only in .NET 4 did we get a corresponding property on the FormsAuthentication object. To access the Timeout property in .NET 4+, you just call

FormsAuthentication.Timeout

To do the same in .NET 3.5, you need to do this:

var defaultTimeout = TimeSpan.FromMinutes(30);
var xml = new XmlDocument();
var webConfigFilePath = Path.Combine(HttpRuntime.AppDomainAppPath, "web.config");
xml.Load(webConfigFilePath);
var node = xml.SelectSingleNode("/configuration/system.web/authentication/forms");
if (node == null || node.Attributes == null) return (defaultTimeout);
var attribute = node.Attributes["timeout"];
if (attribute == null) return (defaultTimeout);
int minutes;
if (Int32.TryParse(attribute.Value, out minutes)) return(TimeSpan.FromMinutes(minutes));
return(defaultTimeout);

I'll file this under "Things that wrong with Forms Authentication, #217"

Friday Puzzle Time! (aka EU Roaming Data Charges)

2014-01-17T20:22:00+00:00

I’m going to France. I want to be able to use my phone for data when I’m out there, so I’m investigating the cheapest way to do this. My phone is on Orange, who are now owned by EE, so I wander over to http://explore.ee.co.uk/roaming/orange/france and have a look.

(I should mention here that I’m generally really happy with Orange. The reception’s good. Their network doesn’t randomly drop calls or send me text messages 12 hours late, and Orange Wednesdays are awesome.)

Anyway. Here’s their data options:

Before we even get into the chaos of daily limits and terms and conditions, notice how the EU 100MB daily bundle gives you 100MB for £3, whilst the EU Data 500MB bundle gives you 500Mb for a hundred and fifty pounds? Same data but the bundle offers “93% saving on standard rates” – which is such a huge discount as to be really quite unsettling. If someone tried to sell you a MacBook for £50, you’d be a bit suspicious, right?

So I tried to work out what that catch might be – using EE’s website; Orange’s EU Mobile Internet Daily Bundles T&Cs, and a phone call to Orange customer services. Here’s what I found.

My Question	Website	T&Cs	Customer Services (referring to an internal T&C document dated Dec 15th 2013)
What happens if I go over 100MB in a single day?	“This bundle reoccurs up to 10 times until midnight of that same day (local time), each time the bundle reoccurs you will be charged an additional bundle fee and recieve additional bundle allowance. If you exceed the reoccuring bundle limit you’ll be charged 45.9p/MB.”” (Does this mean I get ten bundles per day? Or one bundle per day, for up to ten days?)	“Any data used above the amount in your bundle, will be charged at our standard data roaming rate of 45.9p per MB. So, for example, if you have the £1 per day bundle and use more than your 20MB you will be charged an additional £1 for every additional 20MB you use up to 200MB (as the bundle reoccurs ten times). Any data usage in excess of 200MB will then be charged at our standard data roaming rate of 45.9p per MB.” (that sounds to me like you get ten bundles per day – otherwise it would just say “if you use more than 20Mb you’ll pay standard rates for the rest of the day” – no?)	Customer services advised me that bundles only work once per day – so when you reach 100MB, you’ll pay normal roaming charges until midnight, and you’ll get another 100MB bundle the next day.
How many times can I use the bundle in a single day?	See above.	“A daily bundle will last until midnight (local time of the country you purchased the bundle in) on the day you made the purchase, or until you have used up your data allowance, whichever comes first in time. Once you’ve purchased a bundle, you will be opted in for the bundle to reoccur up to ten times” Doesn’t say if that’s ten times per day, or ten times per trip, or per billing period, or what.	Just once – see above. You can’t use two daily bundles in the same day.
How many days does it last for?	Really unclear whether it’s ten bundles per day – for as many days as you like – or whether it’s one bundle per day, for up to ten days. And no indication of whether that ten day limit is per trip or per billing period or what.	See above.	Customer services said there’s no limit – if I went to France for fifty days, I’d get fifty bundles. One per day, for the duration of my trip.
How much does it cost once I exceed my bundle limit?	“If you exceed the reoccuring bundle limit you’ll be charged 45.9p/MB.”	“Any data used above the amount in your bundle, will be charged at our standard data roaming rate of 45.9p per MB.”	£3.70 per MB. Though I was advised that information might be out of date.
What does the “ten bundle” limit actually mean?	Unclear	Unclear	No idea. Neither of the advisors I spoke to could tell me what the “up to 10 times” limit actually meant.

So, let’s spin those various answers into some possible phone bills, bearing in mind nobody can actually give me a definitive answer. Imagine we’re going to France from Feb 1st to Feb 16th, and we’re going to use 25Mb/day most days on e-mail and Facebook and the like, and 250Mb on Fridays, ‘cos we’re watching snowboarding videos on YouTube.

With 1 EU100 bundle per day, unlimited days – that’d cost us £185.70
At 1 EU100 bundle per day, up to 10 days – would cost us £270.98
The EU500 plan? Given our 500MB quota runs out halfway through our trip, we’d pay £322.13
If the chap on the phone was right about £3.70/MB, we’d be looking at £1,110 in excess data charges for our two nights of YouTube, and a total bill of £1,158.

And on top of all that, whoever wrote their (legally binding?) terms & conditions cannot spell ‘receive’ or ‘recurring’.

No wonder I have a headache.

UPDATE: So when I texted EU100 to 2088 to activate the bundle, it failed… I rang them (again), and apparently there’s already a bundle on there. It’s been on there since February 2013. Will it run out? “No, sir”. Is there any kind of limit on how many days I get? “No, sir.”

So that’s an hour of research and phone calls to work out that nobody knows for sure what the bundle is, but it’s probably safe to assume it *doesn’t* run out because I’ve already got one and it’s been active for nearly a year.

O_o

The Joy of GUIDs

2013-12-17T13:17:00+00:00

Like many developers, my first experience of primary keys was the Autonumber field in MS Access. At the time, this looked like a pretty neat idea. When I outgrew Access and moved on to using SQL Server, I adopted the convention of an identity(1,1) field as a primary key - a pattern that's used extensively both within my own code and throughout the industry.

Recently, though, I've been working on big distributed systems where the identity field really doesn't work too well. The biggest drawback is that if you create a new entity in any of your data stores, you need to wait for the response from the database before you can find it again. For RESTful systems where your Create method might just return an HTTP 202 "Accepted for processing" and leave you to work out whether your request eventually succeeded or not, this approach just plain doesn't work. Enter the GUID - globally unique identifier. The idea is that every GUID ever created is completely unique - it uses various system/hardware IDs, the system clock and a degree of randomness to create unique 128-bit values. The lovely thing about working with GUIDs is that when you create your new customer, or order, or invoice, YOU can assign the ID that will identify the object for the rest of its life.

Thing is, working with GUIDs is very, very different to working with ints. For starters, a lot of security practises around integers just don't apply.

A great example is URL-hacking. If you've ever seen an address in your browser's address bar that looks like:

http://www.myshop.com/basket.asp?basketid=2789

Try editing it. Try changing the query string to be basketid=2788 - on many (poorly written!) systems, that'll show you what's in someone else's basket. It doesn't take a lot of work to write a little script that'll GET all the basket IDs from 0-999999 and voila! you've got a folder full of other people's shopping baskets. Even if the developers do something terribly clever - like using identity(27897,17) instead of identity(1,1) - if you can do ten HTTP requests per second, you can still sweep the entire ID range from 0-1,000,000 in just over 24 hours. Sure, only 1 in 17 of your requests will succeed - but who cares? That's still 58,823 compromised shopping baskets.

Now, imagine instead of sequental integers, you use a GUID:

http://www.myshop.com/basket.asp?basketid=fa8f6423-e7d6-ff40-7afc-ae874c755c00

Try hacking it. It won't work. There are 3.4×10³⁸possible GUIDs, and roughly seven billion people on earth. If every single person on the planet used your website every day, at the end of a year you'd have 10,000,000,000 * 365 shopping baskets in your system. Let's see how long it would take an attacker to steal those baskets by URL-hacking:

Number of possible basket IDs B_p = 3.4x10^38

Number of actual basket IDs B_a = 3.65x10^10

Likelihood of a randomly chosen GUID finding a shopping basket = B_p/B_a = 1.0x10^-28

Let's assume the website we're attacking is running on some serious distributed hardware so it'll remain responsive throughout our attack, and you have a massive botnet so you can check baskets until the server starts to struggle under the load. Facebook manages 6 million page-views per minute, or 100,000 requests per second. At 100,000 requests per second, it will take you (on average) 2,953,788,842,323,276 years to find a single shopping basket by URL-hacking. Given those odds, a lot of security concerns around vulnerabilities like URL-hacking just disappear.

I've sat through numerous conversations that get onto "what'll happen if we get a GUID collision?"

There's really only one sensible answer to this: You'll find the place in your code where you're accidentally re-using the same GUIDs, and fix the bug. GUIDs don't collide - you're doing it wrong. If you're using Guid.NewGuid() or NEWID(), you won't get collisions. It just won't happen. There isn't even any meaningful analogy with daily human experience that can convey just how unlikely a GUID collision is. Things with a probability of 1.0x10^-28just don't happen.

Automatic Semantic Versioning with GitHub and TeamCity

2013-10-11T19:20:00+00:00

You’ve quite possibly come across the idea of semantic versioning. It's a set of rules for versioning releases of software, designed to minimise disruption to people who are relying on your APIs not to change. In a nutshell, every release of your software has a three-part version number.

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,

MINOR version when you add functionality in a backwards-compatible manner, and

PATCH version when you make backwards-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Pretty simple, huh? In other words, 2.3.4 should offer the same feature set as 2.3.1 but with extra bug fixes. 2.4.0 is backwards-compatible with 2.3.2, and with 2.2.*, and with 2.1.*, and when you go for the ‘big rewrite’ and break everything, that’s version 3.0.0 which isn’t compatible with any previous versions.

Our main development pipeline is now based on TeamCity 8 and GitHub – both of which are absolutely wonderful once you get the hang of using them properly – and I wanted to automate the versioning of our builds according to semantic versioning principles. Here’s the workflow I settled on:

Every build has a MAJOR, MINOR, PATCH, and BUILD version, and an optional SUFFIX
Each part of the version number is managed as a TeamCity build parameter.
MAJOR and MINOR are managed manually. TeamCity can’t tell whether your commit is a feature or a bugfix, so for now we’ve just settled for manually changing the relevant parameters when you release a major or minor release.
All development is done on feature branches and submitted via pull requests.
Any pull request is assumed to be a PATCH release unless a developer changes the major or minor numbers when they accept it.
The SUFFIX is used to denote a pre-release package, and should indicate which branch’s features are included in that prerelease
- e.g. Dylan.Web-1.2.3.0-branch7 should be the code you’d get if you accepted branch7 into master 1.2.3
Anything built off master should be releasable, and tagged as such
Finally, whenever a production release is built, we’ll tag the GitHub repository with the version of that release – for tagging, we ignore the build number, so the tag will be something like v0.0.0

This lot actually distils to quite a simple guideline: The patch number is the number of pull requests accepted since the last tagged minor release.

Now that sounds like something you could automate… so with Powershell on one screen and the Github API documentation on the other, I hacked together the following script, which is now happily running as a build step on our TeamCity server.

$MajorVersion = %MajorVersion% # The major version of your current build
$MinorVersion = %MinorVersion% # The minor version of your current build
$RepoUrl = "%vcsroot.url%" # The HTTPS path to your repo – https://github.com/MyCompany/MyRepository.git
$Token = "%GitHubAccessToken%" # Your GitHub access token, from https://github.com/settings/applications

try {
    # Parse the supplied Git repo URL to determine the repo name and owner
    $RepoUrl -match "https://github.com/([^/]+)/(.+)\.git$"
    $repoOwner = $matches[1]
    $repoName = $matches[2]
    Write-Host "Reading repo $repoName owned by $repoOwner"

    # Find the tag resource matching the baseline of the current major/minor version
    # Remember, we’re not looking for X.Y.Z, we’re looking for X.Y.0 since patch numbers
    # should increment until the next minor release.

    $uri = "https://api.github.com/repos/$repoOwner/$repoName/git/refs/tags/v$MajorVersion.$MinorVersion.0?access_token=$Token"
    Write-Output "Looking for tag v$MajorVersion.$MinorVersion.0 at $uri"
    $tag = Invoke-RestMethod -Uri "$uri"

    # $tag.object.url will now give us a more detailed tag resource, including the commit that was tagged
    $uri = $tag.object.url
    Write-Output "Getting tag info from $uri`?access_token=$Token"
    $tag = Invoke-RestMethod -Uri "$uri`?access_token=$Token"

    # $tag.object.url is now a link to the tagged commit
    $uri = $tag.object.url
    $tag = Invoke-RestMethod -Uri "$uri`?access_token=$Token"

    # now we can dig into the commit itself and find out WHEN the baseline release was tagged...
    $since = $tag.committer.date

    # Now we can retrieve all the commits in this repo SINCE that date
    $commitsUri = "https://api.github.com/repos/$repoOwner/$repoName/commits?since=$since&access_token=$Token"
    Write-Host "Retrieving commit log from $commitsUri"
    $commits = Invoke-RestMethod -Uri "$commitsUri"
    $merges = @($commits | Where-Object { $_.commit.message -match "^Merge pull request" })

    # Reversing the merges just means they show up in TeamCity’s build log in chronological order.
    # Which is nice.
    [Array]::Reverse($merges)

    $mergeCount = $merges.Count
    Write-Host "Found $mergeCount merges since last release tag"
    for($i = 0; $i -lt $merges.Count; $i++) {
        $merge_number = $i+1
        $merge_message = $merges[$i].commit.message.split("`r`n",[StringSplitOptions]"RemoveEmptyEntries") -join "`r`n    "
        $merge_sha = $merges[$i].sha
        $merge_web_url = $RepoUrl.Replace(".git", "/commit/$merge_sha")
        "`r`n Merge #$merge_number`: $merge_web_url"
        "    $merge_message"
    }
    Write-Host "`r`n"
    Write-Output "##teamcity[setParameter name='PatchVersion' value='$mergeCount']"
} catch [System.Exception] {
    Write-Host "Exception trying to determine patch number from Github API"
    Write-Host $_
    Write-Host "Using default patch number 0"
    Write-Output "##teamcity[setParameter name='PatchVersion' value='0']"
}

A couple of fun details to look out for:

If we’re doing the first build of a new minor version, there won’t be any ‘baseline’ tagged version yet – which is fine; we just set the patch number to zero and off we go.
You can control TeamCity build parameters by outputting specially-formatted messages in your build log – that’s what those lines

Write-Output "##teamcity[setParameters name='SomeParameter' value='NewValue']"

are doing. See Build Script Interaction with TeamCity for more info.

Finally, if you'd like the commit/merge hyperlinks in your TeamCity build log to be clickable, check out this awesome tip from Christian Rodemeyer about how to embed clickable URL links in TeamCity build logs.

Happy versioning!

Building a ServiceStack-based OAuth2 resource server using DotNetOpenAuth

2013-08-19T21:07:00+00:00

Last week, I asked on StackOverflow if anyone had used DotNetOpenAuth to secure a ServiceStack API using OAuth2. Well, it would appear not… so alongside a week of band rehearsals and a big old party last weekend, I’ve put a proof of concept up onGithub that demonstrates how to use ServiceStack and DotNetOpenAuth together to create a REST API that’s secured using OAuth2. The app I’m working on is aimed specifically at existing customers, who will already have a login account on our site, so I don’t need to use OpenID at all. In fact, I spent a good few days puzzling over this before I realised my mistake. After digging through the various specs and innumerable blog posts, I decided to use the OAuth2 authorization framework – specifically, a feature of that framework called the resource owner password credentials grant

The DotNetOpenAuth project includes a couple of great OAuth2 samples, including an authorization server (the bit that does the customer login), but they’re built primarily around ASP.NET MVC or WCF. I’ve worked with both of these in the past, but recently I’ve switched to using the ServiceStack library to build ReST APIs, and I absolutely love it. It’s easy to use, easy to extend, and wonderfully lightweight, so I wanted to use ServiceStack to build the data API that our app is going to use. I assumed it wouldn’t be too hard to build our own authorisation server, and then use a standard protocol like OAuth to control access to the resources exposed by this server.

There’s a working prototype / proof of concept up on GitHub now, at https://github.com/dylanbeattie/OAuthStack – complete with commentary on how the various integration points worked. OAuth2 is actually conceptually quite simple, but getting the two libraries to work together proved a little fiddly; both ServiceStack and OpenAuthDotNet use their own abstractions over the ASP.NET request/response objects, making me really wish Microsoft hadn’t made the .NET HTTP framework quite so inflexible back in the day… but after a week or so of tinkering, testing, poring over RFCs and bouncing around in the Visual Studio debugger, I’ve got something that seems pretty lightweight, doesn’t reinvent the wheel, and seems to satisfy the requirements.

Suffice to say that if DotNetOpenAuth and ServiceStack weren’t open source this kind of integration would have been practically impossible, so I want to say a huge thank you to the authors and maintainers of both of those projects. Thanks, guys. You rock :)

Linking Subversion Commits to Pivotal Tracker on Windows using Powershell

2013-06-24T12:03:00+00:00

My team at work now keep our entire lives in PivotalTracker, and rather like it. We recently linked it to our GitHub repository using the built-in PivotalTracker integration that’s provided by GitHub, which worked really nicely. Thing is, we’re still in the process of migrating our codebase to GitHub – there’s lots of older projects with TeamCity jobs still linked to our in-house Subversion repo – and I wanted to add a similar facility for our projects that are still in Subversion.

This is all based on the Source Control Management Post-Commit Hook Integration feature in the Pivotal Tracker API, and getting it working required two scripts.

post-commit.bat – this is the standard batch file used by Subversion on Windows to run code after a successful commit. Subversion runs this automatically, and passes in two command-line arguments – the (local) path of the repository that accepted the commit, and the revision number created by that commit. Ours now looks like this:

@echo off
$ps = %SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe
$ps Set-ExecutionPolicy unrestricted
$ps -command "D:\svn\data\repositories\projects\hooks\post-commit.ps1" -repopath %1 -revision %2

A couple of things to notice. First, we’re running the powershell Set-ExecutionPolicy as part of the script. This is brute force and there’s probably a nicer way of doing it, but since the Powershell script runs as the user who owns the Subversion server process, and that user doesn’t have interactive logon rights, I couldn’t see a quicker way of getting that script to run reliably. Ah, Powershell security, how we love you.

Second – we’re passing the two command-line arguments into the powershell script as named parameters. Subversion will call the batch file as:

post-commit.bat D:\svn\data\repositories\projects 12345

and we’re translating that into

powershell.exe –command post-commit.ps1 –repopath D:\svn\data\repositories\projects –revision 12345

The bit that actually does the heavy lifting is post-commit.ps1

param([string]$repopath, [string]$revision)

$svnlook = "C:\Program Files\Subversion\bin\svnlook.exe"

$message = & "$svnlook" log $repopath -r $revision | Out-String

if ($message -match "#\d\d\d\d\d\d\d\d") {
    $reponame = Split-Path $repopath –leaf
    $author = & "$svnlook" author $repopath -r $revision | Out-String

    # Grab a reference to System.Web so we can HtmlEncode things.
    Add-Type -AssemblyName System.Web
    $author = [System.Web.HttpUtility]::HtmlEncode($author)
    $message = [System.Web.HttpUtility]::HtmlEncode($message)

    # We run ViewCV on our Subversion server to browse code via the web. Tweak this to suit,
    # or remove the <url></url> element from the XML completely.
    $url = "http://subversion.mycompany.com/viewvc/$reponame`?view=revision&revision=$revision"

    $headers = @{"X-TrackerToken" = "put your Pivotal Tracker API Token Here"}
    # This is Powershell's "here-string" syntax for multiline string literals.
    $body = @"

<source_commit>
    <message>$message</message>
    <author>$author</author>
    <commit_id>$revision</commit_id>
    <url>$url</url>
</source_commit>
"@

    $r = Invoke-WebRequest -Uri http://www.pivotaltracker.com/services/v3/source_commits -ContentType "application/xml" -Method POST -Headers $headers -Body $body

    Write-Host $r
} else {
    # if nothing in the commit message matched the #12345678 syntax, then don’t bother calling the API.
}

To test it, first run the Powershell script directly from your command line. Say you’ve just committed revision 12345 to the “projects” repository, remembering to include [#12345678] in your commit comment where 12345678 is the PivotalTracker story ID.

PS D:\svn\data\repositories\projects\hooks> ./post-commit.ps1 –repopath D:\svn\data\repositories\projects –revision 12345

Your commit comment should appear in the story history within a few seconds.

Now check the batch file harness can run your Powershell properly – see the note below about revisions:

D:\svn\data\repositories\projects\hooks> post-commit.bat D:\svn\data\repositories\projects 12346

If that works, the whole kaboodle should work. A couple of caveats to watch out for:

If your post-commit hook fails, your Subversion client will give you something cryptic about “Commit failed: MERGE … 200 OK” – which is not particularly transparent. Best to get things working on the command line first before testing via Subversion itself.
I couldn’t get the same commit to produce multiple Pivotal tickets – when I was testing it, I had to create a fresh commit for each test run of the integration hook. This is possibly just Pivotal automatically de-duplicating identical stories – but worth knowing for test purposes

Adventures in Querying with RavenDB

2013-03-23T17:01:00+00:00

For the last couple of months, we've been working on a project that uses RavenDB as the main data store. We went with Raven for several reasons - it looked like a pretty good fit for what we were doing, we were using NServiceBus which now includes Raven as its default persistence store, and we were keen to see what it could do. Generally, it's been really, really positive. However, RavenDB is the first document database I've ever used, and so once in a while the learning curve just leaves me scratching my head in frustration...

So, last week, a request comes in for a straightforward one-off report. We're using Raven to store details about media clips - video and audio files - that are linked to our customers' online CVs, and the product owner wanted a list of customers along with the total duration of the media clips linked to those customers' CVs.

In SQL, this would be

SELECT Customer, SUM(Duration) as TotalDuration FROM MediaClips GROUP BY Customer

The two developers who actually built the system weren't in the office last week, so this request ended up on my desk. "Ah, no problem", thinks me. After all - it's a database; how hard can it be? I'd spent a bit of time building custom indexes to drive an SLA report we built last week, and with quite a lot of Googling, managed to get some really impressive results out of Raven.

Thing is - I'm not building a feature here, I just need to copy & paste a bunch of numbers into a spreadsheet and get on with my life. And I have no idea how you do that in Raven. I spend an hour or so trying to get Ronnie Overby's RavenDB provider for LinqPAD to install. This is a third-party library that isn't part of RavenDB, but I like LinqPAD, I'm pretty comfortable with Linq query syntax, and this looked like a good place to start. Half an hour later, after downloading the source, compiling it, creating a certificate, signing it and getting LinqPAD to recognise and install the driver, I gave up because I just couldn't work out how to get any data out of the damn thing.

Instead, I throw together a .NET console app and start playing around with indexes, working from "Using Linq to query RavenDB indexes". I get as far as creating a named map/reduce index that's pulling out the data I need - this takes a good hour or two of trial and error. The eureka moment is when I work out that you can define Raven indexes as raw LINQ code - they're not even embedded strings, they're actually strongly-typed, compiled Linq expressions that are passed in to the query definition. I get as far as this:

class Total {
public string CustomerId { get; set; }
public int Duration { get; set; }
}

public void BuildIndex() {
var store = new DocumentStore() { ConnectionStringName = "raven" };
store.Initialize();
store.DatabaseCommands.PutIndex("ClipInfos/TotalDurations", new IndexDefinitionBuilder<ClipInfo, Total> {

    Map = clips => from clip in clips select new {
    Customer = clip.CustomerId,
      Duration = clip.DurationInSeconds
    },

    Reduce = results => from result in results group result by result.Customer into g select new {
      Customer = g.Key,
      Duration = g.Sum(x => x.Duration)
    }
});
}

Now, Raven encourages "safe" query patterns. Which means it'll only give you 128 records per query by default, and if you run more than 30 queries in a single session it'll fail with an exception telling you you're doing something dangerous. The only way I can find to actually retrieve all the data I need is to circumvent these defaults to increase the batch size from 128 to 1024, and then repeated use .Skip(1024*batch).Take(1024) to pull records back 1,024 at a time and add them to my result set.

using (var session = raven.OpenSession()) {
session.Advanced.MaxNumberOfRequestsPerSession = 512;
while (true) {
    var totals = session.Query<Total>("ClipInfos/TotalDurations").Skip(1024 * batches).Take(1024).ToList();
    foreach (var total in totals) {
      // Do something useful.
    }
    if (totals.Count() < 1024) break;
    batches++;
}
}

At this point, I'm thinking one of two things:

This is fine. It's an ad-hoc requirement, and if we end up doing this a *lot* I should read up on the index replication bundle - a plug-in that, I gather, will export indexed data into a relational DB for easier querying.
This is completely wrong. It should not take three hours and a dedicated console application to run a simple ad-hoc query against a set of Raven data. I am missing something obvious...

As somebody who's been writing SQL for nearly 20 years, it's really frustrating to hit a brick wall like this... and it's been a long day, and I'm getting fed up, and I wander over to Twitter to have a bit of a rant and @ayende pops up:

Ah. Dynamic Queries. That looks interesting. So I head over to Raven Studio and sure enough, there's a "Dynamic Queries" facility. So I paste in a simple query and hit "Execute":

No results? Huh? But there's tens of thousands of documents in there! At this point I give up and go to the pub.

This morning, I go back to it and notice the little "i" icon next to the Query header, and see this...

Ah. I have no idea what Lucene syntax is... but that probably explains why none of my dynamic queries worked.

Raven is really impressive. We've managed to get it to do some very cool things; it's fast, and most of the time it Just Works. And there is an inevitable learning curve associated with adopting a new technology - particularly one that represents a paradigm shift in approach. It's kinda like learning Powershell and treating it as another CLR/.NET language... which is really, really frustrating, until you realize it's not "scriptable .NET", it's more like command.com on steroids. The shift from relational to document databases is much the same. That said - it would be really, really useful if the dynamic query tool in Raven Studio would distinguish between "I understand that query but could find no matching records" and "I have no idea what you are trying to do" - I have no idea how strict the Lucene query language or parser is, but knowing whether your query is syntactically valid or not would be a big help when you're trying to work out why it didn't return anything.

Adventures with Powershell and Amazon EC2 (part 1)

2012-12-03T13:18:00+00:00

Managing servers isn’t fun. Setting up servers isn’t fun. I don’t want to do it any more, ever – it’s tedious, repetitive and error-prone, and, dammit, we have machines to do those kinds of jobs these days.

So, here’s what I’m trying to do. I want to be able to spin up a blank Windows instance on Amazon EC2 and deploy our apps onto it without ever logging into the box directly. This means no remote desktop sessions, no VNC, no SSH – nothing. These blog posts are as much a diary of work in progress as anything else… I’ll probably make mistakes and get things wrong along the way, but hopefully I’ll end up with something usable.

Step 1: Hello, World!

Prerequisites:

An Amazon EC2 security group with firewall rules accepting all connections from your local network.
A fresh Windows instance running in that security group, bound to an Amazon elastic IP
The Windows Administrator username and password for that instance.

Create this script on your local workstation:

$remoteUsername = "Administrator"
$remotePassword = "your Amazon EC2 instance Administrator password"
$remoteHostname = "the elastic IP bound to your Amazon EC2 box"

$securePassword = ConvertTo-SecureString -AsPlainText -Force $remotePassword
$cred = New-Object System.Management.Automation.PSCredential $remoteUsername, $securePassword

Invoke-Command -ComputerName $remoteHostname -Credential $cred -ScriptBlock {
Write-Host "Hello, World (from $env:COMPUTERNAME)"
}

Run it. You should get:

Invoking command on Remote host X.X.X.X
Hello, World (from AMAZONA-???????)

If that works, you’ve got Powershell remoting working against your Amazon EC2 instance. Hurrah!

The Joy of Forms Authentication, IIS7 and Device Profiles

2012-10-30T20:08:00+00:00

We’re setting up a thing called PRTG Network Monitor to add monitoring and logging to one of our new MVC3 web applications, and I spent this morning scratching my head and wondering why an app that works just fine when you view it in a browser was consistently failing when I tried to attach an HTTP monitor to it.

A little digging with Wireshark later, and it turns out that IIS/ASP.NET isn’t sending the FormsAuthentication cookie in the response to requests made via the PRTG monitoring software. Which is… odd. We haven’t explicitly configured cookieless authentication or anything – although we’re doing some funky stuff with cross-domain auth cookies, the PRTG monitor is as simple as it gets – POST credentials to the login handler, get a response, check the response includes a cookie. Not rocket surgery.

After a fun hour of comparing TCP traces and HTTP headers, it turns out it’s the HTTP User Agent that’s controlling this behaviour – if ASP.NET running under IIS7 sees particular user agents, it just doesn’t set any cookies. And this is where it gets interesting. Here’s some real-world HTTP user agents that work just fine:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0);

Opera/9.80 (Windows NT 6.1; WOW64; U; en) Presto/2.10.289 Version/12.02

Here’s the PRTG monitoring software’s default user agent. Now this looks to me like a completely sensible user agent for a piece of monitoring software – but IIS7 clearly believes this to be a client that can’t handle HTTP cookies, and so won’t set any:

Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)

But the really fun part – here’s a bunch of user agents that work absolutely fine:

Bozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)

Batman/1.2 (compatible; spiderman; vigorous jazz-hands)

CAKE CAKE CAKE CAKE CAKE I LIKE CAKE

CookieHater1.2 (incompatible; does not accept cookies; no cookies; cookies are evil)

This doesn’t work:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML)

But these do:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like a BOSS)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like anyone cares)

In short - it looks like any string that starts with “Mozilla” triggers some sort of complicated parsing algorithm that looks for magic values in the user-agent string and decides that means they’re cookie-capable. I have this horrible, horrible feeling that somewhere in the bowels of the IIS team there’s somebody who still thinks browscap.ini was a good idea – and I’m amazed that something as fragile and idiosyncratic as determining device capabilities based on user agents would be the default behaviour in IIS7 and ASP.NET 4.

The good news is that this bizarre behaviour is easily fixed – just add cookieless="useCookies" to the system.web/authentication/forms element of your web.config file. The default is apparently “useDeviceProfile”, and there’s more details in the IIS documentation on Microsoft Technet.

This is also another classic example of how easily you can lose an entire day of developer time to something that should have taken about ten minutes. We have an app that uses HTTP and out-of-the-box cookie-based authentication. We’ve got something approaching 500 tests verifying the behaviour of this app under every scenario we could think of. We have a mature, stable monitoring platform that hundreds, if not thousands, of people are using to monitor HTTP applications every day. And yet BANG! you connect them together, everything falls apart and next thing you know you’re up to your elbows in network packets and wondering if it’s too early in the week to start drinking…

UPDATE: Thanks to @duncansmart for pointing me at this article on Scott Hanselman’s blog where he discusses this very problem.

Rocksmith for Guitar Players

2012-10-22T23:50:00+00:00

OK, having finally convinced those necrotic turbinates at Ubisoft that I'm not a dirty filthy pirate (yarrr!) I have an install key for Rocksmith. And I've installed it. And I've played it. Now, I'm not a gamer. I've played around with Guitar Hero and gone "meh"... But I have been playing guitar for 25 years and wasted more hours than I care to recall plugging assorted guitars into various kinds of computer gear to try and make them do interesting things, so I was really curious to see how Rocksmith did it.

First impressions?

It looks fantastic. Lovely atmospheric visuals, stylistically quite beautiful. Second impression? Lag. Seriously. The folks telling you you won't notice 30ms of latency aren't musicians - they're marketing shills who work for the computer games industry. Having 30 milliseconds of delay between your fingers and your ears is horrible - after ten minutes, it was actually making me feel ill. Kinda like when you're on a roller-coaster and your eyes are disagreeing with your inner ear about which way is up - the same kind of slightly nauseating sensory disconnect. Yuck.

Eliminating Latency

So... just when I was about to give the whole thing up as a failed experiment, I had a daft idea. I plugged in an electro-acoustic and turned the guitar volume in the game right down - I figured the game can play the backing track and the guitar can look after itself, sonically speaking. The experience was instantly transformed - the game engine itself doesn't seem to suffer from any kind of latency, and suddenly I was jamming along to the Stones with a big stupid grin on my face. So I started thinking - maybe there's some way I could link up my existing guitar gear so I can get the same zero-latency experience but with an electric guitar and all the tones and presets in my trusty Line6 PodXT... easy. All you need is something to split the signal as it leaves the guitar. Now although I've not tried, I suspect an ordinary jack splitter would do the trick - plug it into your guitar, then plug the Rocksmith jack-USB cable into one output, and connect the other to your amp or effects rig using a normal guitar lead.

Problem is, I didn't have anything as simple as a splitter amongst my boxes of audio gear... so I ended up doing this instead:

The UA25EX is a really good USB audio adapter, but here I'm just using it as an active splitter, piping one output into the PC via the Rocksmith lead and then running the other one into the PodXT; the Pod's outputs are wired to the PC line-in, and then I've tweaked the Windows volume control so that the line-in is mixed back into the playback signal. Finally, remember to turn the guitar volume in Rocksmith down to zero, and voila - Rocksmith does the backing, Pod does the guitar sound with zero latency, and I'm back to grinning like an idiot. (Yes, I realise I'm using about £1500 worth of gear as a glorified gamepad... but hey, it's not like I bought this gear just to play computer games like Jeff Atwood over at CodingHorror did...) One more thing I changed - under Options -> Game Settings -> String Layout - set this to inverted. It wasn't obvious at first, but the standard setting mimics the physical layout of guitar strings (i.e. treble at the bottom, bass at the top) while guitar tab and normal notation are written the other way up - things suddenly got a lot more fun once I tweaked this.

So... latency solved, is the game any good?

Well, yeah. It is. And I'm really impressed at how well it equates improvement in guitar technique with in-game progress. Using a combination of training exercises, arcade mini-games and full band backing tracks, it'll take you from tuning up, to playing simple one-note riffs, to slides, bends and hammer-ons. If you can already play, you'll be amazed at how quickly it exposes your weak spots and bad habits - and if you can't, I suspect you'll be really surprised at how quickly you progress. The techniques are clearly demonstrated on-screen, there's a built-in tuner that pops up at regular intervals to make sure your guitar's still in tune, and even following the on-screen color-coded notation isn't too hard - as a guitarist who never learned to sight-read, I was surprised how quickly I started to recognise patterns and phrases as they flew down the screen towards me at 140bpm. It also has distinct guitar and bass modes - including an "emulated bass" mode if you want to learn to play basslines but you've only got a six-string guitar. No five-string bass support, though... here's hoping we get that in Rocksmith 2.

The Guitarcade mini-games deserve a special mention, if only for the stupid amount of time I've spent playing Super Slider. Coloured blocks fall down the grid... you move them left and right by playing slide notes on the guitar. It is stupidly addictive and - because it requires you to change strings and positions without taking your eyes off the screen - it's also great for improving fretboard technique.

My one frustration is that the game constantly adjusts the difficulty, so there's no way - that I've found, anyway - to just bring up a song on "accurate" difficulty setting and keep practising it until you get it right. A couple of times I actually managed to move up a level just by noodling around and throwing in notes that weren't on the screen - turns out I was getting it right (?) and so the difficulty increased accordingly. I can imagine this being great fun if you already know the tune and you're working on your guitar technique, but it'd be nice if you could just switch it off and dive in at the deep end, so to speak.

In conclusion: great visuals, great gameplay, great implementation. And it actually *works* - once I'd solved the lag problem, I never once felt like I'd hit a note and the game had missed it, which is really impressive when you consider it's analysing the ragged mess of waveforms coming off a six-string guitar in real-time. Sure, the lag sucks and the missing CD keys and crappy attitude really sucks, but neither of those problems is insurmountable with a little patience and ingenuity. And once you get up and running, it's great fun. I can't wait to hear what happens when a generation of kids who grew up on this move onto GarageBand and start recording demos.

Ripping Off Paying Customers the Ubisoft Way

2012-10-19T21:47:00+00:00

I got into work this morning to find Dave excitedly waving a box containing something called Rocksmith - which, it turns out, is a "Guitar Hero"-type game, but you play it using a real guitar. Now, whilst I'm not a serious gamer, I am a tech nerd, and a guitar geek, and this sounded like the best thing ever. If nothing else, for the first time in 15 years, it might actually be a computer game I could beat my brother at... So, this lunchtime, I left the office, wandered to HMV on Piccadilly, and bought it. No download, no Steam, no piracy or dubious torrenting, just good, old-fashioned law-abiding retail. I walked into a physical store, picked up a boxed product, paid £39.99 for it, and left.

Now, here's what the guy in HMV did NOT say when I bought it. He didn't say "Keep the receipt because you'll need to send a copy of it to Ubisoft in order to install your game". He DID offer to sell me "game protection" for a pound, which sounded like (a) a way of charging me money for my own statutory rights, and (b) the biggest rip-off I'd ever heard - ha, maybe he knew something I didn't. But no, after I'd paid for my game, he wished me a pleasant weekend and off I went. The receipt went in the bag, along with the game, and, by the time I returned to work, some samosas and a Chinese pork bun. When I got back to the office, I put the game in my rucksack, the samosas and the pork bun in my belly, and stuck the carrier bag in a drawer. Why am I mentioning this? Ah, just wait and see...

So I get home around nine this evening and open it - look at all the things! It even comes with a jack-USB cable to plug your guitar into your PC - that's cool! OK, and a sheet of rather lame-looking fret stickers. I shudder at the thought of someone sticking these on a '57 Les Paul Standard that they've only dragged out from under the bed because they think Rocksmith might FINALLY get them playing it...

Anyway, I digress. I go to install it, and I'm asked to enter a CD key. This is pretty normal.

OK... on the jewel case? No. Manual cover? No. Disc insert? Under the disc? No. Oh dear. This is invariably where things get unpleasant. I Google rocksmith CD key and - wow, it looks like I'm not the only one. The top link is to a Ubisoft support article published 4 days ago:

Which links to this article, which says:

I try giving them a call, but "unfortunately [their] offices are now closed". So it looks like that support article is all I've got until Monday morning. I love the "encountering problems during installation". No, it's not a "problem during installation", it's that the manufacturer screwed up and omitted to include an essential part of the product in the retail packaging. And they'd like a copy of the receipt. Which, as I've mentioned, is in a bag eight miles away.

So, leaving aside that little detail, let's see how far we get. First I have to submit a request. I can't do this without signing in, I can't sign in without creating an account. Of course, this doesn't work first time because their sign-up/login system has 'issues'. Eventually I get as far as submitting a support request:

OK, question submitted at 21:52 UK time. And now, I guess, I wait and see what happens. They aim to respond to me "within 24-48 hours, although response times may be longer." Looks like I won't be playing Rocksmith this weekend.

Let's just recap. So far, I've gone out in the rain to purchase a product because it looks like fun. I've spent £39.99. I've spent ten minutes searching through packaging for a non-existent key. I've had to create an account in order to politely request that the manufacturer allow me to use the product. Ubisoft have ~~effectively accused me of having stolen the game~~ assumed I'm not a legitimate customer and I therefore can't install it until I prove otherwise. This isn't a minor manufacturing defect affecting a single unit, like a cracked picture frame or a corked bottle of wine - this looks like they've just deliberately excluded the installation key from boxed retail copies of the game.

What's completely ridiculous is that this is one of the few games in recent history that requires a physical component - the aforementioned USB-guitar cable. They could so easily have encoded a unique key in the cable and used that as a copy protection dongle - after all, you can't download a cable from the Pirate Bay. But no, instead they go for a CD key and don't bother to include one. I appreciate game development is a hugely involved and expensive exercise, and that piracy has the potential to seriously undermine the commercial viability of game development - but treating your customers like criminals to compensate for your own short-sightedness and lack of imagination is not the answer.

Next time I want to spend £40 on something fun to do over a rainy weekend, I'll forgo the delights of proving my innocence to the games industry and just buy some more Lego. At least I've never had to grovel to the manufacturer for permission to start building a Lego set.

UPDATE: E-mail from Ubisoft on Saturday afternoon saying they can't/won't help until I send them the original purchase receipt:

Oh well.

UPDATE 2: I e-mailed them a photo of the receipt on Monday morning at 10am; they e-mailed me a CD key at noon. I wonder whether they actually verify the proof-of-purchase details against the retailers’ records or are just under instructions to give out CD keys to anyone with a convincing-looking receipt…

SECURITY_DENIED_BY_MIMEMAP Hosting WCF Services in IIS 7 Classic Mode

2012-08-18T10:55:00+00:00

Another one of those gotchas where the error message doesn't actually have anything to do with the problem...

One of our apps includes a WCF service endpoint that's called by some of our other systems. This works perfectly in production, but when I fired it up locally this morning to fix a bug, I just couldn't get the thing to run properly - the right handlers were mapped, the file permissions were OK, but every time I pointed a browser at http://www.mysite.com.local/some_app/endpoint.svc , I got an HTTP 403 Forbidden.

Turning on IIS failed request tracing gave me one of those incredibly detailed XML dump files:

So... SECURITY_DENIED_BY_MIMEMAP. Which is a little odd, because I was pretty sure MIME mapping only applied to static content - but sure enough, I tried setting up a MIME mapping for .svc files - mapped to application/xml. This didn't work - IIS just sent me the .svc definition file instead of actually executing it...

Turns out that the problem is our production servers are 32-bit and our development boxes are x64 - and I hadn't gone into the IIS Application Pool for this app and enabled 32-bit applications.

Quite why this results in the misleading MIME type error is one of those little mysteries that makes IIS such fun to work with, but that solved it for me.

Planning Poker… with LOLCATS

2012-08-03T12:36:00+00:00

So… there’s this thing called planning poker, which although it looks a little odd at first, is actually a really effective way to flush out uncertainty during software planning meetings. It’s played with a special deck of cards, and for the last year or two we’ve been using a couple of decks that I got at a SkillsMatter event.

Now, in normal poker, all the cards need to be from the same deck – otherwise you could read the card backs. Obvious. But with planning poker, only the cards in your own hand need to be the same – there’s no reason why your deck needs to be the same as the other people in the planning session – so there’s nothing stopping you making your own planning poker cards. So, thanks to a bit of interwebbing, a little bit of magick and those wonderful people at MOO.com, I give you my one-of-a-kind LOLcat planning poker deck.

Whilst I’m here, I want to say a public thanks to MOO for their outstanding customer service… I got this e-mail from them on Tuesday:

Dear Dylan,

I'm afraid we have been experiencing some problems rendering your order. I think I have managed to fix it now, but we had to enlist the help of our developers to ascertain the cause of the problems. As I can see these are Agile estimation cards, I thought I'd share the more complex development solution to the problem :-)

Basically, the PNGs are a bit odd: all but the first two have offsets that position our canvas outside of the actual image. Our cropping implementation obeys the offsets, and finds that cropping produces a 0 width image. This is bad. You'll find that if you open the images with GIMP or, probably, Photoshop it'll report that something a bit odd is going on. Unfortunately some software (browsers, the flash) either helpfully ignore the duff offsets, or just don't support offsets at all (so mask the problem.)

Now for some guess work, the offsets in the PNGs suggest that they were originally all part of one large (2080x2340) image. Did you perhaps use an automated process to extract the images, to make them suitable for uploading to Moo? The obvious way to achieve this using ImageMagick is to use the -extract command, but with pngs this has an odd side effect: it gives you an output image of the size you asked for, but sets the canvas to the width and height of the original image, with the offset as specified in the extract command. This is not very helpful.

I hope that this all makes sense and helps to avoid this problem in the future. I have also upgraded your order to rush printing, and they will be shipped tomorrow.

If you have any questions, please do not hesitate to get in touch by replying to this email.

They are absolutely spot-on – the way I made the cards was to start off with a 2080x2340 PNG file, use ImageMagick to crop it into 4x3 cards, and then upload the cropped tiles to MOO for printing. So, if you fancy getting your own deck printed, here’s what you’ll need to do:

Download this template.png – it’s a huge PNG with transparent windows for you to arrange your images of choice behind the frame on each card. (No, you can’t have my LOLCAT pictures. Mainly because I don’t own the copyright, but also ‘cos choosing your own pictures is half the fun)
Put photographs in it, tweak it to suit, flatten it, and save it as agile_cards.png – this should be a single 2080x2340 PNG file
Download ImageMagick if you don’t have it already.
Use ImageMagick’s convert.exe to slice your file into cards:

D:\lolcards>C:\Program Files\ImageMagick\convert.exe –crop 520x780 agile_cards.png card%02d.png
Use ImageMagick to strip the offset from the individual card files – this is the bit that’ll stop you having the same problem I had above:

D:\lolcards>C:\Program Files\ImageMagick\mogrify.exe +repage card*.png
Design an image for your card back
Head over to MOO.com, and use your agile card pictures to order a pack of rounded-corner business cards. I only used ten of the card images, so a pack of 50 cards includes five full sets of cards – with their classic paper and rounded corners, my deck was £13.49 + postage and VAT.
Wait for the postman

Who knows… maybe personalized planning decks will become the software developer equivalent of personalised guitar picks or golf balls… ? Or maybe they’re just fun. Mine certainly brought a few smiles to people’s faces this morning.

Cross-Domain Forms Authentication with ASP.NET

2012-03-30T12:55:00+00:00

I'm looking into splitting bits of a web application across multiple domains, so that - for example - all logins, password changes, etc. are handled at http://security.example.com/, payments and order history are handled at http://accounts.example.com/ and so on. Just to keep things interesting, some of these apps are in .NET 2.0 and some are greenfield apps we'll be building in .NET 4.0.

Good news is - it's completely possible. And with IIS 7, it's apparently even possible to secure static pages, classic ASP and other non-.NET pages using the same authentication mechanism.

Step 1: Make sure your machineKey values are identical

Each application that's going to participate in cross-domain authentication needs an identical machineKey element - this defines the encryption keys that are used to encrypt and decrypt the .ASPXAUTH cookie that contains the authenticated user data.

In my solution, I generated a machineKey using http://aspnetresources.com/tools/machineKey and then added this to the web.config files for each application individually - it goes under <system.web> like:

<system.web>
<machineKey validationKey="yada yada yada" decryptionKey="yada yada yada" validation="SHA1" decryption="AES" />
</system.web>

In theory, you could also define this in your machine.config or machine-level web.config file - remembering that you may have up to four of these (x86 .NET 20, x86 .NET 4.0, x64 .NET 2.0 and x64 .NET 4.0) - but I haven't tested this approach.

Step 2: Configure your FormsAuthentication attributes

Add this to the <system.web> section of the applications that are going to participate in cross-domain authentication:

<authentication mode="Forms">
<forms loginUrl="https://security.example.com/login" timeout="2880" domain=".example.com" path="/" />
</authentication>

Key points to note here:

The loginUrl is an absolute reference to the site that hosts your security app - so if you try to view a secure page, it'll redirect you to this website to log in.
The cookie domain is set to ".example.com" = notice the leading dot. That's important - it means the same cookie will be sent to example.com, something.example.com, another.url.at.example.com, and so on.

You'll need to make sure you're calling FormsAuthentication.SetAuthCookie(username) on your login server, but once that's done, you should be able to retrieve HttpContext.Current.User.Identity.Name from anywhere on any participating server. It is still up to you to verify that the username is valid and authorized - but this should save you a bit of plumbing when it comes to getting things working.

Snowcode 2012

2012-02-16T16:25:00+00:00

Snowcode is an awesome combination of skiing holiday, unconference and hack camp. Snowcode was started in 2010 as an experiment, organized by Alan Hemmings whom you may know from the Cambridge .NET User Group. We did it again in 2011, and in the spirit of continuous improvement, we're making a couple of changes to the format this year. We have a new venue - Chalet Joseph in Morzine - and we're going to try a more focused approach to the "code" part of Snowcode. Previously, we've used an "anything goes" unconference format, which has led to workshops on OpenRasta and REST frameworks, WaTIR, ASP.NET MVC, MongoDB, automated acceptance testing, and lots of good code and great ideas.

This year, it's JavaScript all the way - from deep-diving into the obscure corners of pure JavaScript to the latest JS frameworks and tools such as node.js, CoffeeScript, Backbone, Knockout - and, of course, jQuery. We'll discuss them, we'll play with them, and we'll work out fun ways to glue them together to build apps, websites, services and games.

One of the things we've struggled with on previous Snowcode events is sustaining the unconference format over an entire week, and so I'm really excited to announce that Alan has invited Pete Mounce and I to coordinate the "code" part of Snowcode 2012. Pete and I are preparing a programme that will provide a little more structure to the week, but still be modular enough that people can dip in and out, participating in the bits that interest them. If you're exhausted from a day on the snow and want to skip a session, that's fine - and if we find ourselves going off on interesting tangents - no problem. We're here to have fun and to learn. (And to ski, of course.)

Every Snowcode is different, but a typical day will go a little something like this:

Wake up (a mild hangover is optional but not unlikely)
Breakfast - coffee, fresh bread, croissants, fresh juice... and a dozen or so people sat around laptops doing a quick bit of coding or catching up on Facebook or giggling at the pictures from dinner last night.
Hit the slopes - if that's your thing. Morzine is in the Portes du Soleil ski area, with access to over 600km of stunning runs and spectacular scenery. The sun is shining, there's fresh powder on the ground, and your hangover's gone by the time you hit the bottom of the first run.
Break for lunch - normally an opportunity to rendezvous with other Snowcoders and swap notes on the mornings' ski runs
Ski some more - or snowboard, or head back to the chalet for a lazy afternoon's coding, reading or napping...
Coding & "geeking out" - get back to the chalet, have a shower, open a beer and a laptop, and spend a couple of hours hacking, learning, coding, talking and generally working out cool ways to build cool stuff.
Dinner - three courses, wine, stories, the inevitable jokes at Alan's expense, that kind of thing...
...and then whatever you like! Wander into town and go clubbing; watch movies, play X-box, more hacking, or just crash out early and get ready to do it all over again. A jaunt into Morzine is well worth it - it's a fantastic town with some really good bars, and the Mutzig beer is practically a Snowcode rite of passage.

You should come. No, really, you should. It'll be a fantastic week, and we'd love to see you there. If you ski or snowboard, you'll love it. If you code, you'll love it. And if you're not a skier or a coder, you're still more than welcome - it's a great opportunity to learn to do either (or both!)

Full details of the event are over on the SNOWCODE 2012 page on meetup.com - but in a nutshell:

It's in Morzine, France, from 18-25 March 2012.
Book your place via the SNOWCODE 2012 page on meetup.com - Snowcode costs £569 per person including half-board accommodation and airport transfers.
Get a cheap flight to Geneva with Easyjet (don't forget to book your skis as extra baggage)
Don't forget your laptop...

" when I ski, I live... the rest is just waiting "

Snowcode 2010

VIEW SLIDE SHOW

DOWNLOAD ALL

GiveCamp UK 2011 – A Retrospective

2011-10-25T23:52:00+00:00

I spent this last weekend at UCL’s Bloomsbury campus in London, with a hundred or so charitable geeks, at the first UK GiveCamp. I came away from it amazed – at the generosity of the volunteers and sponsors alike; at the sophistication of some of the solutions that were delivered within a single weekend, and at the reactions from the charities involved when we presented our projects on Sunday afternoon. It was a wonderful experience, and one I’d happily do again, but in the spirit of agile and continuous improvement, here’s my own personal retrospective on GiveCamp UK 2011.

What went well?

The project. I was lucky enough to be working with a team who were helping Scene and Heard, a mentoring project based in north London that arranges for children to work with volunteer theatre professionals to write and perform plays. Simma , their Head of Development, showed up on Friday with a great pitch, obvious enthusiasm, a wealth of knowledge about the domain, and a “wish list” of projects and ideas for us to investigate. Top of the list was a ticket booking system – and I was immensely heartened when our entire team of coders – you know, people who like to build stuff – unanimously agreed that the pragmatic solution would be to hook them up with EventBrite. By 8pm on Friday, we’d drawn up a backlog of requirements, made sure EventBrite would actually deliver what they needed, and a couple of the team were already working on the integration. This left the rest of us free to start looking at some of the more blue-sky ideas on Simma’s list… and somehow by Saturday afternoon we were building a full relational database, MVC web front-end, and a set of data migration and normalisation routines to import their data from a collection of Access and Excel sheets into SQL Server. It’s not quite live yet – we’re having some problems getting the Entity Framework config code to work with AppHarbor’s SQL Server databases – but by the end of the week we hope to have it online, hosted, and handed over to them, for immediate use, or as a solid platform for future development.
The venue. The power worked. There was ample desk space and seating. The wi-fi was pretty solid – for most of the weekend we were actually using their wi-fi network for all our database access and development, as well as simple web browsing & e-mail, and other than an occasional IP address change, it worked.
The catering. The food was excellent, there were ample supplies of drinks, snacks, awesome tea thanks to Teapigs, fresh fruit and enough Haribo to keep you wired and coding well into the small hours. Result.
The wrap-up. Without Paul & Phil imposing a strict code cut-off, we’d have happily coded until they threw us out – but in retrospect, having a hard deadline with plenty of notice really focused our efforts towards the end of the project.
The platforms. We used Github for hosting, Trello for organising our backlogs and a whole lot more besides; EventBrite solved the ticketing problem, and we’re in the process of deploying to AppHarbor. All fantastic, powerful tools – and all completely free.
The swag. 80Gb SSDs for all the attendees? Best. Swag. EVER. Plus a hugely generous range of licenses, software, books and ebooks. Check out the list of sponsors at http://www.givecamp.org.uk/sponsors – I’m hugely grateful to every single one of them for making this amazing event a reality.
The people. Everyone was helpful, cooperative, collaborative and enthusiastic, and I hope everyone learned as much working with each other as I did working with them.
The demos. I was really, really impressed at what the teams delivered. From discovery to implementation to – in some cases – deployment onto a live website, in under 48 hours… as Ben Hall put it, it “makes you wonder what we normally do all day…” I am also not at all biased by how genuinely delighted Simma and Jasmine looked when we showed them our work. Honest.

What could have gone better?

I thought the Friday start was too early. Lots of people – including several of the charities and one of the organisers – were held up in transit and missed the opening pitches and kick-off. The only downside of a venue as wonderfully accessible as UCL Bloomsbury is that at 5pm on a Friday it’s smack in the middle of the worst rush hour in the country. Just an idea – but I’d suggest next time a preliminary session, maybe earlier in the week, with the team leaders and the charity reps? A solid couple of hours capturing requirements and understanding the domain. We’d have been up to speed much quicker if one or two of us had had the chance to think things over ahead of time.
That’s actually the only criticism I have of the event itself. The rest are memos to myself and the team for next time:
- Bring an ergonomic keyboard! By Sunday afternoon my arms were starting to seize up… I’m not a laptop coder by choice, and having one of my trusty MS Natural 4000s would have made a *huge* difference. Plus a proper mouse. And a second monitor.
- Stick to what you know. This is not an event for trying out new technology – unless someone on your team knows it well, don’t go near it. We spent a good half-day investigating Visual Studio Lightswitch before concluding that we just didn’t have the expertise to tailor it to our requirements… it was gone 3pm on Saturday when we finally settled on ASP.NET MVC 3 with Entity Framework, and an hour later we were absolutely flying.
- Get a Skype chat or IRC channel set up ASAP. You’re going to be sharing lots of addresses, API keys, URLs – stuff that’s time-consuming to read out loud or write on paper.

In conclusion – wonderful event, I’m really pleased to have been part of it, and I hope the rest of our team will be at the next one because I’d love to work with them again some time. And huge thanks to Paul Stack, Rachel Hawley, Phil Winstanley, Kendal Miller and Dave Sussman, who all looked absolutely worn out by Sunday evening – it wouldn’t have happened without you!

Moleskine + Kindle = ... Moleskindle?

2011-09-17T15:10:00+00:00

Faced with the harrowing prospect of trying to fit the entire Song of Ice and Fire into hand luggage on my next holiday, I bought a Kindle. It's quite magic. It won't switch off, it doesn't light up, it looks utterly fake - like some sort of plastic prop tablet device where they've used a printed cardboard screen... and it's absolutely lovely. My paperback books tend to end up rather battered from being slung around in bags all the time, and I wanted a case to keep the Kindle safe from knocks and scratches. Rather than spend money on one of the ridiculously overpriced cases you can get for it, I wanted to try something a bit different. You remember reading spy books as a kid where people would hide stuff inside hollowed-out books?

I found an old Moleskine notebook that was just the right size for it, and started hacking away - a couple of happy hours playing with craft-knives and glue, and here it is: the Moleskindle

It's a bit fiddly - and messy - getting the cutouts just the right shape; I found wood glue worked just fine - and once it's dried, the compacted glued paper is quite easy to carve & trim using a sharp craft knife. I cut a notch in the right-hand side so I can reach the page-turn buttons whilst it's in the case, but you need to pop it out to reach the power button or recharge it. Still, I think it looks pretty cool, it'll stop the Kindle getting knocked and scratched, and you can fool people on the Tube into thinking you're reading something incredibly intellectual that's been hand-written in a Moleskine notebook when you're secretly reading rock star autobiographies.

Software Development - HORSE-style

2011-09-11T21:31:00+00:00

There's as many ways to lose at poker as there's ways to fail at delivering software, but one variation I have yet to experience is a game called HORSE. In HORSE, each hand follows a different set of rules - you'll play a hand of Hold'Em, a hand of Omaha, a hand of Razz, a hand of Stud, and a hand of Stud Hi-Lo; then you go back to the beginning and do it all over again, until my brother has all my chips.

Inspired by this, I've devised the following brilliant software methodology for all those teams who can't quite settle on a system that works for them. It's called WALKS, and you work in two-week sprints, using a different methodology for each sprint to ensure you get the maximum efficiency from all these wonderful processes and systems:

Weeks 1-2: Waterfall
You spend the first two weeks making bold, ambitious, big-design-up-front plans, and not actually writing any code or shipping any features.

Weeks 3-4: Agile
You spend the next two weeks trying desperately to get *something* built and releasable.

Weeks 5-6: Lean
Realizing that your "big design" is probably killing your attempts to be agile, you start hacking out unnecessary features and trying to pare the design back to something you might actually be able to build.

Weeks 7-8: Kanban
You still don't know what you're doing, so you decide to write everything on Post-It notes and stick them to a board, figuring that if you start pulling jobs off the queue, you might at least get *something* done.

Weeks 9-10: Scrum
You have two weeks of daily stand-up meetings, in a desperate attempt to try and get a handle on things. Finally, you sit down on Friday afternoon, have a two-hour timeboxed retrospective, and decide that what you really need is a full set of requirements and a definitive spec.

Then you take a weekend off, come in on Monday, and start at the top again.

If that sounds familiar, it's OK - you're not hopelessly lost, confused or unproductive; you're just taking a structured approach to being multi-disciplinary...

(This is a joke post. Please don't use WALKS to build software. Ever. The world has enough problems as it is...)

SkillsMatter Progressive.NET Tutorials 2011

2011-08-01T11:59:00+00:00

Some of you may have heard about - or attended - SkillsMatter's Progressive.NET Tutorials over the last couple of years. This is a three-day program of in-depth workshops covering the latest languages, frameworks and techniques in .NET development. The great thing about the workshop format is that it provides enough time to actually get some hands-on experience; instead of the rapid-fire 45-minute lectures you'll find at most conferences, you can actually try things out, ask questions, work through examples, and really get to grips with the techniques or frameworks you're exploring.

This year, Ian Cooper - whom you may know from the London .NET User Group - has put together a great programme of speakers and topics, and I'm really excited to say that, for the first time, I'll be there as a speaker instead of an attendee.

My workshop is called "Front-End Tips for Back-End Devs". I'll be looking at how many of the techniques we take for granted in back-end development - including DRY, abstractions, packaging and dependency management - can be applied to your page layouts, stylesheets and scripts. We'll cover semantic markup, we'll take a whirlwind tour of all the wonderful new tags introduced by HTML5, we'll look at CSS sprites and media queries, and we'll see how you can keep your web UI as clean, elegant and maintainable as the rest of your codebase.

There's also Damjan Vujnovic talking about TDD in JavaScript; I went along to one of Damjan's talks earlier this year and was really impressed, so I'm looking forward to the opportunity to go over his ideas in a little more depth. Jon Skeet will be talking about async/await programming on C# 5 (no word yet on whether Tony the Pony will be there). There's workshops on continuous deployment, on web development in F# using the WebSharper framework, on packaging and dependency management, on REST, on Nancy, on SimpleData - in fact, if you've heard the .NET community buzzing about anything in the last year or so, chances are there's a workshop here that will show you what they're all so excited about.

It's at the SkillsMatter eXchange in London, from 5-7th September 2011. The cost is £425 (yes, it's not free, but you do get three days of top-notch content without giving up your evenings or weekends) - and you can use the promo code PROGNET50 to get £50 off.

Sign up online here. To keep up with event news, follow the hashtag #prognet on Twitter, and it'd be great to see you there.

Just Do It: Command-Query Segregation, Nike-Style

2011-06-29T20:10:00+00:00

OK, CQRS is a hugely mis-applied and mis-understood architectural style. The insight I'm sharing here is based on my attending Udi Dahan's Advanced Distributed Systems Architecture course, then applying what I'd learned to a project we were building for 3-4 months, then having one of my colleagues go on the same course, and then have him come back and point out everything we'd done wrong. Let's assume you are already using CQRS. Maybe you're using it appropriately; maybe it's over-engineering; maybe it's completely misapplied. Doesn't matter, for what I have to say here; there are smarter folks than I who can tell you whether you should be doing it in the first place. No, I'm here to share a particular insight about CQRS with you.

Your commands should be like a psychotic drill sergeant screaming orders.

When you issue a command, your work is done. End of story. Maybe it happens immediately. Maybe there's a delay, and someone has to wait a few seconds. Maybe it doesn't go according to plan, and somebody else notices afterwards, and they call someone else, and it gets fixed up. Maybe it fails spectacularly. You don't care. (Hell, you're probably dead by now. Nobody would ever have won any wars if everyone threw an exception when they found the sarge face-down in a fox-hole with a bullet-hole in his head.)

You're the sarge. You are IN COMMAND. You order someone to destroy the ammo dump in North Camp, then you get on with your life. Tomorrow morning, you'll get a fresh intel report. Maybe it'll say that ammo dump in North Camp has been destroyed - maybe it won't. You'll review the fresh intelligence, decide what to next, issue a fresh batch of orders - and get on with your life. That's CQRS. Your data is stale, your word is LAW, and you have better things to do than hang around wondering if you maybe did the wrong thing. If you give a command, and it doesn't get obeyed, there's exactly two potential outcomes:

Nobody notices
Somebody notices

Nobody notices? Cool. No problem. Somebody notices? Well - that's where you hope it's one of your guys (i.e. alerts, logging, infrastructure, monitoring) instead of one of THEIR guys (i.e customers/clients) That's how you do CQRS. Get your intelligence - your queries. The freshest data you can get, but don't bust a gut if it's a little out of date. Give your orders. Trust your intelligence. Get on with your life. Rinse. Repeat.

How to install "Active Directory Users and Computers" on your Windows 7 Workstation

2011-06-11T14:12:00+00:00

This one stumped me until I hit upon the magical combination that makes it work.

Install Windows 7 Service Pack 1.
Download the Remote Server Administration Tools for Windows® 7 with SP1
Install them.
Reboot
Go into Start -> Control Panel -> Programs and Features, and go to "Turn Windows features on or off" - because the installer will download and install the admin tools, but won't actually switch them on. Helpful.

What makes it fun is that it just fails silently until you get it right. No error messages, no warnings... installer didn't even tell me I was missing SP1 - which I thought I already had.

Why Cloning Classic Games in Javascript Makes for a Great Hack Day

2011-06-08T23:12:00+00:00

Hack days should be about code. Anything that stops you writing code (or talking about / editing / refactoring code) is friction.

There's two things in any software project that tend to cause huge amounts of friction - certainly during the early stages. One is tooling. Installing compilers takes time. Installing libraries takes time. Configuration takes time. I remember a day at Snowcode last year when we spent literally five hours installing Ruby, various build tools, make files, modules, browser automation components, plug-ins... I don't think I wrote a single line of code that day. It was interesting, and educational, but a hack day should be about building stuff.

The other huge source of friction in software development? Debates. There's X ways of doing something, and you can't make any progress until you've chosen one. Should we allow HTML in the comments? How big do we make the gallery thumbnails? What colour should we paint the bike-shed? On a "real" project, these decisions are made by the product owner - but for something like a hack-day, if you one person in charge of all the design, decision-making and prioritisation, they'll rapidly become a rather frustrated bottleneck.

So - pick a problem that's clearly-defined and well-understood, and solve it using a language that everyone's already got, that doesn't need a compiler, linker or build environment, and that everyone can run just by opening a web browser.

In other words - clone a classic game in JavaScript.

Choose something everyone's played. Bomberman. Tetris. Lemmings. Asteroids. Pac-man. Have a copy of the game on hand - on a laptop, or an emulator, or bring a console, or whatever - so if anyone asks questions, you can just refer to your definitive reference implementation and get back to work. That'll eliminate debate without handing anyone the poisoned chalice of product ownership on a volunteer-based project.

And embrace the awesome lightweight expressiveness of JavaScript - the only language that you can write and run, out of the box, on every single computer since Windows 98. There's no compiler. There's no IDE, no build chain, no runtime or virtual machine or standard libraries to install. People can use vi, Visual Studio, TextMate, Notepad - whatever they like. (Personally, WebStorm is rocking my world right now - JavaScript intellisense and refactoring with built-in Git support... it's fantastic. Just remember that Ctrl-Y doesn't redo by default and you'll love it.)

OK, this weekend we were using NodeJS, so the build/run/test cycle involved restarting the node server (which is Ctrl-C, up, enter) - but the guys working on the renderer didn't even need a server. They built a client-side test harness (index.html), and their build and deployment cycle was Ctrl-S, Alt-Tab, F5.

That's low friction. That's walking in off the street, opening up your laptop, pulling the code, and starting to build stuff straight away. And I like that.

Notes from the KaboomJS! LonDev Hack-Day

2011-06-05T15:01:00+00:00

A couple of weeks back, I had this crazy idea to build Bomberman, in Javascript, in a day. I floated the idea on Twitter and got a pretty enthusiastic response, and so I set up the first LonDev hack day. 12 people, in a room, for one day, working together to build and ship a working game.

Did we do it? You'll have to read all about it on the new LonDev wiki, but personally, I'm really pleased at how it went, and really excited at the idea of organising the next one.

What's really encouraging is the mix of people and expertise who contributed. We had a couple of .NET coders, some Ruby/Rails guys, some JS web hackers who'd not done node/socket stuff before, and @palfrey who, as far as I can tell, spends his days switching between Erlang and PHP to stop himself getting bored.

A fun day. Some really solid code. Some really interesting lessons learned. And there's a couple of us hanging out in #kaboomjs on Freenode over the next few days to get it finished off and up and running.

Want to work for Spotlight?

2011-04-26T12:55:00+00:00

Spotlight are hiring! We're looking for someone to join our software team full-time, in a senior development position. An experienced scrum master, who knows how to work with business and technical people to make things happen. Somebody who understands how to create great software. From database optimization to SOLID principles to TDD to user experience and accessibility, you understand what makes software great - great to use, great to maintain, great to extend.

We’re at a turning point. Five years ago, we were an award-winning publishing company who maintained our own website. Five years from now, we’re going to be a software company who publish award-winning directories. It’s a great place to work, and it’s a really exciting time to be here. It’s full-time, permanent job, working at our office just off Leicester Square. We’re upstairs from the Prince Charles Cinema – London home of Sing-along-a-Sound-of-Music and The Room – and surrounded by excellent bars, restaurants and theatres.

If you understand 90% of the postings on my blog, you’re probably in the right ball-park in terms of technology – but if you want buzzwords, it’s C#, .NET, agile, scrum, MVC, Castle Windsor, NHibernate, NServiceBus, jQuery, IIS, SQL Server, NUnit, SOA, TeamCity, FinalBuilder, msdeploy, and various other bits that are occasionally referred to as the “alt net stack”.

Interested? Read the full job spec, and details of how to apply, at www.spotlight.com/jobs/developer.html

NO AGENCIES. Seriously. If we want to deal with agencies, we’ll call you. If you call me, I will put my phone handset in a drawer, close the drawer, and let you talk to my stationery while I wander off and make some coffee. If you’re lucky, it’ll only waste 90 seconds of your time. If you’re unlucky, your phone system still uses analogue-switched PSTN and you’ll find you can’t hang up. It’s hard to earn commission when you can’t use your phone, and you’d be surprised how long it takes to make a really good cup of coffee.

Slides and Notes from “So You Think You Know JavaScript”

2011-04-10T20:00:00+00:00

The slides, notes and references from my JavaScript talk are now online, at

http://www.dylanbeattie.net/javascript/

A huge thanks to everyone who came along, to Ian Cooper and the LDNUG User Group for organising, and to SkillsMatter for the venue, the projector, the publicity, the video and the ginger tea. A full video of the talk is also available on the SkillsMatter website – and you’ll be pleased to hear that their awesome new video processing rig means you can now see my grinning face AND read the code samples on the slides.

The NodeJS demo code is open source and is online at https://github.com/dylanbeattie/BomberJS – fork it, pull it, do whatever you like with it. No warranties as to whether it’s any good or not… but it’s there and it works.

A couple of people asked afterwards about running Node on Windows, as I was doing in the demos. I was using a compiled binary from http://node-js.prcn.co.cc/, which worked absolutely fine for little demo apps with 5-6 concurrent client connections. I've no idea how it scales, but the general consensus seems to be that you should stick to Linux / MacOS for hosting any significant Node applications.

Churn-down Charts

2011-04-10T16:05:00+00:00

Our team have just finished a sprint on a project that’s using loads of new technology – MSMQ, NServiceBus, WCF – that we’ve not worked with before, and it’s played havoc somewhat with our estimates of how long everything was going to take. We hit our deadline, but only thanks to the product owner shifting a group of features into the next sprint, and at the retrospective everyone agreed that the process worked just fine but we didn’t have any really good way of visualising it. We have a burn-down chart – actually, we have two, ‘cos there’s one drawn on the whiteboard and there’s one in FogBugz as well – and what we’ve been doing is at the end of every day, we’ll just mark the number of hours left. On days when we discover more problems than we ship features, this looks like we’re moving backwards… which is true in terms of monitoring progress and planning, but isn’t great for morale, and it doesn’t really explain what’s going on.

So, I’ve come up with this, as a way of tracking estimation accuracy and churn as well as straightforward progress. I’ve no idea if it’s original or not, I don’t know whether it has a name, but I’ve called mine a churn-down chart. I’ve annotated this example to show you what happens over the course of the project – click for a bigger version.

Basically – when the green line hits the red line, you’re done. The product owner controls the red line, by adding and removing features from the sprint. Yes, I know you’re not allowed to add stories to a sprint that’s in progress - I think the chart actually demonstrates why. The little blue tails were inspired by this fantastic visualisation of budget forecasts compared with reality (which I found via Chart Porn), and they track how many hours of features we actually delivered that day, as opposed to how many hours we have left at the end of the day. This clearly shows the difference between days when we were productive but discovered lots of unplanned work, as opposed to days when we were just stuck.

I like the way it empowers the product owner to actually work with the team to hit the deadline – you’re not working towards a fixed target, you’re both dealing with shifting requirements as you find bugs and have ideas, and you can see at a glance whether you’re on target or not, and if not, why not.

Development Methodologies

2011-04-04T08:48:00+00:00

By now, everyone’s seen test-driven development (TDD), behaviour-driven development (BDD) and domain-driven design (DDD) – but there’s some other, so far development paradigms that haven’t got nearly the attention that they deserve.

Attention Deficit Disorder Driven Design (ADDDD)

Most commonly seen in open-source projects. You begin by implementing a core feature. After a couple of days, when either it gets boring or you’ve coded yourself into a corner and can’t work out how to get out, you pick a new feature and start implementing that one instead. Advantages of this approach are that you can tick “in development” on the feature comparison charts when evaluating your solution against the alternatives. Disadvantages are that it leads to crappy software that doesn’t work.

Attention Deficit Hyperactive Disorder Driven Development (ADHDDD)

Just like ADDDD, but features are only ever added in brief caffeine-fuelled bursts of manic coding, usually around 4am, accompanied by dozens of tweets, blog posts and Facebook status updates.

Developer Developer Developer Driven Development (DDDDD)

Projects are started twice a year, normally the week immediately after the popular DDD community event at Microsoft headquarters, and generally involves building something really ground-breaking like a wiki or a blog engine, just to “get your head around” all the amazing new stuff you’ve seen at DDD. You’ll generally lose interest about two days after you put the code up on Github as a “pre-alpha technology demo”, and then six months later you’ll do the whole thing all over again.

Advanced Dungeons & Dragons Driven Development (AD&DDD)

Everyone sits around drinking Red Bull, eating Doritos, boasting about their accomplishments and pretending to be some sort of tenth-level software architect when deep down they’re still not quite sure what a pointer is. A “dungeon master” (also known as a “project manager”) occasionally rolls some dice or reads a Gartner report, and then tells them that their project has died. Then they do it all over again, once every couple of months, sometimes continuing well into middle age.

Acronym Driven Development (ADD)

The HORSE of development methodologies; you consistently blame the failure of your last project on the fact that you picked the wrong methodology, and resolve to try something different on your next project. The conventional approach is to go test-driven, then behaviour-driven, then domain-driven, then extreme, then back to domain-driven. It’s a very educational way of wasting your employer’s time and money, and there’s normally someone in a back room happily coding away who doesn’t have the faintest idea what the rest of you are doing, but is probably shipping enough features to keep your company afloat.

I’ll be talking about JavaScript at Skills Matter on April 5th

2011-03-22T17:41:00+00:00

On April 5th I’ll be giving a talk on JavaScript at SkillsMatter here in London. It’s being organized by the London .NET User Group, but it’s not a .NET talk. Instead, it’ll cover a range of topics related to JavaScript’s history, the current state of the language, and the future of this widely-used and widely-misunderstood language.

JavaScript is fifteen years old, and the principles that influenced JavaScript’s architecture go back to the very dawn of computer science. It’s a powerful, expressive, dynamic language, that’s now being used to deliver some of the biggest and most popular software application in the world – and yet a whole generation of developers still thinks of JavaScript as being a scripting language that’s barely good enough to make pop-up windows appear on a web page.

There’s a lot of very cool stuff going on in the JavaScript world right now. With HTML5’s offline storage, you can use JavaScript to write client applications that you can install to your phone or your laptop and run them even when you’re offline. With CommonJS, there’s finally a unified effort to create a standard runtime library for JavaScript so we can write JS programs that support file systems, networking, loadable modules and unit tests. With NodeJS, there’s a fast, scalable framework for writing HTTP servers as collections of discrete JavaScript components. With frameworks like KnockoutJS, there’s declarative support for building rich web user interfaces in JavaScript - and it’s still pretty good at doing pop-up windows as well.

So you think you know Javascript? Sign up, come along and find out. I’ll bet you a pint there’s something in there you’ve never seen before.

True Names, and Other Dangers: What Dr. Seuss Can Teach Us About SoA

2011-02-26T23:23:00+00:00

Did I ever tell you that Mrs. McCave, Had twenty-three sons, and she named them all Dave?

Well, she did. And that wasn't a smart thing to do; You see, when she wants one, and calls out "Yoo-Hoo!
Come into the house, Dave!" she doesn't get one; All twenty-three Daves of hers come on the run!

- from “Too Many Daves” by Dr. Seuss

They say there’s only two hard problems in software – cache invalidation, naming things, and off-by-one errors. Cache invalidation’s hard because it’s difficult to clarify requirements. Off-by-one errors are hard because the joke wouldn’t work without them. But naming things? How hard can that be?

If you’ve ever worked in IT support, you’ll have had calls saying “the system is down”. Sometimes, a more enlightened caller will helpfully tell you that it’s ‘the network’ or ‘the database’ that’s broken. I once started at a job where everybody referred to everything as “sequel.” There had been a big database migration a few years earlier, resulting in a new website and new desktop software, and the whole process had been referred to as “upgrading to SQL Server”. Everyone kept hearing the techies talk about “upgrading to sequel”, and so when they got something new on their desktops, they conclude “Ah – this must be that sequel thing that everyone’s been talking about!”. Two days later, they call you up and say there’s a problem with ‘sequel’ – and in this context, ‘sequel’ could refer to just about anything. The name was overloaded to the point of uselessness.

“Ah yes - but it’s
services
all the way down!”

What’s scary is that this happens all over the industry. People talk about “Software as a Service”, when what they’re actually dealing with is an XML web service, that’s connecting to a WCF service hosted in a Windows service to provide a business service. Like dear old Mrs. McCave, we’re finding out that names are great if they’re unique, but when different things start laying claim to the same names, you’re going to end up cross-eyed.

So, as my team start teasing apart our proverbial big ball of mud, I’m trying out a new naming policy for the new components and modules we’re building. Pick a word that sounds nice and doesn’t mean anything within our business.

The last three projects I worked on were called Rosemary, Tarragon and Kamogelo. No namespace conflicts, no semantic overloading and no clashing with reserved keywords. Rosemary’s almost like an employee – it has an event log, and a mailbox, and a sufficiently clear sense of identity that people seem to get it. When they say there’s a problem with Rosemary, they’re right; it doesn’t take 15 minutes to work out what they mean, and that’s a good thing. It also encourages clear separation of concerns, and facilitates good discussion thereof – lots of “does this feature belong in Rosemary or Tarragon?” instead of just adding another class to the legacy codebase.

So it’s goodbye, “customer e-mail service” and “accounts system” and “web shop”, and hello to Sundance, Monolith and Aquarius. And before too long, Moonface and PuttPutt and Shadrack, in recognition of dear old Mrs. McCave.

Making HttpContext.Current Available Within a WCF Service

2011-02-24T14:28:00+00:00

I needed to add a quick’n’dirty WCF service to an ASP.NET MVC web application, so I could call a handful of methods from a different application.

The MVC app in question is using Windsor, NHibernate and the repository pattern, so we’ve got a fairly standard pattern where we spin up a ManagedWebSessionContext in the Application_BeginRequest handler (in global.asax.cs) and then flush and close the session in Application_EndRequest(). I used the Windsor WCF facility to inject a bunch of dependencies into a little WCF service, but I was finding that SessionFactory.GetCurrentSession() was always returning null – because when you’re using the ManagedWebSessionContext, your NHibernate session is bound to your HttpContext.Current, and by default you don’t have one of these inside a WCF service.

However - if you can live with tight coupling between your WCF service and IIS hosting, there’s a couple of little config things you’ll need to do to get this working. What doesn’t help is that until you’ve got all this just right, you’ll get a really helpful “Failed to Execute URL” error from IIS that’ll tell you absolutely nothing about what’s wrong.

First, make sure WCF HTTP activation is installed on your server – in Windows 2008, it’s under Server Manager –> Features:

Next, make sure you’ve registered the WCF service model with IIS, by running:

C:\WINDOWS\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\> ServiceModelReg.exe –i

Next, make sure your web service is running in ASP.NET compatibility mode. First, check you’ve got this:

<system.serviceModel>
<serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
</system.serviceModel>

in your web.config file, and then decorate your service implementation with the AspNetCompatibilityRequirements attribute:

[AspNetCompatibilityRequirements(RequirementsMode= AspNetCompatibilityRequirementsMode.Required)]
public class WcfMagicService : IMagicService {
. . .
}

The last thing I had to do was necessitated by WCF not supporting multiple host headers; I had to hard-wire the WCF endpoint to listen on a specific hostname. In this case, this involved tweaking the serviceHostingEnvironment section of web.config, which now looks like this:

<serviceHostingEnvironment aspNetCompatibilityEnabled="true">
    <baseAddressPrefixFilters>
        <add prefix=http://services.mydomain.com” />
    </baseAddressPrefixFilters>
</serviceHostingEnvironment>

And then adding another attribute to the service implementation class:

[ServiceBehavior(AddressFilterMode=AddressFilterMode.Any)]
[AspNetCompatibilityRequirements(RequirementsMode= AspNetCompatibilityRequirementsMode.Required)]
public class WcfMagicService : IMagicService {
}

Once that’s done, you’ll have an instantiated HttpContext.Current inside your service methods, so your code – and useful things like NHibernate’s ManagedWebSessionContext – will behave just as they do in normal MVC controllers or WebForms code-behind.

Check Out smtp4dev if You Build Mail-Enabled Software on Windows

2011-02-21T23:46:00+00:00

One of my cow-orkers pointed me at a great little utility a while back called smtp4dev. It’s an SMTP server that listens on your local machine, and instead of relaying e-mail, it’ll capture them and store them in a queue so you can review and open them.

It’s brilliant – simple and elegant and incredibly easy to use. Just configure your application (website, debugger, logging framework – whatever it is you’re building) to send mail on localhost:25, fire up smtp4dev, and watch the messages pile up. I’ve been building SMTP appenders for log4net this evening, and it’s been really, really useful.

Binaries and source are at smtp4dev.codeplex.com – well worth a look if you ever write software that sends e-mail.

How to host Git in the same Apache server that comes with CollabNet Subversion

2011-02-07T23:28:00+00:00

CollabNet Subversion Edge is a great Subversion distro that includes Apache 2.2 and the ViewVC web-based repo browser, and makes it really, really easy to get up and running with Subversion and WebDAV. I’m setting up a project server to host something we’re working on, and it’s been generally decided that whilst Subversion is all very well for keeping Word documents in, we’d quite like something a touch more… distributed for the actual source code repo. And when James Gregory mentioned on Twitter that git would mean “no more tree conflicts ”, I may have actually started salivating… ahem.

Anyway, yes. Apparently Git is quite good.

Jeremy Skinner has some fantastic notes on how to get git up and running with Apache 2.2. on a Windows server – I followed these pretty much to the letter to get my first incarnation up and running, but had to comment out a bunch of the Collabnet/Subversion settings in the Apache config files to get the Git server running properly. A bit of tinkering, though, and I’m pretty much there. What makes this interesting is that CollabNet includes a web-based admin console, which makes configuring the built-in modules very straightforward, but it does mean several of the config files have this rather ominous warning at the top:

#
# DO NOT EDIT THIS FILE IT WILL BE REGENERATED AUTOMATICALLY BY COLLABNET SUBVERSION
#

so – any changes we make in there will be just peachy until someone touches the web interface, at which point BOOM! they’ll spontaneously stop working. So whatever we’re going to do, we need to do it without touching any of those files. I wasn’t sure at first whether this would be possible, but it seems to be up and running now and hanging together quite nicely.

Fire up the Apache httpd.conf file in your favourite editor – by default it’ll be in C:\Program Files\Subversion\data\conf\ – and add the following lines at the end:

# Configure Apache to listen for named virtual hosts on port 80
NameVirtualHost *:80

# Include the configuration file for our git http hosting
Include "C:\Program Files\Subversion\data\conf\git_httpd.conf"

Now create a new document called – yep - C:\Program Files\Subversion\data\conf\git_httpd.conf – and make it look something like this:

# HTTP settings for using Apache with MSysGit on Windows
# Based on Jeremy Skinner's notes at http://www.jeremyskinner.co.uk/2010/07/31/hosting-a-git-server-under-apache-on-windows/

<VirtualHost *:80>

    # Set this to the root folder containing your Git repositories.
    SetEnv GIT_PROJECT_ROOT D:/Git/

    # Set this to export all projects by default (by default,
    # git will only publish those repositories that contain a
    # file named “git-daemon-export-ok”
    SetEnv GIT_HTTP_EXPORT_ALL

    # Route specific URLS matching this regular expression to the git http server.
    ScriptAliasMatch "(?x)^/git/(.*/(HEAD | info/refs | \
        objects/(info/[^/]+ | [0-9a-f]{2}/[0-9a-f]{38} | pack/pack-[0-9a-f]{40}\.(pack|idx)) | \
        git-(upload|receive)-pack))$" \
        "C:/Program Files (x86)/git/libexec/git-core/git-http-backend.exe/$1"

    # The canonical DNS hostname that you want to use for your git server
    ServerName my_git_server

    # Any other DNS aliases that point to your git server
    ServerAlias my_git_server my_git_server.mydomain.com my_git_server.my_intranet.local

    # The root folder for non-GIT-hosted documents (e.g. phpgit or some other Web front end)
    DocumentRoot "D:\gitserver\htdocs\"
    <Location />
        # This section is duplicated from the Collabnet SVN LDAP authentication
        AuthType Basic
        AuthName "Spotlight GIT Repository"
        AuthBasicProvider csvn-file-users ldap-users
        Require valid-user
    </Location>
</VirtualHost>

Check your configuration by running httpd.exe from the command line, like so:

C:\Program Files\Subversion\bin>httpd.exe -f "c:\program files\Subversion\data\conf\httpd.conf" –t
Syntax OK

and if all looks good, go into services.msc and restart the CollabNetSubversionServer service (which is actually Apache)

Finally, I followed Jeremy’s instructions to get GitPhp running, but then replaced it with a different project – also called GitPhp – from http://www.xiphux.com/programming/gitphp/, which provides a full repository browser, revision history, etc.

All I had to do to get GitPHP running was to copy the gitphp.conf.php.example file to gitphp.conf.php, and then tweak the following settings:

/* The root folder of my Git repositories */
$gitphp_conf['projectroot'] = 'D:\git';

/* On 64-bit Windows, C:\Program Files (x86) ends up as C:\Progra~2\ so these need to be configured manually */
$gitphp_conf['gitbin'] = "C:\Progra~2\Git\bin\git.exe";
$gitphp_conf['diffbin'] = "C:\Progra~2\Git\bin\diff.exe";

Job done. I now have:

http://my_svn_server/ returning regular HTML pages hosted by Apache
http://my_svn_server/svn/ giving me a WebDAV browser view on my raw Subversion repositories
http://my_svn_server/viewvc/ giving me ViewCV web-browsable revision historys and diffs on my Subversion repos
http://my_git_server/ giving me GitPHP browsing of my Git repositories
http://my_git_server/git/ exposing URLs for GIT cloning, fetching and pushing
http://my_iis_server/ is still running IIS 7 with .NET 4 and MVC 3

and that’s all on one box, with two IP addresses, with the svn and git servers sharing an instance of Apache on one address, the IIS server running on the other, and DNS records pointing svn and git at the first address and IIS at the second.

Running IIS and Apache on the same Windows 2008 R2 Server

2011-02-07T19:18:00+00:00

I’m trying to get a Composite C1 site and the Apache WebDAV front-end to Subversion running on the same Windows 2008 R2 server, and doing so requires a bit of trickery with IP address bindings and such, and I thought I’d share it – partly ‘cos it’s useful, and partly because I’m bound to have to do this again in three months time and there’s no way I’ll remember how I did it. First off, make sure your box has (at least) two IP addresses – I’ve bound mine to 192.168.0.13 and 192.168.0.14
To get IIS to listen on ONLY 192.168.0.13, you’ll need to run the netsh.exe utility.

C:\Users\dylan.beattie>netsh
netsh>http add iplisten ipaddress=192.168.0.13
IP address successfully added
netsh>http show iplisten
IP addresses present in the IP listen list:
-------------------------------------------
192.168.0.13
netsh>exit

(note that netsh.exe is a Windows 2008 utility – if you’re running Windows 2003 or earlier, look up the docs on using httpcfg.exe to achieve the same thing)
If you now fire up a web browser and go to http://192.168.0.13/, you should get the default IIS7 “Welcome” screen, and http://192.168.0.14/ shouldn’t return anything at all. Now to get Apache listening on 192.168.0.14. Find your httpd.conf file – if you’ve just installed CollabNet Subversion (like I have) it’ll be in the \data\conf folder of wherever you put your SVN install.
You’ll need to find the Listen directive in httpd.conf, and modify it to say:

Listen 192.168.0.14:80

That’s all. Next time – to get Git running on the same Apache installation… until then, happy hacking.

UPDATE: After running this for several years, I've found that occasionally, following an unscheduled shutdown or power outage, IIS won't come back up properly after the box is restarted. Sites will respond on http://localhost/ but trying to access them via hostname gives an ERR_CONNECTION_RESET message.
This can be fixed by removing and re-adding the HTTP binding:

C:\>netsh
netsh> http
netsh http> delete iplisten 192.168.0.13netsh http> add iplisten 192.168.0.13netsh http> exit
C:\>iisreset

“Choose Life” For DBAs

2011-01-17T23:11:00+00:00

I’m really really sorry. Someone tagged #sqlmoviequotes on Twitter and I got carried away…

SELECT TOP(1) * FROM LIFE.

SELECT TOP(1) * FROM JOB.

SELECT TOP(1) * FROM CAREER.

SELECT TOP(1) * FROM FAMILY.

SELECT TOP(1) * FROM television ORDER BY SIZE DESC

SELECT * FROM washing_machine CROSS JOIN car CROSS JOIN compact_disc_player CROSS JOIN electrical_tin_opener

SELECT * FROM health, cholestorol, dental_insurance
    WHERE health.status = 'good' and cholestorol.level < 5

SELECT * FROM mortgage WHERE interest_rate = 'fixed'

SELECT TOP(1) * FROM home WHERE TYPE = 'starter'

SELECT * FROM person
    INNER JOIN friendship ON person.id = friendship.person_id and friendship.friend_id = 'ME'

SELECT * FROM leisurewear
    INNER JOIN luggage ON leisurewear.color_scheme = luggage.color_scheme

SELECT TOP(3) * FROM lounge_furniture WHERE payment_plan = 'hire purchase'
    AND range_id IN (SELECT range_id FROM fabric_option GROUP BY range_id HAVING COUNT(*) > @RANGE_SIZE)

SELECT 'diy', CURRENT_USER FROM activity WHERE DATEPART(dw, activity_date) = 1 AND DATEPART(hh, activity_date) < 12

DECLARE @your_mouth INT
DECLARE junk_food CURSOR FOR SELECT * FROM FOOD WHERE TYPE = 'junk'
WHILE @@CURRENT_SHOW IN (SELECT * FROM SHOW WHERE keyword IN ('mind_numbing', 'spirit-crushing')) BEGIN
    FETCH NEXT FROM junk_food INTO @your_mouth
END

BEGIN

    sp_start_job 'brat'
    sp_start_job 'brat'
    sp_start_job 'brat'

    KILL @@SPID
END

SELECT * FROM events WHERE EventDate > GETDATE()

SELECT TOP(1) * FROM LIFE

Mapping a Drive Letter to a Subversion Repository with CollabNet, WebDrive and WebDAV

2011-01-10T19:26:00+00:00

This is quite neat. As part of a business-wide agile initiative, I’m looking into solutions for storing and collaborating on documents – something that gives *me* the history and auditing capabilities of something like Subversion, but gives the rest of the team something clean and easy that fits well with current working practises.

So… mapped drive letters. Everyone knows about drive letters – “just stick it on the R: drive” is nice and easy, and as long as everyone’s R: drive points to the same place and the fileserver behind it’s getting backed up, you’re sorted. Unless you want to revert a document that’s been corrupted, or accidentally deleted, or you just want to get back an earlier revision because you realize you’ve done something dumb. Then you need to mess around with tapes and stuff, and that’s just no fun at all.

Plus, of course, we have a wiki, which isn’t much fun to edit because the constant round-tripping from WYSIWYG->markup->HTML->WYSIWYG tends to clobber newlines and formatting, but it *is* a great place to keep stuff because it doesn’t get lost.

So, requirements for document storage:

As easy to use as a drive letter.
Security. Windows / LDAP authentication to control who can read and who can write.
Revision history – just a record of who made changes, and when.
Ability to revert changes to an earlier revision
HTTP accessible so you can read stuff with just a web browser.

There’s two things you can do with a list of requirements like that… speak to vendors, or hack something together yourself. You wanna speak to vendors, you go ahead; I shan’t stop you. Still here? Good. Let’s hack.

1. Install Subversion on the server.

For this, I’m using the CollabNet Subversion Edge stack – a single installer combining Subversion, Apache and the ViewVC web front-end. It’s very, very neat, and (having done this the hard way) much easier than setting up mod_dav_svn yourself. Once it’s up, use the web interface (linked from the Start menu on your server) to set up a new repo – call it doc_repo – and then verify that if you browse to http://myserver/viewvc/doc_repo/ you get the ViewVC web front-end view of your new, empty repository.

If you’re after Windows/LDAP authentication, that’s also configurable from the Subversion Edge web interface – and CollabNet has detailed notes on how this works.

2. Install WebDrive on your workstation.

This is a commercial package that’ll map a Windows drive letter to a WebDAV share. This is supposedly something that Windows is capable of doing natively, but I have never, ever got this to work, not even once. I would be glad to hear recommendations for free / open-source alternatives for this, since it’s currently the only bit of this set-up that costs money. There’s apparently also a netdrive.exe floating around but licensing for NetDrive seems to be a little confused.

3. Hack the Subversion config file.

On the server, open up C:\Program Files\Subversion\data\conf\svn_viewvc_httpd.conf. We’re not going to edit this file – you’ll need to find the bit that looks like:

<Location /svn/>
   DAV svn
   SVNParentPath "C:\Program Files\Subversion\data\repositories"
   SVNReposName "CollabNet Subversion Repository"
AuthzSVNAccessFile "C:\Program Files\Subversion\data/conf/svn_access_file"
SVNListParentPath On
Allow from all
AuthType Basic
AuthName "CollabNet Subversion Repository"
AuthBasicProvider csvn-file-users ldap-users
Require valid-user
</Location>

and copy it. Then open up C:\Program Files\Subversion\data\conf\httpd.conf – which is the regular Apache configuration file – and paste the copied section right at the end, and make the following changes:

# Change Location to be the URL path of your WebDAV repo – I’ve used webdrive here
<Location /webdrive/>
    DAV svn
    SVNParentPath "C:\Program Files\Subversion\data\repositories"
    SVNReposName "Subversion WebDAV"
    AuthzSVNAccessFile "C:\Program Files\Subversion\data/conf/svn_access_file"
    SVNListParentPath On
    Allow from all
    AuthType Basic
    AuthName "Document Repository"
    AuthBasicProvider csvn-file-users ldap-users
    Require valid-user
# Add the two lines below
    ModMimeUsePathInfo on
    SVNAutoversioning on
</Location>

Now restart the web server (using ApacheMonitor.exe from C:\Program Files\Subversion\bin\ on your server) and check that you can see http://myserver/webdrive/ in a normal Web browser – the screen should say “Collection of Repositories” with your doc_repo repository listed underneath.

4. Connect WebDrive to your new WebDEV-Enabled Repository

Nearly there. Finally, fire up WebDrive on your workstation and create a new connection. Enter the Site Address/URL as http://myserver/webdrive/doc_repo/ – note that you must put the repo name in the URL otherwise WebDrive will complain with an error something like:

Unable to connect to server, error information below

Error: Socket receive failure (4507)
Operation: Connecting to server
Winsock Error: WSAECONNRESET (10054)

Correct settings will look something like this:

Hit Connect, and Windows explorer will fire up a new N: drive window pointing at your repo.

5. Witness the Awesome Power of Autoversioning!

Create an empty folder, create a text file in it, then browse to http://myserver/viewvc/doc_repo/ and you should see your new folder, and file, along with the Subversion history recording who created the file and when:

I like that a lot.

Christmas, Crisis and the Cloud

2010-12-29T00:20:00+00:00

This Christmas, I volunteered at Crisis, the UK charity that provides food, shelter and help for homeless people and rough sleepers over Christmas. It’s the first time I’d done anything like this, and – amongst stints of washing-up, manning doors, looking after showers, mopping and playing Articulate – I spent quite a lot of time helping with the suite of PCs that were provided for our guests to use.

It was a profound and enlightening experience. I use Gmail and Google Docs because they’re convenient – I regularly use 3-4 different PCs and online services are just a convenient way of keeping things accessible and in sync. It never really occurred to me that for someone who doesn’t have a computer of their own, or any regular access to one via a university or workplace, something as simple as a webmail account can be the difference between staying in touch and disappearing completely. People with no regular income, no phone and no fixed address were using Yahoo Mail to keep in touch with family and friends, apply for jobs and look for accommodation. It was quite amazing. I don’t know whether the folks at Yahoo, Gmail, Hotmail et al realize just what a profound difference this makes – but if any of them are reading, well done. You rock, and I never realized quite how much until this week.

Another interesting observation – Microsoft’s market saturation is so complete that even people who store their worldly possessions in a plastic shopping-bag insist on Microsoft Word. Their CVs are in Word format, they know the Word menus and commands – many of them are remarkably accomplished Word users. This is sad, because Word means Windows, and – even with the awesome job the people at the Aimar Foundation had done setting up a locked-down Citrix system - some machines became infected with a malware product called “Security Shield”. Security Shield masquerades as an anti-virus package, makes a lot of bad noise about (non-existent) viruses and then asks the user for a credit card number to “activate the virus protection”. Well, Security Shield, this Christmas you actually asked the homeless for money. Well done. You scared the hell out of frightened, vulnerable people who thought they’d broken the computer. You scared and upset the volunteers who were giving up their Christmas holiday to help those same people. I really hope you take all the money you’ve made from your little scam and spend it on something that’ll bring you joy in this life, because karma’s a bitch and your next life is going to suck.

Anyway. Karmic retribution aside, it really got me thinking. For every iPad-toting hipster raving about how they keep all their stuff “in the cloud” because it’s the Next Big Thing, there’s a rough sleeper out there who genuinely doesn’t have anywhere else to keep it. There’s web mail, web docs, online tools for retouching photos, online games (spidersolitaire.com proved very popular with some of our guests), online video – you have no idea how cool YouTube is to someone who doesn’t own a PC or a television - and I started wondering what else technology could offer.

What about a complete online tool set and sandbox for people looking to learn about software development? Tutorials, exercises, a lab, workshop, CV and portfolio, accessible anytime, anywhere, for the cost of a virtual machine and a few gigs of disk space (i.e. practically nothing). A full development environment where they can learn to code, store projects, compile, test and deploy applications and websites – but accessible entirely within the sort of stripped-down web browser you'd find in most internet cafés. There’s no material reason why someone couldn’t learn to build software, establish an online presence, contribute to open-source projects, develop a reputation on Q&A sites like StackOverflow, and generally become a demonstrably employable developer, entirely without any investment in physical resources. The days of needing expensive computer time or equipment to become a coder are gone. The financial barrier to entry in our industry is effectively zero. We should be shouting this from the rooftops, doing everything possible to put information and resources in the hands of anyone who cares to take advantage of it, and celebrating this amazing consequence of the openness that’s sustained our industry for so long.

Anyway. Crisis was awesome, you should all do it next year, and here’s to a 2011 filled with compassion, kindness, enthusiasm and excitement.

Happy New Year.

Git Logos and Icons

2010-11-28T19:25:00+00:00

I’ve recently started working on a couple of projects that are hosted on Github, most notably the Kurejito payment gateway project for which I’m a member of the core team on an OS project for the first time (yeah, scary…)

Git is pretty cool. A bit of a learning curve but I suspect once I get the hang of frequent commits and local branching and work out how stop shooting myself in the foot it’s going to be quite hard going back to Subversion.

Thing is, I really don’t like the Git icon and logo that are shipping with Mingw32 Git for Windows. I mean, I really don’t like them. I don’t like them so much that every time I fire up git bash, I get completely distracted by how much I don’t like them and end up wanting to redesign them instead of doing whatever it was I was supposed to be doing… so I did.

This started out as me just wanting a decent Windows icon for the link to Git Bash in my Start menu, I got a bit carried away, and ended up with this. And since Git appears to use a collaboratively-edited Wiki page as the closest thing it has to brand and logo guidelines, if you want to use any of these logos or icons for your Git-related shortcuts, projects or pages, go ahead. I’m not going to stop you, and I suspect they won’t, either.

If you’re interested, they’re released under a Creative Commons Sharealike Attribution License – and there’s downloads (including Windows and Mac icon formats) and original artwork files at http://www.dylanbeattie.net/git_logo/

Direct download links:

Git Icon – Windows .ICO Format – git_icon.ico
Git icon in MacOS .icns format – git_icon.icns
Git icon original artwork – git_icon.ai or git_icon.png
Git logo in PNG format – git_logo.png
Git logo in Adobe Illustrator format – git_logo.ai
Git logo in SVG format – git_logo.svg

or grab the whole package as git_icon_and_logo.zip

Sending Templated E-mail using the Spark View Engine

2010-11-26T11:46:00+00:00

We have a couple of systems that send personalized e-mail notifications to our users, and for a while I’ve been looking for a nice way to use a proper MVC-style view engine to populate the templates for personalizing these e-mails. The problem is, most of the ASP.NET view engines are so tightly bound to System.Web and things like HttpContext.Current and the VirtualPathProvider that running them in a standalone console application is really quite unpleasant.

Well, with a bit of hacking around and some help from @RobertTheGrey, I’ve finally got the awesome Spark view engine running within a console app. No fake VirtualPathProviders, no mocking or spoofing HttpContext.Current – it just works.

The secret is this little snippet of code here:

var templateFolder = @"D:\Templates\";
var viewFolderParameters = new Dictionary<string, string> {
{"basePath", templateFolder}
};
var settings = new SparkSettings();
settings.SetPageBaseType(typeof(TemplateBase));
settings.AddViewFolder(ViewFolderType.FileSystem, viewFolderParameters);
engine = new SparkViewEngine(settings);

which spins up a fresh SparkSettings configuration object, tells it to use your own TemplateBase class and the templates folder you’ve specified. The method that actually does the population looks like this:

public string Populate(string templateFileName, object data) {
var writer = new StringWriter();
var descriptor = new SparkViewDescriptor();
descriptor.AddTemplate(templateFileName);
var view = (TemplateBase)engine.CreateInstance(descriptor);
try {
    view.ViewData = new ViewDataDictionary(data);
    view.RenderView(writer);
} finally {
    engine.ReleaseInstance(view);
}
return (writer.ToString());
}

so you end up with little snippets like this:

foreach(var user in userRepository.RetrieveUsersWhoShouldGetWelcomeEmails()) {
    var htmlBody = templater.Populate("welcome_html.spark”, user):
    var textBody = templater.Populate(“welcome_text.spark”, user):
    mailServer.SendMail(“me@mysite.com”, user.Email, “Welcome!”, textBody, htmlBody);
}

There’s a full working example in the Spark repository on GitHub if you’re interested.

Should “Cancel” cancel the cancellation, or just cancel the cancellation cancellation?

2010-11-03T19:02:00+00:00

This is from SagePay’s new MySagePay admin system, which has brought us all so much joy these past weeks and will go down in history as my second favourite piece of web design ever, after Kibo’s Optimised for WebTV! page.

Today’s happy little discovery. You click to view a transaction. You think “Oops, wrong transaction.” Years of Windows experience means you instinctively hit Cancel – the button marked Cancel, that’s at the bottom-right of the window, just like it is on every other dialog box you’ve seen in the last 25 years. But guess what? In this case, the Cancel button actually cancels the transaction, rather than closing the window and backing out of the operation.

Of course, it wouldn’t do anything that drastic without a confirmation dialog… guess what that dialog looks like?

“Cancel” means exactly the same thing in practically every application, window manager and operating system that’s been released for the last 25 years. The idea that anyone might think that using it for a different purpose in a different context is… just unbelievable.

Moral of the story: if your ubiquitous language contains a word or phrase that everyone knows means something else – change the ubiquitous language. Think of a better word. Here’s some to get you started… Unauthenticate. Withdraw. Retract. Deauthenticate. Discard. Abandon. Reject. Decline. Even “Cancel Transaction” would have been better than this.

So – to cancel a transaction, you hit Cancel, then Cancel again. To cancel the cancellation, you hit Cancel, but DON’T press Cancel, press X and then X.

(and yeah, to cancel the dialog as opposed to the transaction, you click the groovy little X in the top corner, because this is Web 2.0, dontcha know?)

Obvious, really.

“If It Ain’t Broke…” – Why Good Code Doesn’t Guarantee Happy Users

2010-10-26T01:23:00+00:00

For years, my sites have processed online payments using a firm called Protx. Protx was acquired by Sage a few years back, rebranded as SagePay, and since then they’ve been gradually making changes to their system. Nothing too drastic, and always with plenty of warning that changes were forthcoming. Their payment API is generally rock-solid, powerful, flexible, and has allowed us to cope with all sorts of complicated payment scenarios without too much fuss. So far, so good.

About two months ago, SagePay announced that a preview of their new administration system was running on their test servers, and they invited everyone to go and have a look. Their old system was kinda clunky, but it worked, and our accounts team had been using it quite happily for years. So I went. I looked. The new system had lots of shiny Ajax – OK, fair enough, this is 2010 after all – and horizontal scrolling. No, really. Check this out:

I have a 1600x1200 screen because information is nice and I like being able to see lots of it at once. SagePay have decided, for some reason, that our payment transactions are best presented in a fixed-size 800x300px window. Fine, though – that’s what technical previews are for; lots of people reported this and were assured it would be sorted out.

Well, the new system launched over the weekend. The old one’s gone, they new one’s live, and things like the horizontal scrolling have not been sorted out, and customers’ reactions to the new system is, well, a bit negative. A thread on their support forums is full of gems like:

“A complete crock. Should be rolled back immediately.” (here)

“this is a complete and utter disaster. This system needs rolling back as soon as possible.” (here)

“The system is rejecting the order because of the delivery address line 2 which contained 58-62 xxxxx Road. Your system is not allowing the - in the address (This address is the formatted address that is provided by Royal Mail). This needs to be fixed quickly.” (here)

OK, fine. Big company rolls out shiny upgrade, upsets customers, breaks APIs, whatever. Big companies do this a lot.

But - I was expecting something different here, because last week, Mat – SagePay’s “Chief Nerd” – published this blog post, in which he talks quite rationally and eloquently about the forthcoming improvements. And it sounds like quite good stuff, too. Stuff like:

“To accommodate this we’ve changed the way we develop software, not only at a language level, but also at the process level. We’ve embraced Test Driven Development, Agile methodologies, Continuous Integration and parallel automated testing.”

“By writing tests first and only then writing code that passes those tests, we know that our software does what it is supposed to.”

“This demonstrable code quality gives our developers much more confidence in their code, frees them to refactor software that behaves sub-optimally, and ensures the test team’s time isn’t wasted on trivial bugs.”

(quotes from http://sagepay.wordpress.com/2010/10/18/if-it-aint-broke/)

OK. Mat, I believe you. If your comments about improved capacity and security are correct, this actually represents a huge achievement for the SagePay technical team. But I have to wonder… if the chief nerd is doing everything right, why are the customers so upset?

“…if the CHIEF NERD is doing EVERYTHING RIGHT, why are the CUSTOMERS so UPSET?”

Well, this is what I think probably happened. First, they’ve been working on this since June 2009; the first real customer preview was in September 2010, and it’s now live, just over a month later. That’s not a series of incremental improvements, delivering discrete chunks of business value every few weeks or months. That’s the big rewrite wearing a not-very-convincing Agile disguise.

Second, one of the core tenets of the Agile Manifesto is “customer collaboration over contract negotiation” – and to get that right, you have to know who your customers really are, and collaborate with them. This is tricky, because to the developers, the ‘customer’ is probably the product owner – but in reality, the customer should be the person who’s going to use the product. Now a firm like SagePay probably can’t call their customers in off the street to collaborate on a big project like this – but they can talk to them.

I think it’s really easy – particularly in big companies – to get this wrong. I have made this mistake many times. You launch version 1.0, you spend a couple of years babysitting it and fielding the support calls, and you end up thinking you know exactly what all your customers want… except you don’t, because happy users never call support. They just use the product and get on with their lives. You can find out what your users don’t like from support calls and complaints, but to find out what they do like, you need to get out there and talk to them, and it sounds like many of the features in the new SagePay system were based on feature requests and complaints from a vocal minority who weren’t really representative of the user base as a whole. It feels like you’re doing exactly the right thing, but you’re not actually collaborating with your customer.

“REAL USERS don't care about TEST DATA”

Second – technical previews are all very well, but real users don’t care about test data. When I tried the new MySagePay tech preview last month, it had about a dozen transactions in it, from 2006, when we were testing an upgrade to our own payment system. That’s not a realistic test of the system, because there’s no incentive to actually do anything with it. The only way to get real feedback is to get real people to use the software to do real work – and you can’t do real work with test data.

In a nutshell, I think they missed the distinction between have we built the right thing? and have we built it right?

TDD, Continuous Integration, refactoring – these will all help you build it right, but agile’s also about making sure you’re building the right thing, and I think SagePay dropped the ball on this one.

Mat even says (my emphasis)

“…any change introduces risk. … We might produce new software that performs identically to the old for a given payment protocol only to find that two thousand customers are using a non-documented “feature” of the old system that we’ve now written out."

Well, that’s exactly what they did. On Friday, https://live.sagepay.com/txstatus/txstatus.asp was returning real-time status for previous payment transactions – and now it’s gone. Vanished. What used to take 500 milliseconds in an automated script now takes a real person 90 seconds or so – including all that lovely horizontal scrolling they have to do.

Still, all is not lost. If SagePay really do have a clean, new architecture, and full test coverage, and a decent agile process in place, then it should be straightforward to respond to this customer feedback. Respond to change, instead of following a plan. Some tweaks at the presentation layer, maybe a couple of new properties on various view models and controllers, a handful of new methods on the supporting services, and it shouldn’t take long to deliver a product that combines the scalability and security of the new system with a UI that does exactly what the customers need.

Mat, if you’re reading this, I’d be really interested to hear what you guys did today whilst your support team were running around putting out fires. I’d love to see what your product backlog looks like right now, and how you’re condensing the torrent of feedback into user stories and work items. There’s not enough sharing in this industry, and if you and your team can be as open about things today as you were this time last week, you’ll probably help a whole bunch of people – including me – next time we find ourselves babysitting a tricky launch.

Making iPhone Ringtones using Free Software

2010-10-11T23:35:00+00:00

iPhone ringtones are audio files encoded as MPEG-4 audio and then renamed to use the .m4r file extension. If you don’t fancy paying £0.99 for twelve seconds of Van Halen, it’s pretty easy to roll your own iPhone ringtones, and you don’t even need to buy any software to do it. You’ll need:

An MP3 or WAV file of the song or noise that you want to use as a ringtone. Legally speaking, this could constitute copyright infringement, so please stick to original recordings or your own compositions.
Audacity, an open-source audio editor.
ffmpeg, the most awesomely powerful audio/video encoder ever. You’ll want to grab one of the Windows binary builds from http://ffmpeg.arrozcru.org/autobuilds/

Aside: I normally have Cygwin, a set of command-line Unix-a-like tools for Windows, installed into c:\windows\cygwin\, and then I add C:\Windows\cygwin\bin\ to my system path. Having done this, it makes sense to install ffmpeg into c:\windows\cygwin\ – it uses the same bin/doc/lib structure as most *nix ports and means I don’t have to add another path to the system PATH variable.

Import your source track into Audacity. Highlight the section you want, copy it, Ctrl-N to get a new file, and paste it.

At this point, playing around with effects might be worthwhile – especially selecting the first 1-2 seconds of the clip and using Effect –> Fade In, and applying a corresponding Fade out to the last second or two of your clip. When you’re happy with it, File –> Export as WAV…, and put it somewhere useful – D:\Ringtones\ works for me.

Then you’ll want to turn your WAV into an m4a file:

D:\ringtones>ffmpeg –i ringtone.wav ringtone.m4a

Input #0, wav, from 'ringtone.wav':
  Duration: 00:00:40.30, bitrate: 1411 kb/s
    Stream #0.0: Audio: pcm_s16le, 44100 Hz, 2 channels, s16, 1411 kb/s
Output #0, ipod, to 'ringtone.m4a':
  Metadata:
    encoder         : Lavf52.80.0
    Stream #0.0: Audio: aac, 44100 Hz, 2 channels, s16, 64 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
size=     427kB time=40.31 bitrate=  86.8kbits/s
video:0kB audio:413kB global headers:0kB muxing overhead 3.447648%

Once you’re done, either via Windows Explorer or the rename command, rename ringtone.m4a to ringtone.m4r

Finally, fire up iTunes, go to File –> Add File to Library… and browse to your new ringtone.m4r file. After a moment’s processing, it should show up in the Ringtones library. Right-click, Get Info… and you can replace the name, title, artist and so on, and then you just need to make sure it’s selected under the Ringtones heading on your iPhone sync menu.

Happy ringtoning.

/usr/bin/tawgo: Instant Geek Cred for Movies

2010-10-07T22:27:00+00:00

I had this idea a while back. You know how computers in movies are always some sort of interactive brightly-coloured touch-screen kind of deal? (“It’s a UNIX system… I know this!”) And you remember how excited all us techno-geeks got when Trinity used sshnuke to crack the security mainframe system in The Matrix Reloaded?

There should be a utility /usr/bin/tawgo, that’s standard on all Linux distros, that combines the audience appeal of brightly coloured cheese with the geek cred of technical accuracy.

[root@fortress]$ cd /home/dr.evil/
[root@fortress]$ tawgo "PREPARING TO COPY SECRET FILES..."
[root@fortress]$ cp -Rf * /mnt/floppy
[root@fortress]$ tawgo "SECRET FILES COPIED"

[root@fortress]$ tawgo --help

tawgo: Tell Audience What's Going On

Usage: tawgo [option] MESSAGE

Displays MESSAGE in big bright coloured letters, probably in some sort of futuristic animated dialog box.

-a --animation Show cheesy animation
-w --warning Use yellow & black warning stripes
-s --self-destruct Initiate fake countdown sequence
-v --voice Reads MESSAGE in a Female Computer Voice

Use -v -s if you need Female Computer Voice counting down the seconds to our hero's impending destruction.

[root@fortress]$ tawgo "INITIATING SATELLITE ALIGNMENT"
[root@fortress]$ /usr/sbin/comsatctl -a --lat=53.47.6 --lon=1.29.2
[root@fortress]$ tawgo "SATELLITE ALIGNED."
[root@fortress]$ tawgo "BEGINNING FIRING SEQUENCE"
[root@fortress]$ /usr/sbin/comsatctl --target 01 --fire
[root@fortress]$ tawgo -v -s "FIRING SEQUENCE INITIATED"

They’d save a fortune on expensive UI mockups, too – all you’d need is a laptop running Ubuntu and you’ve got your snazzy whizz-bang computer hacker Unix system all ready to go.

Stap Me Vitals! Scanning Domain Servers using System.Management

2010-10-04T17:00:00+00:00

In preparation for an overhaul of our hosting & network infrastructure, I wanted to gather stats on exactly what was running on all of our remote servers – stuff like how much physical memory is installed in the servers, what’s their Dell service tag, what sort of disk interface they’re running, what version of Windows, that kind of thing.

There’s a command-line tool wmic.exe that does some neat stuff by hooking into the Windows Management Instrumentation interface – little snippets like:

C:\>wmic /node:myserver cpu get description
Description
Intel(R) Xeon(TM) CPU 3.00GHz
Intel(R) Xeon(TM) CPU 3.00GHz
Intel(R) Xeon(TM) CPU 3.00GHz
Intel(R) Xeon(TM) CPU 3.00GHz


C:\>

- hey, looks like myserver has a quad-core Xeon CPU. Nice. Anyway, doing this across a whole lot of properties on a whole lot of servers would clearly be a bit time-consuming, so here’s how you do it using C# and the System.Management namespaces. The username/password specified in ConnectionOptions will need admin rights on all the servers you’re connecting to, and I’m sure you can reformat the output to look a little nicer, but in a nutshell, this’ll scan all the servers you specify and dump their vitals into D:\servers.txt

Simple - especially once you work out that the documentation for the cryptic magic strings inside the WMI classes is at http://msdn.microsoft.com/en-us/library/aa394084.aspx

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Management;

namespace ServerScan {
    class Program {

        static string[] servers = new string[] { "hugh", "pugh", "barney", "mcgrew", "cuthbert", "dibble", "grubb"};
        // see notes on Win32 instrumentation classes at http://msdn.microsoft.com/en-us/library/aa394084.aspx

        static Dictionary<string, string[]> wmiClasses = new Dictionary<string, string[]> {
            { "Win32_DiskDrive", new string[] { "Caption", "InterfaceType", "MediaType", "Name","Size" }},
            { "Win32_PhysicalMedia", new string[] { "Tag", "SerialNumber" }},
            { "Win32_Processor", new string[] { "Name", "ExtClock", "CurrentClockSpeed", "DeviceID" }},
            { "Win32_OperatingSystem", new string[] { "Name", "CSDVersion", "Description", "TotalVisibleMemorySize" }},
            { "Win32_SCSIController", new string[] { "Caption" }},
            { "Win32_SystemEnclosure", new string[] { "SerialNumber" }}
        };

        static void Main(string[] args) {
            var output = File.CreateText(@"D:\servers.csv");
            ConnectionOptions options = new ConnectionOptions();
            options.Username = @"MYDOMAIN\big.boss";
            options.Password = "password";
            foreach (var server in servers) {
                ManagementScope scope = new ManagementScope(String.Format(@"\\{0}\root\cimv2", server), options);
                try {
                    scope.Connect();
                    foreach (var wmiClass in wmiClasses.Keys) {
                        ObjectQuery query = new ObjectQuery("SELECT * FROM " + wmiClass);
                        ManagementObjectSearcher searcher = new ManagementObjectSearcher(scope, query);
                        Console.WriteLine("Searching {0} / {1}", server, wmiClass);
                        var results = searcher.Get();
                        foreach (var result in results) {
                            foreach (var thing in result.Properties) {
                                if (!wmiClasses[wmiClass].Contains(thing.Name)) continue;
                                output.Write(String.Format("{0}\t{1}\t{2}\t{3}\r\n", server, wmiClass, thing.Name, thing.Value));
                            }
                        }
                    }
                } catch (Exception ex) {
                    output.Write(String.Format("{0}\tFAIL\tFAIL\t{1}\r\n", server, ex.Message));
                }
            }
            output.Close();
            Console.Write("All Done!");
            Console.ReadKey(false);
        }
    }
}

Five Things I Wish My iPhone Did

2010-09-30T10:15:00+00:00

Wake Me Up When I Get To Bristol

For falling asleep on the train – or on the last bus home – this would be a killer feature; an alarm that goes off when you get within 1km of your destination, waking you up so you don’t miss your stop. For bonus points, it could calculate your average speed and last known distance from your destination, so if you've lost GPS signal, it’ll wake you up early just to be on the safe side.

Geographical Task Reminders

Every night I get home and realize that I’ve left my headphones in my desk drawer at work – again. It’d be cool to set a reminder so that next time I’m at work, it’ll remind me to put my headphones in my bag so I don’t forget them.

Charging Reminders Based On Wi-Fi Signals

I sometimes forget to charge my phone at work, and since it won’t last a full day & night without a top-up, this means it dies sometime during the evening, leaving me out & about with no music, no movies, no Twitter, no e-mail – oh, yeah, and no phone.

If my phone’s connected to my office wi-fi network, then I’m at work – so if I haven’t plugged it in to charge after 10-15 minutes, it should beep at me “hey, I know you’re at work – plug me in!”

Don’t Ask for a Unlock Code When On A Known Wi-Fi Network

Just like above – if my phone’s on my office or my home wi-fi network, then it’s hopefully not been stolen, and it’s probably safe to use it without entering the unlock code first.

Make Phone Calls and Send Text Messages Reliably

I know… wishful thinking. The iPhone is a lovely internet gadget, music player, movie player, sat-nav and all-round geek toy. But for actually communicating with other people, I still carry a battered old Nokia. It makes calls, it sends texts, and the battery lasts well over a week.

Fun with Powershell, SQL Backup and Automated Restores

2010-09-24T21:46:00+00:00

Our database backup system is based on taking a full backup of every database every night, and transaction log backups every 20 minutes. Backups are shipped offsite to a backup server sitting in a data-centre across town, and what I was trying to do was to automate the process of restoring the latest full backup and all subsequent transaction logs on the remote server.

So: ingredients. I have :

Red Gate SQL Backup (and the SqlBackupC.exe command-line client)
Windows Powershell
A list of databases in a text file (“databases.txt”)
A folder full of database backups, and a folder full of transaction log backups.

The backup filenames look something like:

D:\backup\
  data\
    FULL_myserver_Animals_20100921_183021.sqb
    FULL_myserver_Animals_20100922_183021.sqb
    FULL_myserver_Animals_20100923_183021.sqb
    FULL_myserver_Northwind_20100921_183021.sqb
    FULL_myserver_Northwind_20100922_183021.sqb
    FULL_myserver_Northwind_20100923_183021.sqb
  logs\
    LOG_myserver_Animals_20100921_190002.sqb
    LOG_myserver_Animals_20100921_200004.sqb
    LOG_myserver_Animals_20100921_210003.sqb
    LOG_myserver_Animals_20100921_220002.sqb
    LOG_myserver_Animals_20100921_230005.sqb
    ...
    ...

What I’m trying to do is, for each backup listed in databases.txt, I want to find the most up-to-date full backup, restore it, and then restore, in chronological order, every transaction log that’s been created since that full backup was taken.

So. Powershell… time to see what all the fuss is about.

Let’s start with a couple of things that threw me until I got my head around them. Powershell is a shell. Don’t think of it like C# or Java – think of it like a batch file on steroids. With objects. If you want to run a program from a Powershell script, you just put its name in the script. For example – this is a valid Powershell script that’ll open itself in Notepad:

C:\Windows\System32\Notepad.exe myscript.ps1

Copy that line into notepad, save it as as myscript.ps1, fire up Powershell.exe, navigate to the folder where you saved it, and type .\myscript.ps1 – bingo.

Secondly, Powershell is designed so that if something has a certain meaning in DOS, it’ll mean the same thing in Powershell. <, | and > will redirect and pipe output – just like in DOS – which means Powershell can’t use > as the greater-than operator – so they’ve used –ge instead. (Mmm. Perly.) Backslashes are path separators – just like in DOS – so Powershell uses the backtick (`) as an escape character.

Thirdly, Powershell supports Perl-style string interpolation.

$foo = 10
echo “I have $foo fingers”

will output “I have 10 fingers”. To avoid this behaviour, use single-quotes instead:

$foo = 10
echo 'I have $foo fingers'

will print "I have $foo fingers” Be careful, though – underscores are valid in variable names, so

echo “LOG_$server_$catalog_$timestamp”

is actually going to try and find variables called $server_ , $catalog_ and $timestamp. To work around this behaviour, enclose the variable name in braces:

echo “LOG_${server}_${catalog}_${timestamp}”

Finally – Powershell supports aliases, which means most commands have more than one name. For example, there’s a built-in command Get-ChildItem – and dir, gci, and ls are all shortcuts for this command, so when you bring up a Powershell command and want to list the contents of the current directory, it doesn’t care whether you type dir, ls, gci, or Get-ChildItem – they all do exactly the same thing.

OK. Time for scripty fun. Here’s the script – you’ll need to supply your own databases.txt and credentials, and probably tweak the

# Read the contents of databases.txt into a collection called $databases
$databases= Get-Content "databases.txt"

# Path to the SQL Backup command-line client.
$exe = "C:\Program Files (x86)\Red Gate\SQL Backup 6\SqlBackupC.exe"

# Credentials for connection to SQL Server. 
$username = 'sa'
$password = 'p@ssw0rd'

# Database backup encryption password (you *do* encrypt your backups, don’t you… ?)
$dbpw = 'T0P$3CR3T'

foreach($database in $databases) {

  # Get a collection of ALL backup files in the backup folder.
  $allSqbFiles = Get-ChildItem D:\backup\data\*.sqb 
  
  # Create a regular expression that'll match filenames
  # against the current database name
  $regex = "FULL_\(local\)_${database}_\d+_\d+\.sqb"
  
  # Filter the list of backup files to find those for the current DB
  $backups = $allSqbFiles | Where-Object { $_.Name -match $regex }
  
  # Sort the backups by LastWriteTime...
  $backups = $backups | Sort-Object LastWriteTime

  # and extract the most recent one
  $latestBackup = $backups | Select-Object -last 1

  # Capture the LastWriteTime of the most recent backup  
  $t = $latestBackup.LastWriteTime

  # Extract the full name of the database backup file  
  $backup = $latestBackup.FullName
  
  # Construct the SQL statement to pass to SqlBackupC.exe - note the 
  # doubled double-quotes used to include a double-quote in the result.
  
  $sql = """RESTORE DATABASE [$database] FROM DISK = '$backup' WITH PASSWORD = '$dbpw', NORECOVERY, REPLACE"""
  
  # This next bit is what actually does the database restore, and 
  # it's a bit fiddly.
  # & (the ampersand) is known as the "call" operator, and 
  # basically says "hey, run this command" Note that you CAN'T just
  # give it a big fat string containing the whole command, arguments
  # and everything. That's not how it works.
  # Second, the arguments beginning with a hyphen need to be
  # backtick-escaped so Powershell knows they're not operators.
  # Finally, note how we've included the $exe in quotes,
  # because the path to the SqlBackupC.exe has spaces in it. 

  & "$exe" `-U $username `-P $password `-SQL $sql
  
  # Use another regex to grab all the LOG files for the current database.
  $regex = "LOG_\(local\)_${database}_\d+_\d+\.sqb";
  $logFiles = Get-ChildItem D:\backup\logs\*.sqb
  $logFiles = $logFiles | Where-Object { $_.Name -match $regex }
  
  # Then we filter this list to return only those that are more
  # recent than the full backup (which we captured earlier)
  $logFilesToRestore = $logFiles | Where-Object {$_.CreationTime -ge $t } | Sort-Object CreationTime
  
  $logFileCount = $logFilesToRestore.Length
  
  # If there's no log files, we break out of the loop and move
  # onto the next database.
  if ($logFileCount -le 0) { continue }
  
  # Now, the LAST transaction log needs a slightly different SQL 
  # command, because we want the last restore to leave the database
  # in a usable state, so we need to split the logs into the final
  # one, and all the others. 
  $splitAtIndex = $logFileCount - 2
  
  # Powershell lets you slice arrays by saying $MyArray[x..y]
  $logFilesExceptFinaltOne = $logFilesToRestore[0..$splitAtIndex]
  
  $finalFogFile = $logFilesToRestore | Select-Object -last 1
  
  foreach($log in $logFilesToRestore) {
    $logFileName = $log.FullName
  
    # Construct the SQL statement to restore the transaction log
    # leaving the database ready to accept further log restores:
    $sql = """ RESTORE LOG [${database}] FROM DISK = '${logFileName}' WITH PASSWORD = '${dbpw}', NORECOVERY"" "
   
    # and run SqlBackupC with that SQL statement:
    & "$exe" `-U $username `-P $password `-SQL $sql
  }
   
  $logFileName = $finalFogFile.FullName
   
  # Construct the SQL statement to restore the last 
  # transaction log and leave everything ready for use
  $sql = """RESTORE LOG [${database}] FROM DISK = '$logFileName' WITH PASSWORD = '${dbpw}',
            RECOVERY, ORPHAN_CHECK "" "
   
   # and run it!
   & "$exe" `-U $username `-P $password `-SQL $sql
 }

Having never really used Powershell before, this whole thing took about three hours to put together – and I spent the first two hours cursing Powershell for being so damned awkward, and then it just clicked and suddenly everything made a lot more sense. Hopefully the tips above will save you a bit of frustration if you’re taking your first steps with it… and I’ve a feeling I’m going to be doing a lot more Powershelling before too long. For stuff like this, it’s so much more powerful than batch files, and so much more maintainable than writing little command-line utilities in C#.

Want a Free Red Gate Tool to Support Your Open-Source .NET Project?

2010-09-20T16:42:00+00:00

Earlier in the year, I helped Red Gate software out with some customer focus group stuff, and in return they’ve offered me a free license to any single Red Gate software tool

Since I’m already a happily-licensed user of Red Gate’s SQL Toolbelt and most of their .NET tools, I thought I’d pass this generous gesture on to the .NET community, so here’s the deal.

If you maintain an open-source .NET project, and you’d like a free license for any single Red Gate tool, drop me a short e-mail, tell me about your project, tell me which tool you’d like, and why it would be useful.

There is a single license up for grabs, for any of the tools listed on Red Gate’s products page at http://www.red-gate.com/products/index.htm

I’m not looking for ten-page essays – if your project is well-known, just tell me which tool you’d like and how it will help. E-mail your entry to dylan@dylanbeattie.net – please put “Red Gate Giveaway” in the subject line so they don’t end up in the spam folder, and remember to include your full name & e-mail so I can let you know if you’ve won. I’ll accept entries until this Friday, October 1st, at 10:00am (GMT), after which I’ll pick a winner and send their details across to Red Gate.

Wheaton’s Law applies; my decision is final, no correspondence will be entered into, etc, etc. Good luck!

.NET String Formatting Cheat Sheet

2010-09-15T17:28:00+00:00

This morning I saw this tweet from @jhollingworth:

which linked to this excellent guide to .NET string formats by Steve Tibbett

Sometimes, something is just so incredibly useful that even bookmarking it isn’t convenient enough – so I’ve hacked Steve’s guide and examples into a one-page A4 cheat sheet and stuck it on the wall next to my desk.

It’s up on my site as a PDF or as a Word document – download it, print it out, stick it on the wall, and you’ll never find yourself Googling {0:n} vs {0:c} again.

All credit to Steve Tibbett for this – all I did was make it fit on one printed page. Happy formatting.

Automating Secure FTP Backups to a Windows 2008 Server

2010-09-13T13:36:00+00:00

I’ve been looking into solutions for shipping database backups and log-files to a remote server as part of our backup strategy. Requirements are pretty simple:

It’s got to be completely automated
A daily run has to take less than 6 hours so it’ll finish overnight whilst our connection is quiet
Transfers need to be encrypted to stop people sniffing our packets
I don’t want to spend a lot of money
If I can get away without lots of complicated (read: fragile) command-line switches, then that would be a bonus

Now, there’s four common protocols for doing secure file transfers - SFTP, FTPS, FTP+SSH, and SCP. The specifics aren’t important – what matters is that they’re incompatible, so your client and server have to be using the same protocol, otherwise nothing will work. There’s also various dedicated tools such as rsync, robocopy, xcopy, SyncBack, DeltaCopy… the list goes on. Setting up rsync on Windows is a bit fiddly – not to mention the dozens of subtle and nefarious command-line switches available to rsync.exe – and whilst I love a bit of cygwin hacking as much as anyone else, I’d really rather use something a little more intuitive here. I did try DeltaCopy, a GUI wrapper around rsync, but I quickly ran into security problems with SSH keys and username/password combinations. Since I wanted to avoid open folder shares on the remote system, tools like xcopy and robocopy were out.

In the past, I’ve had great results with Vandyke Software’s VShell, a commercial Windows SSH server, but whilst putting this lot together, I discovered that in Windows Server 2008, you can set up an FTP server to encrypt connections via SSL (it’ll even use the same certificate as your website). That’s good, by the way – I think things are always cheaper & easier when Windows does them out of the box.

So – assuming you’ve already got a Windows 2008 Server running IIS, and you’ve already got a certificate installed, then log onto the server, fire up IIS, create a new FTP site, switch on SSL, bingo.

Just to sanity-check that everything’s working, fire up WinSCP, create a new session using FTP with SSL Explicit Encryption, and you should be able to connect to your new secure server.

OK – so far, so good. We can connect, we can see a directory listing. Now what I want to do is to automate the client side of things, and for this bit, I’m going to use a superb bit of software called Super Flexible File Synchronizer. This is something I’ve only discovered recently, but so far, I’m delighted with it. It’s got native support for FTP. SFTP, WebDAV and HTTP, as well as Google Docs, Amazon S3 and Azure (!), plus a built-in scheduler, mail notification facility… as I said, it’s looking very promising.

Right, time for some actual computer science – complete with experimental data and graphs and everything. I want to know how fast this thing is going to go, and I want to know how much the encryption’s going to slow things down, so I’ve put together 72Mb of assorted database backups, TIFF files and documents, and transferred them up to the remote server using various settings. Results here are the average of five runs for each combination.

There’s a slightly odd setting in Super Flexible File Synchroniser – when you’re setting up an Internet target, if you pick FTP, it’ll give you a choice of libraries – cryptically called 1, 2 and 3.

I’m guessing that under the hood, there’s three different off-the-shelf FTP libraries in there, and the option lets you pick another one if the default doesn’t play nicely with your particular server. (Yeah, like anyone’s ever going to need that…)

So, numbers. I ran five different settings – network share (i.e. copying straight to \\myserver\backups using Windows file sharing), “open” FTP (no SSL), and then each of the three FTP libraries with the SSL option switched on.

	Total Time	Time per File			Transfer Speed
	Total Time	Average	Fastest	Slowest	Average	Slowest	Fastest
FTP	00:00:46	2	0	16	1,944	146	2,274
SMB	00:00:45	2	0	16	1,976	86	2,451
FTP+SSL (Library 1)	00:08:22	22	0	41	148	1	870
FTP+SSL (Library 2)	FAIL
FTP+SSL (Library 3)	00:00:40	2	0	14	2,161	129	2,521

A couple of things to note:

FTP Library 2 just didn’t work – it wouldn’t even connect to the remote server to retrieve a directory listing.
FTP Library 1 clearly had some issues – a typical run took well over eight minutes, and the subsequent logs were littered with timeouts and stuff like this:

COPY L->R d:\FileDemo\F00090820-0103.tif (889.7kB)
FTP Exception (5): EclSocketError Timeout error occured @ 009283D0
COPY L->R d:\FileDemo\F00090820-0104.tif (851.6kB)
FTP Exception (5): EclSocketError Timeout error occured @ 009283D0

Fortunately – FTP Library #3 worked perfectly, and gave transfer speeds that were actually faster than the raw FTP connection. I’m thinking that’s probably down to variations in test conditions (time of day, traffic. and so on) but even allowing for a little experimental error, it’s definitely fast enough to work with.

We’re looking at transferring about 20Gb worth of database backups and photographs a day. Based on the stats above, we can do 75Mb in 45 seconds, which equates rather neatly to 100Mb / min, which means our 20Gb backups are going to take just over three hours. $60 for a copy of SuperSynchroFlexofile O’Maticalyzer (and I’ll throw in a case of beer if you can come up with a snappier name for the next edition), SSL all the way, and a nice GUI to set it all up.

Isn’t it nice when everything just… works?

Heisenbug of the Day: IIS 7.0 Discarding POST Data From Firefox 3 when using Custom 404 Handlers

2010-08-23T12:17:00+00:00

Our site uses IIS custom error handlers, so that when you request /not/a/real/page.html, it’ll actually run /errors/404.asp – there’s a nice article on 4guysfromrolla about how you do this in classic ASP.

In theory, this works for both GET and POST requests, but last week we hit a snag – some of our jQuery Ajax code wasn’t working properly in Firefox 3. More specifically – it worked fine locally, it worked fine in all other browsers, but when we deployed the code to any of our test or live servers, it wouldn’t work in Firefox. IE, Opera, Safari, Chrome – all fine; it seems like only Firefox was affected.

Method	Server	Request URL	Result
GET	IIS 7.5 (Windows 7)	/errors/404.asp	OK
POST	IIS 7.5 (Windows 7)	/errors/404.asp	OK
GET	IIS 7.5 (Windows 7)	(404 handler)	OK
POST	IIS 7.5 (Windows 7)	(404 handler)	OK
GET	IIS 7.0 (Windows 2008)	/errors/404.asp	OK
POST	IIS 7.0 (Windows 2008)	/errors/404.asp	OK
GET	IIS 7.0 (Windows 2008)	(404 handler)	OK
POST	IIS 7.0 (Windows 2008)	(404 handler)	FAIL

Firebug didn’t show up anything unusual, so we fired up Fiddler, a web debugging proxy that’ll show you what’s actually being passed between the client and the server. At least, that’s the idea… what actually happened is that when we started running Fiddler, the bug went away. Yep… we had ourselves a real live Heisenbug:

Heisenbug: “…a computer bug that disappears … when an attempt is made to study it.” [via Wikipedia]

Fiddler runs as an HTTP-level proxy – in other words, it understands the HTTP protocol, and sits between your web browser and your web server, and – in theory – transparently forwards information between them, whilst recording all the bits that fly backwards and forwards so that you can dissect them and see what’s going on. I’d guess that, somehow, Firefox was sending dodgy requests, and Fiddler was cleaning up these requests as part of the proxying process – hence why the problem disappeared when Fiddler was running.

Time to dig a little deeper. Wireshark is a deep-level network protocol analyser that’ll sniff your network traffic right down to the frame level. What I did next was to load up Wireshark, set up a filter [1] to show only HTTP traffic to/from our build server, and then submit the same request from a couple of different browsers – including Firefox.

The first grab there is what’s travelling over the wire when you POST that form using Google Chrome; the second is the same POST submitted using Firefox. Remember – at this point, we’re totally lost and so looking for absolutely anything that’s different. If you look closely, you’ll see the Chrome trace includes an extra line - [Reassembled TCP Segments (680 bytes)] – that isn’t in the Firefox trace. They’re otherwise identical other than known differences like the User-Agent string and so on. Curious. A bit of experimentation verifies that Safari and IE are doing the same thing as Chrome – submitting two frames of data for each POST – where Firefox is only submitting one.

It turns out this triggers a bug in IIS 7.0 when you’re using custom 404 handlers.

Bad Analogy Time…

Imagine it’s your birthday. You’ve got a load of packages to open - you open the first one, and there’s a card inside saying “Happy Birthday! Enjoy the Lego! Love Granny xxx”

Now – at this point, you’re expecting some Lego, right? Well, if Granny is Chrome, IE or Safari, she’s been sensible – she’s sent the card in its own envelope, and put the Lego in the next parcel. But, if Granny is Firefox, then Granny has done something foolish, and has crammed as many of the Lego bricks as she can into the same envelope as the birthday card. If the Lego set is only tiny, then she can fit all the bricks into the envelope – and so won’t bother sending the now-empty Lego box.

So… imagine the envelopes/parcels are TCP frames, the birthday card is your HTTP request, and the Lego is the associated POST data. The card (headers) say “hey, there’s more stuff coming” – and then somewhere close behind, there’s another package with that “stuff” in it.

Now, onto the IIS 7.0 bug. Under normal circumstances, IIS 7 copes just fine with POST data being in the same frame as the actual request. (That’s why this bug doesn’t affect every Firefox user who visits an IIS7 site.)

Thing is - when a request is processed by a custom 404 handler, IIS 7.0 is opening the envelope, finding the birthday card, going “whooopeee! Lego!” – and then throwing the envelope away without checking to see if there’s any Lego in it, before looking around excitedly to see where the next parcel is.

For very small POSTs, this results in the Request.Form being empty (because all the Lego has been thrown away with the envelope). If you deliberately pad your POST with a couple of really long fields - <input name=”padding” value=”xxxxxxxx … xxx” /> for 2000 characters or so – then you’ll see that even Firefox now has to split the request over more than one frame, and that any POST values that end up in the second frame are now accessible to IIS via Request.Form in the usual way. Kinda like Granny sending you a really big Lego set, and putting the first 20 or so bricks in the envelope with the birthday card, and the rest in a separate parcel or two – throw away the envelope, and you’ve still got *most* of the bricks, but many of them have gone missing.

So… workarounds. Firefox patch – no good. Too many installed users. Upgrade all our web servers to IIS 7.5? Er, not right now, thanks. IIS hotfix? Lovely – if you’ve got one, send it over.

In the meantime, the best option we could find was to stick two hidden fields at the top of the affected form, something like:

<input type=”hidden” name=”ff_frame” value=”xxxxx . . . 1460 Xs here . . . xxxx” />
<input type=”hidden” name=”ff_split” value=”1” />

<!—everything after this point will show up intact in Request.Form -->

<input type=”hidden” name=”real” value=”some_data” />

The big string of X’s pads the first frame to make sure all your real data ends up in the second one, and the ff_split value ensures that this padding doesn’t mess up IIS’ parsing of subsequent POST values. Yes, this is disgusting - and it adds 1Kb+ to every POST - but it’s only required in a handful of places, and we’re looking to isolate it inside the jQuery code we’re using so it’ll be dynamically inserted into POSTs where necessary.

[1] ((http.request || http.response) && (http.host contains "build")) && !(http.request.uri == "/favicon.ico")

Uncle Bob’s Bowling Kata in Functional C#

2010-04-24T18:20:00+00:00

During the session on functional programming at OpenVolcano10 this week, Uncle Bob Martin spoke of a particularly elegant functional solution to his bowling game kata. Proposed by Stuart Halloway, the solution regards the bowling game itself as a (potentially infinite) sequence of ‘rolls’ (balls), and then uses some nicely elegant list manipulation to extract ten valid scoring frames from the sequence of balls and calculate the score of the game.

C# and IEnumerable<T> allow you to do some very similar things in .NET – basically, you can define operations in terms of mapping one infinite list to another infinite list, and you don’t have to worry about stack overflows or out of memory errors because, thanks to the wonder of lazy evaluation, nothing is actually allocated or returned until you start asking for the resulting values.

Take a look at this code snippet:

static IEnumerable<Int64> Integers {
    get {
        for (var i = 1; true; i++) yield return (i);
    }
}

If you’re thinking “but that’s just an infinite loop!” – you’re partly right. Yes, it’s infinite, but no, it’s not a loop. The magical yield return operator there actually breaks the loop, and allows you go to infinity (and beyond!) – one step at a time. Using the LINQ Take() method, you can easily slice’n’dice this infinite list into useful pieces:

// Calculate the sum of the first 20 positive integers:
var sum = Integers.Take(20).Sum();

// Calculate the product of the first 10 positive integers:
var product = Integers.Take(10).Aggregate((a, b) => (a * b));

By throwing some extensions methods onto Int64, you can use the same approach to filter the list:

public static class ExtensionMethods {
    public static bool IsPrime(this Int64 candidate) {
        if ((candidate & 1) == 0) return (candidate == 2);
        var num = (int)Math.Sqrt((double)candidate);
        for (var i = 3; i <= num; i += 2) {
            if ((candidate % i) == 0) return false;
        }
        return true;
    }
}

// now we’ve defined myInt64.IsPrime(), we can do this:
var primes = Integers.Where(i => i.IsPrime());

We now have a C# variable – primes – that, for all intents and purposes, contains all the prime numbers. If we try to print the entire list, our program will never terminate – but that’s what we’d expect, because the list of prime numbers is infinite. We can, however, do things like printing a list of the first 100 primes:

foreach(var prime in primes.Take(100)) Console.WriteLine(prime);

or calculating the product of the first 50 primes:

var product = primes.Take(50).Aggregate((a,b) => (a*b));

So, what’s all this got to do with bowling? Well – using this approach, we can store the progress of a bowling game as a (potentially infinite?) list of rolls, and by using a couple of useful extension methods on IEnumerable<int>, we can calculate the resulting ten-pin bowling score in a single LINQ statement:

var score = rolls.ToFrames().Take(10).Sum(frame => frame.Sum());

Here’s the supporting extension methods - which basically capture the scoring quirks of spares, strikes, frames and the other idiosyncracies of ten-pin-bowling. The ToFrames() method is particularly interesting – it’s translated from the cons operator in Lisp/Clojure, and effectively returns a list (the rolls that count towards the current frame), followed by a list of lists (where each list represents a subsequent frame in the game):

public static class BowlingRules {
    public static IEnumerable<IEnumerable<int>> ToFrames(this IEnumerable<int> rolls) {
        yield return (rolls.Take(rolls.BallsToScore()));
        foreach(var frame in (rolls.Skip(rolls.FrameAdvance()).ToFrames())) {
            yield return(frame);
        }
    }

    private static int FrameAdvance(this IEnumerable<int> rolls) {
        return (rolls.IsStrike() ? 1 : 2);
    }

    private static int BallsToScore(this IEnumerable<int> rolls) {
        return (rolls.IsBonus() ? 3 : 2);
    }

    private static bool IsStrike(this IEnumerable<int> rolls) {
        return (rolls.Take(1).Sum().Equals(10));
    }

    private static bool IsSpare(this IEnumerable<int> rolls) {
        return (rolls.Take(2).Sum().Equals(10));
    }

    private static bool IsBonus(this IEnumerable<int> rolls) {
        return (rolls.IsSpare() || rolls.IsStrike());
    }
}

The project – including the unit test from Uncle Bob’s kata translated to NUnit / C# – is up on Google Code if you want to take a look.

Dynamic CSS with .less at the TechDays Open Source Evening

2010-04-14T22:45:00+00:00

Thanks to all of you who came along to last night’s Open Source .NET evening (and extra thanks to @serialseb for putting the whole thing together!) I think we managed to cover a really great programme of topics – OpenRasta, Ben Hall using Ruby to test ASP.NET projects, Mike Hadlow talking about the Windsor IoC container, Jeremy Skinner presenting his FluentValidation library (and a late apology from Neil Robbins, whose CouchDB talk had to be cancelled after his laptop went into ‘permanent hibernation’ at the last minute… I guess the long cold winter affects laptops as well as people)

I should make one very important clarification: dotLess is NOT my project! The original Ruby project was created by Alexis Sellier and Dmitry Fadeyev, and it was ported to .NET by Christopher Owen, Erik van Brakel and Daniel Hoelbling. Apologies if this wasn’t clear from the presentation – from the questions afterwards, I appear to have inadvertently given the impression I was one of the project contributors. I’m not; I just think it’s an awesome library and wanted to give it a bit of exposure.

You can download the demo code from last night’s talk here:

http://www.dylanbeattie.net/dot_less_demo.zip

You should just be able to unzip the whole lot into a folder, bring up LessDemo.sln in Visual Studio and hit “Run” – but let me know if you’re having any problems with it.

The other bits I spoke about in the talk are:

Project Homepage and Downloads: http://www.dotlesscss.com/
Ruby LESS Library: http://lesscss.org/
Phil Haack’s T4 Template (.less to .css at build time):
http://haacked.com/archive/2009/12/02/t4-template-for-less-css.aspx
Opening .less files in Visual Studio 2010’s CSS editor
http://stackoverflow.com/questions/2346243/

Finally, I’m very excited to see that for the next version of LESS they’re migrating away from server-side Ruby and porting the entire library to Javascript – so you can run it client-side or server-side depending on your setup. Nothing’s officially released yet, but you can get download the latest source from the less.js project on GitHub – definitely a project worth keeping an eye on.

A Manager’s Perspective - Why Attend Progressive.NET Tutorials?

2010-03-02T14:35:00+00:00

SkillsMatter are organizing another series of their Progressive.NET tutorials this May. I’ve heard from several people who would love to attend but are struggling to convince their managers that it’s worth the money (and to justify the time out of the office) – so I’d like to take this opportunity to go on the record, as a software manager, to endorse these tutorials.

If you’re in a hurry, here’s the quote:

I’m Dylan Beattie. I’ve been programming professionally for over 10 years, I’ve run a software team for 5 years. I believe that the SkillsMatter Progressive.NET tutorials offer more knowledgeable speakers, more relevant content, and better value for money than any other paid training course I have ever attended – and I believe that the skills learned at these tutorials will almost immediately result in improved productivity and better-quality software.

Progressive.NET is not a dry textbook lecture delivered by some professional trainer who doesn’t care what happens after you leave the room. It’s enthusiastic, passionate experts, who hate wasting time and money as much as you (and your boss) do, sharing lessons they’ve learned the hard way, and demonstrating tools that will help you build better software faster.

I manage a team of Web developers, who build software on Microsoft platforms – Windows, IIS, SQL Server, ASP, ASP.NET, MVC, and every kind of data access framework from good old ADO through ADO.NET, Linq to SQL, Castle ActiveRecord and Fluent NHibernate. Between us we’ve built and shipped software using almost all these platforms. We’ve spent painful hours and days tracking down obscure bugs in seven-year-old VBScript code, and we’ve seen how smoothly everything happens once you have unit tests, continuous integration and a decent development environment in place.

Over the years, my team & I have been on various “proper” developer training courses, including a five-day Microsoft-certified ASP.NET course that cost thousands of pounds and was basically a complete waste of time.

Three of us attended the SkillsMatter Progressive.NET tutorials last year, and found the event to be very, very good. The hardest thing about adopting a new technology like NHibernate or Castle is working out where it can help, and what it can do – and the best way I’ve found of getting this “guided tour” of a project’s capabilities is to spend a few hours with an expert like Ayende or Hammett showing you how it works and how it all fits together. One of my team also said that it was the first time he’d really spent time out of the office thinking about how to write better software – and he’s right; it’s hard to find the time when you’re at your desk day in, day out, to really take a step back and think “could I be doing this better?”

From a team lead / management perspective, the lessons learned and insights gleaned from the sessions at last year’s tutorials have made it much easier to base new projects on tools like NHibernate – and this has resulted in genuine time savings. We’re delivering better code, we’re delivering it faster, we’re not wasting time writing stored procedures and SELECT statements. We have less bugs. We have better release processes. We are more productive because we are inspired by learning new techniques, we’re using the right tools for the job, and we feel like part of a larger community.

More importantly, having sat in a room full of people sharing the same interest in software and software development, we don’t feel like we’re going off on some crazy hippy open-source tangent. We’ve talked IoC and ORM with people who write e-commerce systems, insurance software, finance software and major enterprise systems for healthcare organisations. It’s a radically different perspective from trying NHibernate because you read about on some guy’s blog. It’s given me, my team, and our business stakeholders confidence in these tools, and access to a rich network of fellow users and experts who are happy to advise and help out when we get a bit lost.

Fun and Games with NHibernate and String Keys

2010-02-17T11:33:00+00:00

One of the team spent much of yesterday banging his head against the wall. We had a query that took 18ms to run in Query Analyzer, and 1800ms to run from NHibernate. NHibernate Profiler would report 1800-2200 ms – and copying’n’pasting the exact same query statement into Query Analyzer would run it in < 20 ms. Running the same SQL statement via ADO.NET, from the same block of code, proved it wasn’t a network latency issue or anything - ADO.NET came back in 20ms, NHibernate still took over 2 seconds.

Finally, thanks to SQL Profiler, we managed to work out what was going on. The table in question had a primary key which was a varchar(9); in this instance, we were filtering on a particular primary key value – and NHibernate was specifying the value as an nvarchar(6), because NHibernate treats strings as unicode (nvarchar) by default. This C# code

currentSession
    .CreateCriteria<UserFeedItem>()
    .Add(Expression.Eq("User.UserCode", “abc123”)) // is a varchar(9) in the database for historical reasons
    .AddOrder(Order.Asc("Received"))
    .SetFirstResult(firstResult)
    .SetMaxResults(resultCount)
    .List<UserFeedItem>()
);

was resulting in the following SQL statement

exec sp_executesql
    N'SELECT TOP 50 FeedItemId16_0_, FeedUser16_0_, MailSent16_0_, ItemSeen16_0_, Deleted16_0_, Received16_0_
    FROM (
        SELECT this_.FeedItemId as FeedItemId16_0_, this_.FeedUser as FeedUser16_0_, this_.MailSent as MailSent16_0_, this_.ItemSeen as ItemSeen16_0_, this_.Deleted as Deleted16_0_, this_.Received as Received16_0_, ROW_NUMBER() OVER(ORDER BY this_.Received) as __hibernate_sort_row FROM Spotweb.dbo.[FeedUserFeedItem] this_ WHERE this_.FeedUser = @p0) as query WHERE query.__hibernate_sort_row > 50 ORDER BY query.__hibernate_sort_row',
    N'@p0 nvarchar(6)',
    @p0=N'abc123'

This means SQL Server has to do an implicit conversion when comparing your parameter value to the table’s primary key value – instead of using indexes, it has to retrieve every row, cast the primary key to an nvarchar(6), and then see whether the result is equal to your parameter. As you can imagine, this slows things down a little – especially on a table with several million records.

The immediate solution for this was to rewrite the code as follows:

currentSession
    .CreateQuery("from UserFeedItem where User.UserCode = :code")
    .SetParameter("code", “abc123”, TypeFactory.GetAnsiStringType(9))
    .SetFirstResult(firstResult)
    .SetMaxResults(resultCount)
    .List<UserFeedItem>()
);

- notice how on the third line we’re explicitly specifying TypeFactory.GetAnsiStringType(9) to force NHibernate to specify the parameter as a varchar(9). A better solution is to explicitly specify the mapping type for the column – we’re using Fluent NHibernate, so the mapping override code looks like this:

public class SiteUserOverrides : IAutoMappingOverride<SiteUser> {
    public void Override(AutoMapping<SiteUser> map) {
        map.Id(u => u.UserCode).CustomType(“AnsiString”);
    }
}

and now any reference in any query to a SiteUser.UserCode will be properly mapped as varchar instead of nvarchar.

Moral of the story: be careful with varchar and nvarchar keys when using NHibernate; make sure there aren’t implicit type conversions happening all over the place, because they can seriously mess up your performance.

It’s also worth noting that even the excellent NHibernate Profiler couldn’t shed any light on what was going on here – SQL Profiler (included with SQL Server) is still an incredibly powerful and under-used tool, and it’s well worth spending a couple of hours getting to grips with it.

Fun with msdeploy, Extended Protection and Windows Authentication

2009-12-09T16:53:00+00:00

Trying to push a change to our live servers today, we got this wonderful message from msdeploy:

Error: (09/12/2009 16:10:21) An error occurred when the request was processed on the remote computer.

Error: Child object 'extendedProtection' cannot be added to object 'windowsAuthentication'. The 'windowsAuthentication' provider may not support this deployment.

Error count: 1.

We do this several times a day… it worked yesterday, nobody changed anything, and suddenly today it doesn’t work. (Don’t you just love it when that happens?)

Now, msdeploy is wonderful, but has practically no documentation – which means when an error like this happens, you’re on your own.

A bit of Googling and a couple of lucky guesses later, and we worked out what was causing it. “Extended Protection” is apparently some of new fangled security framework that’s included in recent Windows Updates. Our internal servers install Windows updates automatically; our live servers don’t.

In other words – our staging server had quietly upgraded itself in the night to support extended authentication, and was now trying to push that configuration to the live server, which had absolutely no idea what was going on.

Logging on to the live server and installing the outstanding Windows security updates seems to have fixed it.

Axure RP: Lego For Software Designers

2009-11-07T13:36:00+00:00

Someone asked me on Twitter a little while back:

@dylanbeattie Do u currently use Axure? If so, could u pls tell what size team it's effective with? And, is it really better than pen-paper?
11:38 AM Nov 7th from web in reply to dylanbeattie

The short Twitter answers are “yes”, “we’ve used it effectively with up to 4 people”, and “yes, because I can’t draw” – in that order.

But a slightly more detailed response would probably help.

Once upon a time, prototyping web apps was easy. You’d draw every page, and then use a site map to demonstrate which links went where. Every page was static; nothing moved, there was no Ajax, no infinite scrolling, no drag’n’drop, and most websites were actually about as interactive as a Choose Your Own Adventure novel. Well, those days are gone. People expect more – richer UIs, better responsiveness, less postbacks and waiting around for pages to load – and with libraries like jQuery, there’s really no excuse for not delivering code that satisfies those expectations.

Question is – how do you prototype a rich user interface? How do you draw a picture of something that won’t sit still? For me, that’s where Axure RP comes in. Axure is a “tool for rapidly creating wireframes, prototypes and specifications for applications and web sites”. It’s a commercial product, so it is, alas, not free (although to put it into perspective, it costs less than hiring a .NET developer for one day) – but it is a uniquely powerful and expressive piece of software that I find myself firing up on an almost daily basis.

In everyday use, it’s like a weird cross between Balsamiq, Visual Basic, and Lego.

Balsamiq, because it’s easy to mock up static user interfaces by dragging buttons, inputs and form elements on to your page.
Visual Basic, because it’s easy to add behaviour to those elements using click handlers, events and dynamic controls.
Lego, because it’s intuitive, and it’s fun, and there is no way anyone is going to look at what you’ve done and think the project is finished.

The game Populous was designed using Lego. I grew up with Lego*. From a very early age, I learned to use Lego bricks to express ideas. I knew every single brick I owned. I could demonstrate an idea I’d had for a car, or a spaceship, or a robot, by assembling these reusable components into a prototype with spinning wheels and moving parts and a sense of scale and colour. Working entirely in plastic bricks actually become very liberating, because it stops you worrying about materials and finishes, and allows you to focus entirely on expressing ideas.

Have you ever showed someone a Lego house and had them say “Hey, that looks great! When can we move in?” No. People know a Lego house is not a real house. They appreciate that the point of a Lego - or cardboard, or clay - model is to demonstrate what you’re planning to do, not show off what you’ve already done.

Have you ever showed anyone an HTML mockup of a web app and had them say “Hey, that looks great! When do we launch?” – and then they look horrified when you explain that you haven’t actually started the build yet?

People don’t grok the difference between HTML mockups and completed web apps the way they grok the difference between Lego houses and real ones. I can’t say I blame them. HTML is HTML – whether it was hacked together late last night in Notepad or generated in the cloud by your domain-driven MVC application framework. The difference doesn’t become apparent until they actually start clicking things – by which point it’s too late; you’ve made your first impression (“wow, the new app is done!”) and it’s all downhill from there.

I think the hardest questions in software are “what are we doing?” and “are we done yet?”. I think good prototypes are absolutely instrumental in answering those questions, and any tool that can help us refine those prototypes without falling into the trap of “well, it looks finished” has to be a Good Thing.

* Some people look at how much Lego I still have and conclude that I never grew up at all…

A (Slightly Faster) URL Resolver Module for ASP.NET MVC

2009-11-05T00:57:00+00:00

Yesterday, I posted some code I’d hacked together as part of an MVC2 demo that would resolve ASP.NET virtual path URLs on the fly as pages were written to the ASP.NET response stream.

Having run some tests on this code in isolation – it’s actually quite nasty. For running ad-hoc demos on your workstation, it’s fine, but the performance hit of decoding the byte array, doing the regex transform and re-encoding it is something like two hundred times slower than a direct stream copy. Not good. There is now a modified version online at Google Code which is quite a bit faster, but there’s still huge scope for improvement. In particular, although it’s using byte comparisons now to work out where the ~/ combination occurs, it’s still falling back to string comparisons every time it finds a tilde to decide whether that tilde needs replacing or not.

These stats were created using a loop that spins up HTML pages of various sizes – one version full of ASP.NET-style tilde paths, one containing no tildes - and then writes them 100 times to both a normal MemoryStream and a UrlResolverStream in order to calculate the average rendering time. If a page doesn’t contain any tildes at all, performance is 5-6 times slower than the equivalent direct memory copy – i.e. 21ms instead of 4ms. For pages with lots of tildes, the additional string processing hits quite hard and you’re looking at a slowdown factor of around 30-35x.

Tildes?	Page Size	MemoryStream copy (ms)	UrlResolverStream copy (ms)	Ratio
Yes	50Kb	0.05	0.38	7
Yes	101Kb	0.17	2.51	14
Yes	202Kb	0.19	6.05	31
Yes	405Kb	0.37	13.88	37
Yes	809Kb	0.88	29.47	33
Yes	1,619Kb	1.62	60.55	37
Yes	3,238Kb	3.45	123.40	35
No	50Kb	0.02	0.34	17
No	101Kb	0.07	0.66	9
No	202Kb	0.15	1.28	8
No	405Kb	0.48	2.62	5
No	809Kb	0.83	5.26	6
No	1,619Kb	1.66	10.57	6
No	3,237Kb	3.53	21.17	5

It should be possible to make this considerably faster still; since the code basically scans byte arrays, this is one of those areas where using pointer arithmetic could make a huge difference. I’ll dig out my unsafe hat and my pointer-gloves this weekend and see what I can do to it. In the meantime – play with it, experiment with it, but it’s probably a good idea not to let it within a hundred miles of your live servers :)

A URL Resolver Module for ASP.NET MVC

2009-11-04T01:42:00+00:00

Update: An improved version of this module, along with some performance stats, is available here. The original version posted here was very, very slow. Probably not a good idea to use it for anything. Ever.

One of the few things I actually liked about ASP.NET WebForms was that you could do things like

<a href=”~/my/account.aspx” runat=”server”>My Account</a>

and ASP.NET would magically turn the tilde character (~) into the current relative application root – so you could debug your apps on http://localhost:4567/ and then deploy them to http://www.myserver.com/some/app/, and your links wouldn’t break.

ASP.NET MVC doesn’t like things that are runat=”server” – and with good reason, I think – but this does mean you can end up with rather a lot of calls to ResolveUrl() sprinked throughout your code.

To get around this, I’ve hacked together an HTTP module that basically rewrites the output stream on the fly. It wraps the HTTP output stream (the thing you're writing to when you Response.Write stuff) in a 'smart' stream wrapper, and the ~~magic~~ naively optimistic part looks like this:

public override void Write(byte[] buffer, int offset, int count) {
  if (HttpContext.Current.Handler is System.Web.Mvc.MvcHandler) {
    HttpContext.Current.Trace.Warn("Resolving URLs in output stream...");
    byte[] data = new byte[count];
    Buffer.BlockCopy(buffer, offset, data, 0, count);
    string html = Encoding.ASCII.GetString(data);

    // Don't try and use Regex transformations on your 
    // entire output stream. It is slow. Like, really, really slow.
    // Take a look at this updated version instead.

    var re = new Regex("(?src|href|action)=\"~/", RegexOptions.IgnoreCase | RegexOptions.Compiled | RegexOptions.Multiline | RegexOptions.ExplicitCapture);
    html = re.Replace(html, "${attr}=\"" + VirtualPathUtility.ToAbsolute("~/"));
    data = Encoding.ASCII.GetBytes(html);
    sink.Write(data, 0, html.Length);
    HttpContext.Current.Trace.Warn("Resolved URLs in output stream.");
  } else {
    sink.Write(buffer, offset, count);
  }
}

Basically, it looks for HTML SRC, ACTION and HREF attributes whose value begins with ~/, and replaces the ~ with the application’s virtual path on the fly. ~~I haven’t tested this code for performance, so I don’t know what kind of impact it’ll have on your page response times,~~ This code is something like 200 times slower than a straight stream copy, but it’s running in a couple of demo apps I’m working on and it seems to work ~~pretty nicely~~.

The full implementation is over on Google Code if you’re interested.

Code from my HTML5 / MVC2 Talk at SkillsMatter

2009-11-04T01:08:00+00:00

A big thank-you to everyone who came along to my HTML5 and MVC2 talk at SkillsMatter on Monday – and thanks also to SkillsMatter for hosting us! Whilst I’ve done plenty of talking at unconferences and events like BarCamp, this was the first proper full-length technical talk I’ve given, so I’d really appreciate any feedback – especially since we might be doing a re-run in a couple of weeks.

During the talk, I demo’ed a tiny web app – TagAlong - that I’ve built to showcase some of the new features in HTML 5 and ASP.NET MVC preview 2. This is by no means production code – if nothing else, I’m using static List<T>’s instead of having an actual database, so your changes will disappear every time you restart the app – but it should be pretty easy to get it up and running and poke around.

If you’re interested, the code is online at http://code.google.com/p/tagalong – you’ll need MVC 2 Preview 2 installed to run it, but everything else is included.

A couple of other interesting links that I mentioned during the talk:

The page structure diagrams are taken from A List Apart
Dean Edward’s work-in-progress Javascript library to emulate HTML5 in downlevel browsers is online at html5-now.appspot.com
Kevin Roast created the Asteroids demo
The Fireworks demo was created by Kenneth Kufluk

You Forgot to Say the Magic Word…

2009-10-20T18:00:00+00:00

In Microsoft SQL Server, this query won’t work:

SELECT * FROM ( SELECT * FROM Customer UNION SELECT * FROM Supplier) ORDER BY CompanyName

But – if you ask nicely, it does exactly what you’d expect:

SELECT * FROM ( SELECT * FROM Customer UNION SELECT * FROM Supplier) PLEASE ORDER BY CompanyName

You won’t believe the look on your colleague’s faces when you solve their problem using simple good manners.

(Of course, it actually works because PLEASE in that context just acts as a table-name alias for result of the UNION sub-select, and sub-selects in SQL Server need to have a name. But don't let that stop you using it for fun and profit.)

Is doctype.com a License Too Far for Stack Overflow?

2009-10-20T13:27:00+00:00

Short answer:

No, because doctype.com doesn’t use technology licensed from Stack Overflow. Sorry. I got this one completely, completely wrong. D’oh.

Long answer:

This post was originally inspired by doctype.com. I now understand, thanks to an extremely informative comment from one of the doctype.com developers, that doctype.com doesn’t actually run on Stack Exchange. It looks and feels very similar, but is in fact a completely separate codebase built by the guys at doctype.com using Ruby on Rails.

This post is therefore based on completely incorrect assumptions. I’ve struck-out the bits that are actually factually incorrect, although my concerns about fragmenting the user based remain valid – even more so since I discovered that ask.sqlteam.com and ask.sqlservercentral.com are both Stack Exchange sites - but clearly doctype.com has nothing to do with it, and in fact, their platform offers a lot of design-centric tools that Stack Overflow doesn’t.

There’s also this disucussion at meta.stackoverflow.com that addresses a lot of the same concerns.

(Note: In this post, where I say Stack Overflow I’m referring to the website, and where I say StackOverflow LLC, I’m talking about the company behind it.)

I’ve been using stackoverflow.com since it was in beta, and I love it. I ask questions. I answer questions. I hang out and read and comment and vote and generally find the whole thing a hugely rewarding experience. I think it works for two reasons.

First, the technology platform (now available as Stack Exchange – more on this in a moment) is innovative, usable and packed with great ideas.

Second, by actively engaging with people who followed Jeff Atwood and Joel Spolsky’s blogs, they gathered exactly the right audience to breathe life into their product. Stack Overflow launched with a committed, dedicated community of experts already in place. They created a forum where people like Jon Skeet will donate endless hours of their time for nothing more than kudos and badges. (I bet Jon’s employers are wishing they’d thought of that…)

Here’s a few choice quotes from Joel Spolsky’s Inc.com column I’m referring to (my emphasis)

“Between our two blogs, we felt we could generate the critical mass it would take to make the site work.”

“I started a business with the objective of building a big audience, which we would figure out how to monetize later.”

“we promised the audience that the site would always be free and open to the public, and that we would never add flashing punch-the-monkey ads or pop-up windows.”

Now this is the web, where “monetize” usually means “slap advertising all over everything.” – but when Stack Overflow introduced advertising, they were sympathetic and responsive to users’ feedback, and quickly evolved an advertising model that’s elegant, unobtrusive and complements the ethos of the site. The Woot! badge was clever. The tiny Adobe logo on tags like flex and actionscript was really clever – possibly the best use of targeted advertising I’ve seen.

Before long, non-programmers were asking how they could get a slice of the Stack Overflow goodness, and so serverfault.com – for systems admin questions – and superuser.com – for general IT enthusiasts – were born. That clearly worked, so they set up Stack Exchange, to license the platform to third parties, and soon there was moms4mom.com (questions about parenthood), Epic Advice (questions about World of Warcraft), Ask Recipe Labs (cooking and food), Math Overflow (for mathematicians), and various other Stack Exchange sites covering video, photography, car maintenance – all sorts.

A few days ago, I stumbled across doctype.com – a ~~Stack Exchange~~ site for HTML/CSS questions, web design and e-mail design – and some unsettling questions popped into my head.

1. Where am I supposed to ask my jQuery questions now?

I work on everything from T-SQL to a very occasional bit of Photoshop. There is a huge amount of crossover between HTML, CSS, Javascript, AJAX, and web server platforms and their various view/markup engines. Here’s the all-time most popular 20 tags on Stack Overflow, as of 20th October 2009:

1	c#	43,860
2	.net	24,590
3	java	22,924
4	asp.net	20,678
5	php	16,797
6	javascript	16,363
7	c++	15,462
8	python	11,639
9	jquery	11,287
10	sql	10,910
11	iphone	9,686
12	sql-server	9,165
13	html	7,932
14	mysql	7,794
15	asp.net-mvc	6,532
16	windows	6,425
17	wpf	6,370
18	ruby-on-rails	6,095
19	c	6,071
20	css	5,849

The highlighted rows are all Web client technologies – and that ignores all the questions that get tagged as PHP, ASP.NET or Ruby on Rails but actually turn out to involve HTML, CSS or jQuery once the experts have had a look at them. There’s clearly a thriving community of web designers and developers already signed up to Stack Overflow. Should we now all be asking CSS questions on doctype.com instead of using Stack Overflow? I have no idea!

I realize there are HTML / CSS gurus out there who aren’t currently using Stack Overflow because they think it’s just for programmers – but wouldn’t it be better if Stack Overflow was looking at ways to attract that expertise, rather than renting them a walled garden of their own? Getting designers and coders to communicate is hard enough at the best of times, and giving them their own “definitive” knowledge-exchange resources isn’t going to help.

2. What Does This Mean For The Stack Overflow Community?

Shortly after discovering doctype.com, I tweeted something daft about “stackoverflow failed as a business”, which elicited this response from one of the guys at Fog Creek… he’s absolutely right, of course. StackOverflow LLC is clearly doing just fine – their product is being enthusiastically received, and I’m thoroughly looking forward to their DevDays event later this month.

However, I think the success of StackOverflow LLC is potentially coming at a cost to stackoverflow.com – the site and the community that surrounds it – and in that respect, I believe that the single, definitive, free site that they originally launched last year has failed to fulfil its potential as a revenue stream.

The decision to license Stack Exchange to sites who are directly competing for mindshare with Stack Overflow’s “critical mass” worries me, because it suggests that StackOverflow LLC is now calling the shots instead of stackoverflow.com, and making decisions that are financially astute but potentially deleterious to the existing user base.

They are entitled to do this, of course. It’s their site, and I’m extremely grateful that I get to use it for free.

What’s ironic is that the worst case scenario here - for me, for stackoverflow.com, and for the developer community at large - is that doctype.com is wildly successful, becomes the de facto resource for HTML/CSS questions on the internet, generates a healthy revenue stream of its own, and StackOverflow LLC does quite nicely out of the deal. The format is copied by other technology sites, and soon there’s a site for SQL, a site for Java, a site for WinForms, a site for PHP… stackoverflow.com is no longer the definitive resource for programming questions, and we, the users, are back to using Google to trawl a dozen different forum sites looking for answers, and cross-posting our questions to half-a-dozen different sites in the hope that one of them might elicit a response. It’ll be just like 2006 all over again.

OK, So What Would I Have Done Instead?

doctype.com is trying to compete with an established market leader, by licensing that leader’s technology, in a market where the leader has a year’s head start and controls the technology platform. That’s like trying to open a BMW dealership in a town where there’s already a BMW factory outlet, run by two guys everyone knows and loves, whose reputation for service and maintenance is second to none. It has to fail… right? ^[1]

But – I can appreciate what they’re trying to do. I appreciate that StackOverflow LLC is not a charity, and I appreciate why the folks behind doctype.com think there’s a niche for an SO-style site focusing on designers.

The key to Stack Overflow’s success isn’t the catchy domain name, or that fetching orange branding. The key is the information and the people - I see no technical reason why something like doctype.com couldn’t be licensed as a front-end product that’s integrated with the same database and the same user community as Stack Overflow. Modify the back-end code so that users who sign up at doctype.com get certain filters applied. Use a different web address, a different design, maybe just include questions tagged with html, css, jquery and javascript to start with, so new users see content that’s more immediately relevant to their interests - but when they search or ask a question, they’re getting the full benefit of Stack Overflow’s huge community of loyal experts – not to mention the tens of thousands of accepted answers already in the Stack Overflow database.

How about it? doctype.stackoverflow.com, javaguru.stackoverflow.com, aspnetmvc.stackoverflow.com… each a finely-tuned filtered view onto a single, authoritative information resource for programming questions, from assembler up to CSS sprites. That has to be better than the gradual ghettoization and eventual fragmentation of a thriving community, yes?

Stop Press: Someone just pointed me at ask.sqlservercentral.com. That’s – yep, you guessed it – a Stack Exchange site for SQL questions. As if having to choose between stackoverflow.com and serverfault.com wasn’t bad enough. Does anyone else think this is getting a bit silly?

~~[1] Of course, it’s entirely possible that Joel & the gang know this, and are quite happy to take $129 a month off the folks at doctype.com whilst they work this out for themselves...~~

Is There Such A Thing As Test-Driven Maintenance?

2009-10-16T00:21:00+00:00

One of the great strengths of test-driven development is that systems that are built one tiny test at a time tend to be… well, better. Fewer bugs. Cleaner architecture. Better separation of concerns. The characteristics that make code hard to modify are the same characteristics that make it hard to test, so by incorporating testing into your development cycle, you find – and fix – these pain points early, whilst development is cheap, instead of discovering them three months after you’ve shipped and spending the next two years death-marching your way to version 1.1.

However, there’s a flip-side to this. Not a disadvantage of TDD per se, but something that I think is an unavoidable side-effect of placing so much emphasis on TDD as applied to green-field projects. The “test-driven” part and the “development” part are so tightly coupled that it's easy to assume that automated testing was only applicable to new systems.

I can’t be the only one using Moq and NUnit on new projects, whilst the rest of the time grimly hacking away on legacy code, dancing the Alt-Tab-F5 fandango and spending hours manually testing new features before they go live. And I can’t be the only one who thinks this is just not right – after all, the big legacy apps are the ones with the thousands of paying customers; surely if we’re running automated tests on anything, it should be those?

Last week, two things happened. One was a happy afternoon spent setting up TeamCity to build and man age most of our web projects. The other was a botched deployment of an update to a legacy site – the new code worked fine, but a config change deployed as part of the update actually broke half-a-dozen other web apps running on the same host. Broke, as in they disappeared and were replaced by a Yellow Screen of Death, because the root web.config was loading an HttpModule from a new assembly, and the other web apps were picking up the root’s config settings but didn’t have the necessary DLL. Easily fixed, but rather embarrassing.

If It Runs, You Can Test It

This was a stupid mistake on my part, easily avoided, and it suddenly occurred to me, screamingly easy to detect. We may not have any controller methods or IoC containers to enable granular unit tests, but we can certainly make sure that the site is actually up and responding to HTTP requests.

One of the team put together a very quick NUnit project, which just sent an HTTP GET to the default page of each web app, and asserted that it returned a 200 OK and some valid-looking HTML. Suddenly, after five years of tedious and error-prone manual testing, we had green lights that meant our websites were OK. It took another ten minutes or so to add the new tests to TeamCity, and voila – suddenly we’ve got legacy code being automatically pushed to the test server, and then a way of firing HTTP requests at the server and making sure something comes back.

You can do this. You can do this right now. TeamCity is free, Subversion is free, NUnit is free, and it doesn’t matter what your web apps are running. Because the ‘API’ we’re testing against is plain simple HTTP request/response, you can test ASP, ASP.NET, PHP, ColdFusion, Java – even static HTML.

What’s beautiful is that, once the test project’s part of your continuous-integration setup, it becomes really easy to add new tests… and that’s where things start getting interesting. Retro-fitting unit tests to a legacy app is hard, but when you need to modify a piece of the legacy app anyway, to fix a bug or add a feature, it’s not that hard to put together a couple of tests for your new code at the same time. Test-first, or code-first – doesn’t matter; just make sure they make it into the test suite. If you’re coupled to legacy data models and payment services and ASP session variables, you’re probably going to struggle to set up the required preconditions. But, most of the time, you’ll find something you can test automatically, which means it’s one less feature you need to worry about every time you make a change or deploy a build.

We now have 19 tests covering over 50,000 lines of code. Yeah, I know - that’s not a lot. But it’s a start, and the lines that are getting hit are absolutely critical. They’re the lines that initialize the application, verify the database is accessible, make sure the server’s configuration is sane, and make sure our homepage is returning something that starts with <html> and has the word “Welcome” in it – because I figure if we’re going to start somewhere, it might as well be at the beginning.

There Can Be Only One. Or Two. Or Three, but Never Four.

2009-10-14T23:32:00+00:00

A quick but very simple technique to limit the number of instances of a .NET app that will execute at once:

using System;
using System.Linq;
using System.Diagnostics;

namespace ConsoleApplication1 {
  public class Program {
    static void Main(string[] args) {

      var MAX_PERMITTED_INSTANCES = 3;

      var myProcessName = Process.GetCurrentProcess().ProcessName;
      Process[] processes = Process.GetProcesses();
      var howManyOfMe = processes.Where(p => p.ProcessName == myProcessName).Count();
      if (howManyOfMe > MAX_PERMITTED_INSTANCES) {
        Console.WriteLine("Too many instances - exiting!");
      } else {
        Console.WriteLine("I'm process #{0} and I'm good to go!", howManyOfMe);
        /* do heavy lifting here! */
      }
      Console.ReadKey(false);
    }
  }
}

Very handy if – like we do – you’re firing off potentially expensive encoding jobs every few minutes via a scheduled task, and you’re happy for 3-4 of them to be running at any one time – hey, that’s what multicore CPUs are for, right? - but you’d rather not end up with 37 instances of encoder.exe all fighting for your CPU cycles like cats fighting for the last bit of bacon.

I’m still sometimes really, really impressed at how easy stuff like this is in .NET… I thought this would end up being hours of horrible extern/Win32 API calls, but no. It’s that easy. Very nice.

Hey… My World Wide Web Went Weird!

2009-10-13T12:52:00+00:00

About a week ago, my world wide web went weird. There’s no other way to describe it. Well, OK, there’s probably lots of ways to describe it, but I like the alliteration. Anyway – what happened was, lots of websites just suddenly started looking horrible, for no readily apparent reason. Like the “spot the difference” screenshot below.

See how the snapshot on the left looks really rather unpleasant, while the one on the right is nice and crisp and readable?

First time I saw it, I assumed it was some ill-inspired redesign of a single site. Second time, I thought it must be some new design trend. Then I noticed it happening on some of our own sites - including our wiki and our FogBugz install – and since I definitely hadn’t messed around with them, that ruled out the possibility of it being something happening server-side. Some sites were still working and looking just fine, so it probably wasn’t a browser issue… but, thanks to a bit of lucky exploration and the awesome power of Firebug, I just worked out what’s going on.

All the affected sites use the same CSS font specification:

body { font-family: Helvetica, Arial, sans-serif; }

Of course, Helvetica isn’t a standard Windows typeface, so on most Windows PCs, the browser will skip over Helvetica and render the document using Arial instead. Last week, whilst working on something for our editorial team, I installed some additional fonts on my workstation, which includes – you guessed it – the Helvetica typeface shown above.

Arial, and most other Windows fonts like Calibri, Verdana, Trebuchet, use a neat trick called font hinting which ensures that when they’re rendered at small sizes, the shape of the individual glyphs lines up nicely with the pixels of your display – so you get nice, crispy fonts. The particular Helvetica flavour I’d installed obviously doesn’t do this – hence the spidery nastiness in the left-hand screenshot.

I’m guessing either the designers who built most of these sites had a hinted version of Helvetica (possibly ‘cos they’re Mac-based?), or that they just never tested that particular CSS rule on a Windows system with a print-optimised Helvetica typeface installed.

I guess the moral of the story is that if you want to annoy somebody in a really subtle way, install the nastiest Helvetica font you can find on their Windows PC. I’m pretty sure that if I hadn’t stumbled across the solution, sooner or later I’d actually have reinstalled in despair just to get things looking crispy again.

RESTful Routing in ASP.NET MVC 2 Preview 2

2009-10-13T00:09:00+00:00

Microsoft recently released Preview 2 of the next version of their ASP.NET MVC framework. There’s a couple of things in this release that are designed to allow your controls to expose RESTful APIs, and – more interestingly, I think – to let you build your own Web pages and applications on top of the same controllers and routing set-up that provides this RESTful API. In other words, you can build one RESTful API exposing your business logic and domain methods, and then your own UI layer – your views and pages – can be implemented on top of this same API that you’re exposing for developers and third parties.

Thing is… I think they way they’ve implemented it in preview doesn’t really work. Don’t get me wrong; there are some good ideas in there – among them an HTML helper method, Html.HttpMethodOverride, that works with the MVC routing and controller back-end to “simulate” unsupported HTTP verbs on browsers that don’t support them (which was all of them, last time I looked). You write your form code like this:

<form action=”/products/1234” method=”post”>
<%= Html.HttpMethodOverride(HttpVerbs.Delete) %>
<input type=”submit” name=”whatever” value=”Delete This Product” />
</form>

and then in your controller code, you implement a method something like:

[AcceptVerbs(HttpVerbs.Delete)]
public ActionResult Delete(int id) {
/* delete product here */
return(Index());
}

The HTML helper injects a hidden form element called X-HTTP-Method-Override into your POST submission, and then the framework examines that hidden field when deciding whether your request should pass the AcceptVerbs attribute filter on a particular method.

Now, most MVC routing examples – and the default behaviour you get from the Visual Studio MVC file helpers – will give you a bunch of URLs mapped to different controller methods using a {controller}/{action}/{id} convention – so your application will expose URLs that look like this:

/products/view/1234
/products/edit/1234
/products/delete/1234

Since web browsers only support GET and POST, we end up having to express our intentions through the URI like this, and so the URI doesn’t really identify a resource, it identifies the act of doing something to a resource. That’s all very well if you subscribe to the Nathan Lee Chasing His Horse school of nomenclature, but one of the key tenets of REST is that you can apply a different verb to the same resource identifier – i.e. the same URI – in order to perform different operations. Assuming we’re using the product ID as part of our resource identification system, then:

PUT /products/1234 – will create a new product with ID 1234
POST /products/1234 – will update product #1234
GET /products/1234 – will retrieve a representation of product #1234
DELETE /products/1234 – will remove product #1234

One approach would be to map all these URIs to the same controller method – say ProductController.DoTheRightThing(int id) – and then inspect the Request.HttpMethod inside this method to see whether we’re PUTing, POSTing, or what.

This won’t work, though, because Request.HttpMethod hasn’t been through the ‘unsupported verb translator’ that’s included with MVC 2; the Request.HttpMethod will still be “POST” even if the request is a pseudo-DELETE created via the HttpMethodOverride technique shown above.

Now, MVC v1 supports something called route constraints. Stephen Walther has a great post about these; basically they’ll let you say that a certain route only applies to GET requests or POST requests.

routes.MapRoute(
    "Product",
    "Product/Insert",
    new { controller = "Product", action = "Insert"},
    new { httpMethod = new HttpMethodConstraint("POST") }
);

That last line there? That’s the key – you can map a request for /Product/1234 to your controller’s Details() method if the request is a GET request, and map the same URL - /Product/1234 – to your controller’s Update() method if the request is a POST request. Very nice, and very RESTful.

But – yes, you guessed it; it doesn’t work with PUT and DELETE, because it’s still inspecting the untranslated Request.HttpMethod, which will always be GET or POST with today’s browsers.

However, thanks to the ASP.NET MVC’s rich extensibility, it’s actually very simple to add the support we need alongside the features built in to preview 2. (So simple that this started out as a post complaining that MVC2 couldn’t do it, until I realized I could probably implement what was missing in less time than it would take to describe the problem)

You’ll need to brew yourself up one of these:

/// Allows you to define which HTTP verbs are permitted when determining 
/// whether an HTTP request matches a route. This implementation supports both 
/// native HTTP verbs and the X-HTTP-Method-Override hidden element
/// submitted as part of an HTTP POST
public class HttpVerbConstraint : IRouteConstraint {

  private HttpVerbs verbs;

  public HttpVerbConstraint(HttpVerbs routeVerbs) {
    this.verbs = routeVerbs;
  }

  public bool Match(
    HttpContextBase httpContext, 
    Route route, string parameterName, 
    RouteValueDictionary values, 
    RouteDirection routeDirection
  ) {
    switch (httpContext.Request.HttpMethod) {
      case "DELETE":
        return ((verbs & HttpVerbs.Delete) == HttpVerbs.Delete);
      case "PUT":
        return ((verbs & HttpVerbs.Put) == HttpVerbs.Put);
      case "GET":
        return ((verbs & HttpVerbs.Get) == HttpVerbs.Get);
      case "HEAD":
        return ((verbs & HttpVerbs.Head) == HttpVerbs.Head);
      case "POST":

        // First, check whether it's a real post.
        if ((verbs & HttpVerbs.Post) == HttpVerbs.Post) return (true);

        // If not, check for special magic HttpMethodOverride hidden fields.
        switch (httpContext.Request.Form["X-HTTP-Method-Override"]) {
          case "DELETE":
            return ((verbs & HttpVerbs.Delete) == HttpVerbs.Delete);
          case "PUT":
            return ((verbs & HttpVerbs.Put) == HttpVerbs.Put);
        }
        break;
    }
    return (false);
  }
}

This just implements the IRouteConstraint interface (part of MVC) with a Match() method that will check for the hidden form field when deciding whether to treat a POST request as a pseudo-DELETE or pseudo-PUT. Once you’ve added this to your project, you can set up your MVC routes like so:

routes.MapRoute(
  // Route name - anything you like but must be unique.
  "DeleteProduct",				 
  
  // The URL pattern to match
  "Products/{guid}", 
  
  // The controller and method that should handle requests matching this route 
  new { controller = "Products", action = "Delete", id = "" },   
  
  // The HTTP verbs required for a request to match this route.
  new { httpVerbs = new HttpVerbConstraint(HttpVerbs.Delete) }
);

routes.MapRoute(
  "CreateProduct",
  "Products/{id}",
  new { controller = "Products", action = "Create", id = "" },
  new { httpVerbs = new HttpVerbConstraint(HttpVerbs.Put) }
);

routes.MapRoute(
  "DisplayProduct",
  "Products/{id}",
  new { controller = "Products", action = "Details", id = "" },
  new { httpVerbs = new HttpVerbConstraint(HttpVerbs.Get) }
);

and finally, just implement your controller methods something along these lines:

public class ProductsController {
public ViewResult Details(int id) { /* implementation */ }
public ViewResult Create(int id) { /* implementation */ }
public ViewResult Delete(int id) { /* implementation */ }
}

You don’t need the AcceptVerbs attribute at all. I think you’re better off mapping each resource/verb combination to sensibly-named method on your controller, and leaving it at that. Let proper REST clients send requests using whichever verb they like; let normal browsers submit POSTs with hidden X-HTTP-Method-Override fields, trust the routing engine and route constraints to sort that lot out before it hits your controller code, and you’ll find that you can completely decouple your resource identification strategy from your controller/action naming conventions.

BLATANT PLUG: If you’re into this kind of thing, you should come along to Skills Matter in London on November 2nd, where I’ll be talking about the future of web development - HTML 5, MVC 2, REST, jQuery, semantic markup, web standards, and… well, you’ll have to come along and find out. If you’re interested, register here and see you on the 2nd.)

Coordinating Web Development with IIS7, DNS and the Web Deployment Tool

2009-10-11T21:59:00+00:00

DISCLAIMER: There’s some stuff in here which could cause all sorts of chaos if it doesn’t sit happily alongside your individual setup – in particular, hacking your internal DNS records is a really bad idea unless you know what’s already in there, and you understand how DNS resolution works within your organisation. Be careful, and if you’re not responsible for your DNS setup, probably best to discuss this with whoever is responsible for it first.

I’ve been setting up a continuous integration system for our main software products. We host 20+ web sites and applications across four different domain names, ranging from ancient legacy ASP applications based on VBScript and recordsets, to ASP.NET MVC apps built with TDD, Windsor, NHibernate and “alt-net” stack.

Here’s a couple of things we’ve come up with that make the whole process run a little more smoothly. Let’s imagine our developers are Alice, Bob and myself, and we’re running a three-stage deployment process. Here’s how it works.

Alice implements a new feature, working on code on her local workstation. She has a full local copy of every site, under Subversion control, which she can view and test at www.website.com.local
Once the feature’s done, Alice commits her code. TeamCity – the continuous integration server – will pull the change from Subversion, build it, and deploy the results to www.website.com.build
We run tests – both automated and manual – against this build site. If everything looks OK, we send this .build URL to the stakeholders and testers to get their feedback on the new feature.
Once the tests are green and the stakeholders are happy, the feature is ready for launch. We’ll now use msdeploy to push the entire modified website onto the test server - www.website.com.test
We run integration tests that hit www.website.com.test – and also www.website.com.test/some_app, www.website2.co.uk.test, www.another-site.com.test – basically, they verify that not only do the individual apps and sites all work, but that they’re co-existing happily on the same server.
Finally, we have a couple of msdeploy tasks set up in TeamCity, that will deploy the entire server configuration from the test server to the public-facing servers.

Setting up Developer Workstations

Most of our developer machines are running Windows 7, which includes IIS7, which supports multiple websites (this used to be a huge limitation of XP Professional, which would only run a single local website). We have a standard setup across workstations, build, test and live servers – they all have a 100Gb+ D: drive dedicated for web hosting, which means we can use msdeploy.exe to clone the test server onto new workstations (or just to reset your own configuration if things get messed up), and to handle the deployment from build to test to live.

Note that this doesn’t mean we’re hard-coding paths to D:\ drives – the apps and sites will happily run from any location on the server, since they use virtual directories and Server.MapPath() to handle any filesystem access. However, it does make life much easier to set up configuration once, and then clone this config across the various systems.

Finally, note that our workstations are 64-bit and the servers are 32-bit, which works fine with one caveat – you can sync and deploy configuration from the servers to the workstations, but not vice versa. In practise, this is actually quite useful – anything being pushed onto the servers should be getting there via Subversion and TeamCity anyway

Using DNS to manage .local, .build and .test zones

Unless you want to maintain a lot of /etc/hosts files, you’ll need your own local DNS servers for this part – but if your organisation is using Active Directory, you’re sorted because your domain controllers are all local DNS servers anyway. The trick here is to create “fake” locally-accessible DNS zones containing wildcard records. We have a zone called local, which contains a single DNS CNAME record that points * at 127.0.0.1. This means that anything.you.like.local will resolve to 127.0.0.1 – so developers can access their local copies of every site by using something like www.sitename.com.local.

There’s a DNS zone called build, which contains an ALIAS record pointing * at build-server.mydomain.com, and another one called test, which has an ALIAS record pointing * at test-server.mydomain.com. We’ve also set up *.dylan as an alias for my workstation, and *.alice as an alias for Alice’s PC, and *.bob as an alias for Bob’s PC, and so on.

This seems simple but it’ll actually give you some very neat capabilities:

Every developer can see their own projects at www.website.com.local and www.othersite.co.uk.local
Everyone in the organization can see a particular developer’s work in progress, by going to www.website.com.alice or www.othersite.co.uk.bob
Everyone in the organization can see the build server – www.website.com.build – and the test server, www.website.com.test

Of course, this doesn’t work unless there’s a web server on the other end that’s listening for those requests, so our common IIS configuration has the following bindings set up for every site:

This looks like a lot of work to maintain, but because developer workstations are set up by using msdeploy to clone the test server’s configuration, these mappings only need to be created once, manually, on the test server, and they’ll be transferred across along with everything else.

I’d be interested to hear from anyone who’s using a similar setup – or who’s found an alternative approach to the same problem. Leave a comment here or drop me a line – or better still, blog your own set-up and send me a link, and I’ll add it here.

A Neat Trick using Switch in JavaScript

2009-10-11T14:46:00+00:00

You ever see code blocks that look like this?

if (someCondition) {
    doSomeThing();
} else if (someOtherCondition) {
    doSomeOtherThing();
} else if (someThirdCondition) {
    doSomeThirdThing();
} else {
    doUsualThing();
}

Turns out in Javascript - and, I suspect, in other dynamically-typed languages that support switch statements - that you can do this:

switch(true) {
    case someCondition:
        doSomeThing();
        break;
    case someOtherCondition:
        doSomeOtherThing();
        break;
    case someThirdCondition:
        doSomeThirdThing();
        break;
    default:
        doUsualThing();
        break;
    }

Of course, by omitting the break keyword you could wreak all sorts of havoc – but I can think of a couple of scenarios where you want to apply one or more enhancements (e.g. adding one or more CSS classes to a piece of mark-up) and this would fit very nicely.

I love this photograph.

2009-09-26T21:31:00+00:00

Robin Red-breast Looking Quizzical
Originally uploaded by Dylan Beattie

We have an olive tree growing in a huge pot in our garden, which ended up horribly waterlogged and looking rather unhappy, so this afternoon I set about the back-breaking task of digging it out, emptying the (disgusting) waterlogged soil from the bottom of the pot, and generally sorting the whole thing out.

Turns out you get quite a lot of earthworms in 500 litres of waterlogged soil - and whether he was attracted by the worms, or just curious, this little robin showed up in the garden. I ran inside to grab a camera, hoping he'd still be there when I got back - and he was. He stayed around for most of the afternoon, perching on the tree, the washing line, the shovel, chasing spiders around - at one point he was sitting literally two feet away from me, and actually started singing. It was absolutely mesmerising.

The photo is complete luck; he was hopping around so much I didn't have time to frame or set up a shot, so I just snapped him whenever he stopped for a moment. I was thrilled that this one came out so well.

A Better Example of Command-Query Separation

2009-09-17T15:23:00+00:00

Daneel3001 just posted this little quartet on Twitter:

@udidahan decisions. And I guess the command will either succeed, or fail by raising a compensating action (booking couldn't succeed).

@udidahan Yet, there are some cases where the command will not enable the domain, cinema room here, won't have the freedom to make much.

@udidahan You were rightly stating that grid like views of data as having negative effect on understanding intent of user actions.

@udidahan Maybe instead of using the hotel booking analogy for explaining CQS you could have used the cinema seat booking.

- and that reminded me of one of my all-time frustrations with the internet – booking theatre tickets – which, coincidentally, dovetails very neatly with something Udi Dahan was describing last night.

I work in Leicester Square. I walk past a dozen theatres every day, so if I want to see a show, it really doesn’t matter when I go and see it. I can go any time – and yet, every single theatre and ticket website starts the search process by saying “which day do you want to go?” – and then this happens:

Me: Friday.
Computer: Sorry, sold out.
Me: OK, Wednesday.
Computer: Sorry, sold out.
Me: OK, Tuesday.
Computer: Great! Tuesday! We can do that! Now, choose a seating section:
Me: Dress Circle, please.
Computer: Sorry – sold out.

What I really want is a to be able to say “I want to see Mamma Mia. I want to sit anywhere in the front ten rows, anytime between now and Christmas, and I can’t make Sep 23rd or any Saturday” – and, as Udi put it, let the computer do the busy-work. I know that the data is in there. I know that there are algorithms capable of fulfilling that request. Why the hell am I sitting here brute-forcing a solution and whacking the back button like I’m playing Track & Field 2 all over again?

Most current ticket-booking websites will reduce your request to “seats H24, H25, H26 and H27 for Chicago at 19:00 on Saturday 25th” – but that request is just too detailed, and far too prone to failure, and doesn’t reflect AT ALL what I’m actually trying to achieve. Maybe I’m only in town for Saturday night and don’t care what show I see. Maybe I desperately want to see Chicago but I’m prepared to go on any night. Maybe I’d be happy with two pairs of tickets instead of four together. I’d love a website that actually asked me what I wanted on my terms instead of presenting me with a bunch of data and expecting me to do the grunt work.

Command-query separation would appear to be the first step towards this – but it seems that what’s really important here is that your command model reflects the intention of your users, rather than the shape of your data.

AltNet Beers – Command/Query Separation with Udi Dahan

2009-09-17T00:47:00+00:00

Tonight was the twelfth of SerialSeb’s alternative network beers events, and this evening we were lucky enough to have Udi Dahan joining us. Clearly a lot of people in the room were very interested in hearing what Udi had to say because for the first time in history (I think?) the usual altnet musical chairs didn’t happen – the speakers hardly changed for the entire hour. That said, I feel like tonight raised a whole lot of fascinating questions for me, and I’m thoroughly grateful to Udi for taking the time to share his insight, to Seb for organizing the whole thing, and to Tequila\ and Thoughtworks for beer and pizza – thanks!

Command/query separation is a relatively new concept to me, and I’m sure I’ve got the wrong end of the stick here, but I’m going to share my reflections on this evening anyway so I can look back on this in a year or so and laugh at myself. Feel free to laugh now if you’ve already got your head around all this.

Anyway. The underlying principle of CQS seems to be that reading data and changing data are actually fundamentally different operations in any business system, and that trying to use the same architectural style for both of these operations can lead to all sorts of chaos.

It also seems pretty obvious that CQS is a topic with a lot of potential “false peaks”. Maybe you’ve refactored your Customer object to use a ChangeName() method instead of exposing property setters for Forenames and Surname. Maybe you’ve exposed a bunch of denormalized data based on optimised stored procedures for your common query scenarios, and you’re still using a rich domain model to do your inserts and updates. In each case, you probably think you’re doing command/query separation – but there’s more to it than that. Until tonight, I thought CQS just meant having some special methods on your data access code for returning big lists of stuff in a hurry. Now, I’m pretty sure I don’t really know what it is at all.

A couple of great highlights from Udi’s contribution to the discussion tonight:

Users are always working with stale data. The information on their screen could potentially be stale by the time they actually see it. In any multi-user system, people are always making decisions and requests based on stale data. (“This is obvious, it’s physics – you can’t fight it. Well, you can, but you’ll lose”)
Separating queries from commands allows the commands to model the user’s intention more clearly – which in turn allows the software to deal gracefully with conflicts and failures (“Sorry, room 133 is taken – would you like room 155?”), where a more granular system might just throw an exception because the data is no longer valid. (“Booking failed – not in the database!”)
The reason we create domain models is really just that we need somewhere to store all our complex business rules, but it’s easy for elements of business rules to leak into the controllers or presentation layers when we’re manipulating domain objects directly.
The ideal CQS approach is that every business operation involves exactly three things:

Find a domain entity
Execute one method on that entity
Commit the transaction.

With this approach, it’s impossible for any business logic to ‘leak’ into the presentation or controller layers – because they’re not making any decisions. Every business operation, complete with all the validation and processing and rules associated with that operation, has to be exposed as a single entry point to the domain model.
The domain entity that exposes the method will probably behave as an aggregate root for the purpose of that operation – but different entities will act as aggregate roots for different operations. Again, this was a bit of an eye-opener for me; talking about DDD gave me the impression that an aggregate root was a fixture of your business model, not something you could chop and change based on what makes sense for a particular operation.

Finally, an analogy of my own that came to me on the way home, that might help, or might be horribly naive and misguided, but which I rather like and which I’ll share here in the hope of provoking some conversation. ‘Traditional’ domain modelling is like home baking; your data store is a supermarket, where the various products on offer are your objects. They’re all there, laid out for you to search through and count and process. To do anything complicated – like making a soufflé – you need to acquire all the various objects required for that operation, then manipulate and combine them in all sorts of complicated ways to achieve the result you’re after. If anything goes wrong – you forget the butter, or you over-cook the eggs – boom! No soufflé for you. Transaction aborted.

CQS seems far more like eating at a fine restaurant. You don’t choose your meal from an array of component products; instead, you get given a menu – a read-only representation of the domain that’s optimised for rapid retrieval. Based on the information on the menu, you then execute a command – you tell the waiter what you’d like to eat – but the structure of that command expresses your intention far more explicitly than the complex series of interactions involved in doing it yourself. If the data that informed your decision is stale - say they’ve just run out of haddock -the command carries enough context that the waiter can offer you the sea bream instead, or perhaps the mackerel, and the entire dining transaction isn’t abandoned.

I guess the question is, do you want your users to feel like they’re making a soufflé, or dining in a Michelin-starred restaurant?

Determining FluentNH Schema Mappings based on Entity Namespaces

2009-09-14T22:40:00+00:00

I’m setting up some Fluent NHibernate mappings for a rewrite of some of our legacy code, and one of the issues I’ve hit is that we make extensive use of cross-database views and joins – the data supporting our app is split across three separate SQL Server databases (which, thankfully, are all hosted by the same SQL Server instance).

Turns out this is pretty easy to do – Mike Hadlow has a great post here which covers the fundamentals.

I’ve extended this principal a bit, using the Conventions facility provided by Fluent NHibernate, so that you can determine the SQL database for each entity based on your entities’ namespaces, so I have a model that looks (very) roughly like this. Let's imagine that my core web data is in a database called WebStuff, my accounts system is in CashDesk and my CRM database is in DylanCrm. Each mapped entity is declared in a sub-namespace of my common Dylan.Entities namespace, with these sub-namespaces named to reflect the database they’re mapping:

namespace Dylan.Entities.WebStuff {
	public class WebUser {
		public int Id { get; set; }
		public Customer AssociatedCustomer { get; set; }
	}
}

namespace Dylan.Entities.CashDesk {
	public class Invoice {
		public int Id { get; set; }
		public Customer Customer { get; set; }
	}
}

namespace Dylan.Entities.DylanCrm {
	public class Customer {
		public int Id { get; set; }
		public IList Invoices { get; set; }
	}
}

NHibernate will quite happily retrieve and update data across multiple databases, by prepending the schema name to the table names - so you end up running SQL statements like SELECT ... FROM CashDesk.dbo.Invoice WHERE .... If you're only mapping a handful of tables, it's easy to specify the schema for each table/object as in Mike's example - but you can also use FluentNHibernate.Conventions to achieve the same thing.

First off, you'll need to add a new class which implements IClassConvention and modifies the Schema property of each class mapping:

public class SchemaPrefixConvention : IClassConvention {

	private string ExtractDatabaseName(string entityNamespace) {
		return (entityNamespace.Substring(entityNamespace.LastIndexOf('.') + 1));
	}

	public void Apply(IClassInstance instance) {
		instance.Schema(ExtractDatabaseName(instance.EntityType.Namespace) + ".dbo");
	}
}

Once you've done that, you just need to reference this convention when you set up your mappings; if you're using the auto-mapping facility, it looks like this:

mappings.AutoMappings.Add(
	AutoMap
		.AssemblyOf<Invoice>()
		.Where(t => t.Namespace == "Dylan.Entities.CashDesk")
		.Conventions.Add<SchemaPrefixConvention>()
	);

mappings.AutoMappings.Add(
	AutoMap
		.AssemblyOf<Customer>()
		.Where(t => t.Namespace == "Dylan.Entities.DylanCrm")
		.Conventions.Add<SchemaPrefixConvention>()
	);

mappings.AutoMappings.Add(
	AutoMap
		.AssemblyOf<WebUser>()
		.Where(t => t.Namespace == "Dylan.Entities.WebStuff")
		.Conventions.Add<SchemaPrefixConvention>()
	);

Fluent NH will run your Apply() method to each mapped class in each of these three mappings, which means the resulting configuration will qualify each table name with a schema name derived from the mapped class’ namespace – and once that’s in place, you can query, retrieve, update, join and generally hack your objects about at will, and completely ignore the fact that under the hood they're actually being persisted across multiple data stores.

I think that's quite neat.

The Problem with Stack Overflow

2009-08-18T10:53:00+00:00

I love Stack Overflow. I think it’s a fantastic resource, not to mention a beautifully-engineered social community. But sometimes mixing knowledge resources with social networking can be… distracting. Let’s say you’re, I don’t know, trying to choose an open source blog engine to add to a .NET site you’re working on.

It starts like this:

Then this:

Ah-ha… *click* -

Oooh – how exciting! A comment! *click*

Some time later:

Thanks to xkcd for the cartoon, and for many, many more I have loved over the years.

Inspired by the last comment to my accepted answer for this. Where do I even begin?

Also - you have any idea how hard it is to get a screenshot of StackOverflow's squawk bar? Once you've seen it once, it goes away and never comes back. And you can't spoof it by setting up another user account to post a temporary comment on your own thread, because your second user account can't comment until it has reputation... I'm actually really quite impressed now at how hard it would be to game the system.

Alt.Net UK Conf – Unit Testing WinForms UIs

2009-08-02T15:14:00+00:00

A couple of links and the code we hacked together during the session on unit-testing desktop UIs during the Alt.Net UK Conference on Sunday:

Ben Hall’s Blog Post on Project White and automated UI testing
Project White on Codeplex
The “Hello World” sample app we put together during the session is here – you’ll need the Project White download as well, and you’ll need to edit the hard-coded EXE path files.

Tool Time

2009-07-24T22:05:00+00:00

So.. having installed Windows 7 beta twice and the release candidate three times, it feels like I’m turning bare Windows boxes into working developer workstations about once a week at the moment, so here’s the low-down of what I put onto a bare Windows install to get things real nice‘n’kentucky. Partly because it’s nice to share, but more because I’ll need a list to work from when Windows goes gold in October and I end up doing this all over again.

Microsoft Office 2007

Office 2007, because everyone else in the world uses it, and because Exchange calendaring is actually pretty good. I use completely separate accounts for work and personal mail – my work e-mail is all in our Exchange server at work, which means if need be I can share my work mailbox with my co-workers without sharing any personal stuff.

Visual Studio 2008 and SQL Server 2008

VS2008 and SQL2008 are kinda obvious – I can’t really imagine building .NET business apps without them. Unless you’re some sort of C# ninja who codes against MySQL and ProgreSQL libraries using vim.exe, in which case send me a picture – I’ll put you on a T-shirt.

Although I’m not a fan of Resharper, Coderush or any of the other ‘heavyweight’ refactoring add-ins, I do use a couple of little VS2008 utilities.

PowerCommands for Visual Studio 2008 – worth it just to be able to right-click and copy/paste project and assembly references within a solution.
JScript Editor Support for “-vsdoc.js” IntelliSense documentation files – to get magic jQuery intellisense in .ASPX views and controls.
Axialis IconWorkshop Lite for Visual Studio 2008 – a free cut-down version of Axialis wonderful IconWorkshop icon editor, that’ll install & run absolutely free as long as you’re running a fully-licensed version of VS2008 Professional or Standard.

Show Me The Money

As well as the full Microsoft / MSDN licensing bundle, there’s a couple of high-end commercial apps that I absolutely swear by. They’re not open source – you can’t share, modify, hack or fork them – and when there’s so many great free apps around, paying hundreds of pounds for an application can be a bit of a shock, but they’re powerful, flexible, beautifully-crafted tools, and they are worth every single penny.

Red Gate SQL Toolbelt

Red Gate’s database tools are fantastic. Awesomely powerful, intuitive, rock-solid, and polished – if you do anything at all with SQL Server databases, you need these tools. The SQL Toolbelt includes the whole lot for just under a grand (i.e. roughly the same as hiring a decent contract DBA for three days) and once you’ve used them, you’ll never want to build a project without them again.

Axure RP Pro

There are great tools out there for writing code, editing photos, writing documents and creating databases and debugging CSS, but for designing software, Axure RP blows everything else out of the water. It’s expressive, it’s intuitive, and the resulting interactive prototypes show people exactly what you’re planning to deliver - which is great, because you find out what you’ve got wrong after three hours instead of three weeks.

SnagIt

Capture your screen, annotate it, scribble on it, move things around, snip and cut and paste and shuffle and reorganize – SnagIt is intuitive, powerful, and works extremely well. The latest version even supports basic video capture – and if you need more advanced video capture, Camtasia Studio from the same people, TechSmith, is well worth a look.

Beyond Compare

For when Subversion’s built-in diff doesn’t really cut it. Beyond Compare is the best file-compare utility out there, bar none. One little touch I really like – their 30-day trial license only counts days that you actually run the software, so if you only use it once a week, it’ll be a good six months before the trial expires. More software should do this.

The Best Things In Life Are Free

Chrome, Firefox, Safari and Opera

Latest versions of these (plus Internet Explorer, of course) are pretty much essential for testing final release web apps. I have a slightly odd set-up – I use IE for “work browsing” (MSDN, FogBugz, our wiki), I use Firefox for GMail, and I use Chrome for pretty much everything else. On Windows 7, you might want to try the Chrome Channel Changer which will pull updates from Google’s weekly alpha builds instead – might be a bit wobbly but generally works a lot better on Windows 7 than the mainstream build.

Firebug

The only Firefox add-in I ever use; Firebug gives you full, detailed, debug views into your CSS, HTML, HTTP, and Javascript, all at your fingertips. Building web pages without Firebug is like playing the piano in boxing gloves.

7-Zip

The best archiver and archive manager out there, bar none. 32-bit and 64-bit versions; supports every archiving format you can think of, with a decent GUI on the top. Open source. Free. Fast. Bye-bye Winzip. It’s been… emotional.

Digsby

Twitter, Facebook, Linkedin, MSN Messenger, AIM, ICQ and Jabber (Google Talk) in one client. Just watch out for all the heinous bloatware in their installer – don’t say yes or accept anything except the first screen, and you should be fine.

NotePad2, TextPad and A. N. Other Editor…

Notepad2 is fast, free, lightweight, and lovely. You’ll want to download Kai Liu’s installer that replaces Windows notepad.exe.

For a long time I swore by TextPad, which back in the day was a truly impressive editor, but recent releases have felt to me like they’re treading the water a bit; some subtle UI changes between 4.x and 5.x meant it didn’t really feel like the upgrade I’d been waiting for, and since I now do most of my actual coding in Visual Studio, switching text editors isn’t the life-changing transition it would have been a couple of years ago. I’ve been playing around with Notepad++, EditPadPro, UltraEdit, and probably end up installing all of them at some point, but I don’t really have a favourite editor right now.

Cygwin

Cygwin provides a huge collection of Unix command line utilities – sed, grep, bash, tar, that kind of thing – that are just useful to have around. I install Cygwin at C:\Windows\Cygwin\ - which keeps it neatly out of the way - and then add C:\Windows\Cygwin\bin\ to the system path, and then forget about it, because grepping for stuff just works and that’s the whole point.

If you want to use cygwin’s git client, you’ll need to add the optional git and openssh packages, because you’ll need ssh-keygen.exe to set things up, and then git.exe to wrangle your repositories.

SlikSvn and TortoiseSvn

TortoiseSvn is the wonderfully smooth and polished Windows shell extension that gives you right-click version control menus in Windows. It’s not just a great revision-control system; it’s a wonderful example of how you can seamlessly integrate your software into the OS instead of needing lots of clunky great windows and forms all over the place. SlikSvn is the command-line Windows binaries; although Tortoise does 99% of the day-to-day stuff, once in a while it’s useful being able to call svn from batch files and scripts, and that’s where SlikSvn comes in.

NUnit, Moq and TestDriven.NET

I like the simplicity of NUnit; I like Moq’s Linq-driven syntax, and I like the way TestDriven.net gives you all this from a right-click anywhere in your project. I particularly like that once these are all in place, they become the easiest way to run a chunk of experimental code, so your successful experiments often end up as unit tests without even trying. I like that.

Snippet Compiler

Snippet Compiler compiles snippets. It’s like the notepad.exe of .NET IDEs, and it’s wonderful for just hacking together tiny programs to automate ad-hoc tasks or try out an idea.

Paint.NET

It’s free, it’s open-source, it works, and it’s powerful. If you’re used to Photoshop it can take a bit of getting used to, but otherwise it’s a great application to have around.

Managing Loosely-Ordered Collections : Lopsided Fun with NHibernate

2009-06-14T22:34:00+00:00

NHibernate’s One-To-Many mapping allows you to map database one-to-many relationships into various kinds of .NET collections, but I recently hit a bit of a snag regarding the ordering of the collections. For example, consider a simple content management system for a music store’s website. This table:

Id	Name	ParentPageId	Position
1	Home	null	0
2	Products	1	0
3	Guitars	2	0
4	Basses	2	1
5	Keyboards	2	3
6	Drums	2	4
7	Services	1	0
8	Guitar Tuition	7	0
9	Guitar Repairs	7	1

represents a simple page-tree structure that looks like this:

Home
    + Products
        + Guitars
        + Basses
        + Keyboards
        + Drums
    + Services
        + Guitar Tuition
        + Guitar Repairs

The key here is that the order of the elements is controlled by the Position column, so I can change the order of any page’s child pages. NHibernate can cope with this just fine by using an indexed collection – but this only works as long as the Position column is always populated with unique, sequential, non-null values. Null Position values cause it to blow up. It’ll also leave gaps in the list if there’s gaps in your sequence – five records with Position values 1,2,4,5,8 will be mapped into a nine-element list with NULL entries at indices 0,3,6,7, which turns your foreach() loops into little baby minefields and requires liberal use of guard clauses.

That’s not quite what I’m after here. All I want to do is preserve the order of elements if an order has been defined. There’s other apps talking to this database that aren’t using NH, which respect ordering by Position when retrieving records but don’t necessarily use sequential zero-based indices when saving changes. The actual data I’m dealing with is often going to look more like this:

Id	Name	ParentPageId	Position
1	Home	null	0
2	Products	1	0
3	Guitars	2	2
4	Basses	2	null
5	Keyboards	2	5
6	Drums	2	6
7	Services	1	7
8	Guitar Tuition	7	null
9	Guitar Repairs	7	4

All I’m after is that when I retrieve elements, they are ordered by the Position column (using whatever ORDER BY semantics are in use on the database server), and that if I move things around in the list and then save it, the order of my list is preserved when saving. I don’t care about null values – if there’s no explicit positions defined, just stick whatever you’ve got in a list in any old order and give it back to me. Likewise duplicate values are OK, and if there’s missing values, don’t give me NULLs, just skip to the next element.

One possible solution I’ve come up with looks like this. First, define a Position property on the entity. The getter returns the item’s current index in the parent’s children collection; and the setter is private and does nothing (but NHibernate won’t let you leave it out)

public virtual int Position {
    // Getter returns the current index of this element in its parent's children collection.
    get {
        if (this.Parent == null) return (0);
        if (this.Parent.Children == null) return (0);
        return (this.Parent.Children.IndexOf(this));
    }
    // Setter does nothing - order is determined by the "order-by" attribute in NHibernate mappings,
    // but NH requires that the setter exists.
    private set { }
}

Then there’s the actual mapping. We want to map this Position property to a DB column, so when doing insert/update operations, the value is persisted to the Position column, and we want to add an order-by attribute to the NHibernate mapping so that when we retrieve the collection, it comes back in the right order. If you’re using the lovely functional goodness that is Fluent NHibernate, the mapping looks like this:

public class CmsPageMap : ClassMap<CmsPage> {
    public CmsPageMap() {
        Map(page => page.Position);
        HasMany<CmsPage>(p => p.Children)
            .KeyColumnNames.Add("ParentPageId")
            .WithForeignKeyConstraintName("Page_Parent_Children")
            .Inverse()
            .Cascade.AllDeleteOrphan()
            // Using SetAttribute() here because Fluent NHibernate doesn't support order-by yet.
            .SetAttribute("order-by", "Position");

and if you’re mapping it using XML files, you’ll need something like this:

<property name="Position" type="Int32">
<column name="Position" />
</property>
<many-to-one name="Parent" column="ParentPageId" />
<bag name="Children" inverse="true" cascade="all-delete-orphan" order-by="Position">
<key foreign-key="Page_Parent_Children" column="ParentPageId" />
<one-to-many class="NinjaCms.Shared.Model.CmsPage, NinjaCms.Shared" />
</bag>

The asymmetry of having a property that’s an order-by on the way out and a column mapping on the way back is slightly weird, but it works.

What If They Ran The Trains For Free?

2009-06-09T09:57:00+00:00

London is gearing up for a two-day Tube strike – industrial action by London Underground staff means tomorrow morning, three million people who’d normally take the tube will be walking, cycling, packing themselves into crowded buses or overground trains, or just pretending they’re sick and staying at home. Papers say the strike is going to cost £100m in lost productivity. The net is a-twitter with harassed Londoners who want to kick Bob Crow in the bollocks.

Now, I recall once hearing a story, which I cannot now find, so the details are somewhat hazy, but I’ll tell it like I remember it and you can tell me if I’m wrong. It’s about a railway strike. I believe it happened in Japan sometime after the war, but the details aren’t really important. It started out like most industrial disputes – the railway staff wanted something, and the Powers That Be didn’t want to give it to them.

So the staff resorted to industrial action. They didn’t strike, picket, protest, or stay at home. They went to work early, they cleaned the trains, fuelled them, and ran them as usual – but they opened all the barriers, shut off all the ticket machines, and let everybody travel for free.

Imagine if the RMT tried this. There’d be trains full of happy, supportive passengers right now. By Thursday, every commuter, employer and tourist in London would be cheering the drivers, encouraging the staff, showing their support and praising Bob Crow as a public hero. Meanwhile, London Underground are panicking, watching their lost revenue shoot into the tens of millions, and probably hammering out a deal pretty damn sharpish so they can get the ticket barriers switched back on.

Surely that’s a nicer way of making your point than pissing off three million people?

White Noise and Chaos

2009-06-07T14:59:00+00:00

It’s been observed, both in science fact and in science fiction, that the radio transmissions of any sufficiently advanced civilization are indistinguishable from random noise. Think about this for one second. The most efficient way of utilising available bandwidth is to encrypt the stuff that needs to stay secret, and compress the hell out of everything else. End result – every repeating pattern in the signal becomes obliterated, either because it’s been deliberately hidden to prevent the signal being decrypted, or because compression algorithms have exploited – and thus eliminated – the redundancy in the repeating patterns. If E.T. is out there, he’s probably using gzip and PGP, and so even if we could pick up his transmissions they’d look exactly like random noise coming from every other corner of the universe.

Does the same thing apply to software development teams? What happens once you’ve established a solid system for code reuse, a culture of refining and refactoring, implemented continuous integration for builds, testing and deployment, and generally automated or eliminated all the routine, repetitive manual processes you can find? When you reach a point where every predictable aspect of the build and release process has been automated, you’ve effectively guaranteed that the only bits left for the people to do are the creative, unpredictable, subjective, random aspects of the process.

Does that mean that, to an outside observer, the behaviour of any sufficiently advanced software team is indistinguishable from chaos?

(And if is is… what does mean for new hires trying to learn your culture and processes?)

Photo from oooJasonOoo via Flickr, used by permission under Creative Commons. Thanks.

NHibernate / Castle ByteCode provider – are you running the right number of bits?

2009-06-05T16:16:00+00:00

Just had one of those head-scratching moments… checked out a known working project including NHibernate and ActiveRecord DLLs, built the whole thing – which built without errors - and then got this lovely message:

Unable to load type 'NHibernate.ByteCode.Castle.ProxyFactoryFactory, NHibernate.ByteCode.Castle' during configuration of proxy factory class.
Possible causes are:
- The NHibernate.Bytecode provider assembly was not deployed.
- The typeName used to initialize the 'proxyfactory.factory_class' property of the session-factory section is not well formed.

Solution:
Confirm that your deployment folder contains one of the following assemblies:
NHibernate.ByteCode.LinFu.dll
NHibernate.ByteCode.Castle.dll

I’ve seen that message quite a lot recently, but thought we’d well and truly sorted it… well, it turns out there’s another possible cause – you’re building for “Any CPU” on a 64-bit machine and your copy of NHibernate.ByteCode.Castle.dll was built on a colleague’s 32-bit machine. Because this DLL is loaded at runtime there’s no compile-time checking that it’s the right bitness – so it’ll build fine, and then blow up.

Setting VS2008’s platform target to “x86” has solved it in this case – I guess doing a 64-bit build of the relevant DLL would also work.

Just for Fun: The Webcam Teleprompter Shroud Experiment

2009-06-03T23:16:00+00:00

My first experience of video conferencing was on a £10,000 dual ISDN Pentium system back in 1997, and it was horrible – low quality, unresponsive, and this weird problem where the person on the other end never actually looked AT you, they always looked slightly off to one side or above your head. Nowadays, of course, the video and audio quality are massively improved, and even a £300 netbook has a pretty good built-in webcam – but they still haven’t solved the problem of the camera and the screen not being in the same place – so if we’re chatting on Skype, and I’m looking at you, I’m not looking at the camera – so to you, I appear to be looking slightly above your head, or off to one side, or whatever...

TV teleprompters have solved this problem – they use a piece of glass, tilted at 45°, inside a darkened “shroud” ; the camera lens is hidden behind the darkened glass, so the newsreader sees the reflection of the hidden screen, and the camera doesn’t see the screen at all – it sees straight through the darkened glass to where the newsreader is sitting. Yes, this is a bit magic (or maybe it’s just sufficiently advanced?) It’s worth remembering that a sheet of glass which is lit on one side and dark on the other actually behaves like a mirror – notice how you see your reflection when you look out of your house windows after dark, but you never see your reflection when it’s bright outside?

Point is, teleprompters work on some pretty basic optics. I’ve wondered for a long time whether the same principle could solve the video-conferencing problem, so I decided to hack together a prototype to see if it would work… and although the image quality is pretty rough, it does actually work. I was quite surprised. In this screenshot, I’m looking at Skype - NOT at the webcam, which is flipped 90° and pointing straight down from inside the cardboard gizmo on top of the screen – and yet the video feed shows me looking straight at the camera.

Optically, here’s what’s going on. Once Skype’s running, just drag the remote video window so it lines up behind the glass exactly (my webcam has a tiny light next to the lens that shows up in the reflection, which makes this part pretty easy), and then when you look at the person you’re talking to, you’ll actually make eye contact instead of staring over their head or down at their chest.

and here’s the actual contraption itself; the glass is from one of those cheap postcard picture frames, and it’s just held in place with blu-tak inside a home-made cardboard “shroud”. The webcam’s a Logitech Communicator Deluxe, which has a neat flexible base/clamp/grip thing that makes it pretty easy to flip, and the last thing I had to do was tweak the webcam settings in Skype to flip the video 90° vertically.

If anyone else fancies hacking together one of these, drop me a line – I’d be curious to see how well it works when you’ve got eye contact and line of sight on both ends of the video feed.

Speaking at SkillsMatter Open Source .NET Exchange III

2009-05-31T15:21:00+00:00

I’ll be doing a short spot about the Web Platform Installer and msdeploy at the forthcoming SkillsMatter Open Source .NET Exchange evening here in London on July 16th. It’s a kind of “bonsai conference” where we get through six or seven fifteen-minute topics in one evening, giving a sort of whirlwind tour of what’s new and what’s cool in the open source .NET development world.

The Web Platform Installer (below) is a slick GUI installer that’ll download and configure everything you’ll need to get started building and hosting web apps on Microsoft systems – if you’re just getting started with web development, or you want to check out one of the popular open-source frameworks like Subtext, Drupal or DotNetNuke, it’ll set up everything you need in a couple of clicks, including Visual Studio Express, SQL Server express, PHP, and whatever else you’ll need to run your chosen apps and frameworks. And yes, it really is as quick and easy as the screenshot makes it look.

msdeploy is the command-line packaging and deployment engine used internally by the Web Platform Installer, but you can also use msdeploy directly to update, package, publish and manage your own servers. It’ll let you publish updated files, configurations, virtual directories, script handlers – basically your entire web application, set up and ready to run –between servers. We’re using it to publish changes from our staging server to our live servers, and it’s an absolute delight to work with.

We’ve also got Ian Cooper talking about Boo, Scott Cowan talking about the Spark view engine, David Ross on MPI.NET, Gojko talking about acceptance testing with Concordion.NET, Phil Trelford showing off units of measure in F#, and Seb talking about his rapidly-evolving REST framework, OpenRasta. It should be a good evening with plenty of time for food, beer and chat with like-minded developers. The last one was extremely popular so if you want to come along, please register and we’ll see you there.

Room Service Oriented Architecture

2009-05-30T23:19:00+00:00

This is a story about breakfasts.

Well, OK, it’s actually a story about hotels and software architecture. But, like anything worth doing, it starts with two breakfasts. (You all know about second breakfast, right?)

The first was the breakfast I ate on my ninth birthday, cooked by my dad on the banks of the Chobe river in Botswana. We were on a camping trip, at a remote spot in the middle of nowhere – no roads, no phones, no shops, no electricity - so to cook this particular breakfast, my folks had brought eggs, bacon, sausages, bread, butter, cooking oil – you name it, they had it, along with a gas-powered fridge to keep it all fresh, and a big tank of water to wash up afterwards, and a gas stove to cook it on, and matches to light the stove, and countless other necessities that were all written down on a big long list taped to the inside of Dad’s camping trailer. This single breakfast, like every meal we’d eat out out there in the bush, involved weeks of planning, preparation, equipment, checklists – did we have enough food? Would it stay fresh long enough? Had we brought enough gas bottles to run the fridge and still have enough left to cook with? Was it safe to wash up with boiled river-water or did we need to use bottled stuff? When it’s just you and your tent in the middle of nowhere, you have to cover every single detail – and if you pack two weeks worth of bacon and then wake up one day and fancy porridge, you’re screwed. It’s bacon or nothing.

The second breakfast was eaten on the 44th floor of a Manhattan hotel, sometime in 2004. Organising this breakfast involved picking up a phone and saying to Steph in room service "Hey, can you send up some pancakes and coffee?” and then going back to sleep until someone knocked at the door with – you guessed it - pancakes and coffee. Planning involved? Zero. Which was just as well; as I’d discovered the previous night, New York does a pretty good line in cocktails as well, and if I’d needed to plan anything whatsoever that morning, I’d probably have given up and gone hungry.

So, let’s compare the two scenarios. In both cases, someone’s making a request for breakfast, and someone is then responding to that request – ask for breakfast, wait a while, get breakfast. Simple. Thing is, you can clearly see that for my dad, responding to this particular request involves an awful lot of work, whereas Steph probably doesn’t even need to stand up. And the irony is that for all his hard work and planning, the only thing Dad can actually respond with is bacon and eggs, whereas Steph – who probably can’t even cook – can do bacon, eggs, orange juice, pastries, espresso, fresh fruit and even those famous silver-dollar pancakes covered in sugar. And martinis. And probably arrange a shoe-shine, a haircut, a massage and get your tux dry-cleaned while she’s about it.

If you’re only here to hear about breakfast, you can leave now; it’s about to get a bit geeky. Still here? Cool, OK, here we go.

If you’ve built web applications for any length of time, at some point in your career you probably wrote something like an ASP page that connects to a database, reads some files, sends an e-mail or two, updates a database, does some validation, resizes an image, saves it to disk, generates a bunch of HTML and returns it to the client – all in one file. It’s OK, you can admit it; we’re all friends here. For quite a long time, I wrote most web apps this way. At the time it all seemed terribly clever, and if you stuck at it long enough, you normally ended up with something that did the job, paid the bills and kept the client reasonably happy. Problem is, this is normally the client who thinks he has millions of potential customers, but actually the load on his Web 1.0 B2B portal is him, his business partner, and about four guys in Russia who they persuaded to sign up. The wonderful thing about scaling issues is that when you get them in your software, it normally means your business is doing OK. Assuming, of course, you have a business model in the first place…

Anyway. Point is, trying to apply this everything-in-one-page approach in any large-scale app is like using my dad’s catering strategy to make breakfast for 1,200 people. Last time I looked, the only people who set off into the middle of nowhere with fifty tons of bacon and eggs were the Army - and whatever your thoughts on the armed forces, I’m sure you’ll agree that if we’re trying to learn about minimalist efficiency and cost-effectiveness, we’re probably better off looking elsewhere.

So, how does Steph arrange a couple of hundred breakfasts a day using just a phone – and what can we learn from her about designing scalable web apps?

What’s happening is that Steph’s acting as a front controller in a service-oriented architecture. The front controller’s job is to grab the incoming request, call whichever services are necessary to fulfil that request, and tell them what to do with the result.

Steph has a single responsibility – she converts customer requests (imagine Peter from Family Guy ordering breakfast: “yeah, uh, do you have, like… ah… hang on… er… like, pancakes? With syrup? And some tea? No, wait, coffee… no, tea… no coffee. Yeah. Coffee. Did I get pancakes?”) into short, snappy, precise kitchen orders (“Mikey, two pancakes, two coffee for forty-four-oh-seven”). Mikey is the breakfast chef, and he’s just one of the services that Steph can delegate to. Someone orders a martini, she can delegate to Raoul, the bartender service. Someone orders a shoeshine, she can delegate to Chip the shoe-shine guy.

Like Steph, Mikey has a single responsibility. He turns short, snappy kitchen orders into plates full of food, and he does it quickly. He doesn’t buy the ingredients, he doesn’t serve the food – he delegates those responsibilities to other services. Like Tony, the delivery service, who turns up at 4am every day with a truck full of fresh supplies. Or Chet, the bellhop service, who takes the plates, puts them on one of those funny little trolley-tables with some cutlery, and takes them up to 4407 where I’m waiting for my breakfast.

Hotels can teach us a lot about scaling, because they’re good at it – they’ve been in the hotel business a lot longer than we’ve been in the software business, after all.

Make Your Controllers Delegate Heavy Lifting to Services

Think about a typical hotel; there’s probably only four or five points of customer interaction – say the front desk, the concierge, the room service desk, and the bar. There’s probably only one or two people at each desk, but because they’re working as part of a service-oriented architecture, they can fulfil a huge number of incredibly diverse requests, whilst keeping their own responsibilities simple, well-defined and manageable. Think of your controllers like the customer service desks in a hotel – clean, friendly, and completely dependent on supporting services to actually get anything done. That’s how it should be.

Cache Whatever You Can (Within Reason)

For a hotel, this is the early-morning deliveries of food, booze, fresh linen, theater tickets – stuff that you’re probably going to need sometime today, and you don’t want clients sat around waiting whilst you have to run out and fetch it. For web apps, this means setting up local caches of images, persistent data, resource files. But don’t go crazy. Just like hotels don’t order a ten-year supply of lamb chops, you’ll need to consider how much space your cache consumes, and how long it’ll stay fresh before you need to reload it.

Acknowledge Now; Deliver Later

The room-service desk doesn’t stay on the line yakking with you until your pancakes arrive. They tell you it’ll be twenty minutes, and then hang up so they can take the next call – if they didn’t, they’d need fifty room service operators instead of one. You can use asynchronous processing, message queues and Ajax callbacks to achieve the same thing in web applications – when you get a request that’s going to take a while, respond immediately saying it’ll take a while, and let your users get on with something else whilst it’s processing. And be careful with the Refresh button. If I call down to see where my pancakes have got to, it doesn’t mean I want to order more pancakes; I’m just chasing up the first batch.

What If SQL Had Abstract Tables And Composite Types?

2009-05-12T16:18:00+00:00

There are well-documented mismatches between the object-oriented paradigm and the relational database model. Despite the early adopters telling us we should all be using F# running against CouchDB, I’m going to go out on a limb and say that object-oriented languages and relational databases aren’t going away any time soon.

The more I see and learn about ORMs, the more I get this feeling like there’s a real “code vs data” mindset going on. To an object-oriented application developer, the database is a necessarily evil, the DBA is a pain-in-the-ass Luddite who won’t let you write code the way you want to, tables are an inadequate and inferior persistence mechanism for your beautiful object hierarchies, and – well, if someone could wave a magic wand and just make the whole database thing go away, you’d all be very happy. Even in the most wonderfully agile, test-driven greenfield project, there’s still this perception that the more we can abstract the project away from the database, the better off we’ll be. It’s like the Utopian ideal of ORMs is not to embrace the power of the database – they’d rather make the DB completely irrelevant.

I sit on both sides of this particular fence. As a developer, I find persistence as frustrating and time-consuming as everyone else – but as a business stakeholder, I want my data stored in normal, sensibly-named tables that I can get at using SQL-92 queries. Not because that’s what I’m doing now, but because I really have no idea what I’ll be doing with that data in 5-10 years, and I believe the relational model has stood the test of time.

I’ve spent today at Ayende’s NHibernate workshops at the Skillsmatter Progressive .NET event, and he’s shown us all sorts of NHibernate magic, including various strategies for modelling inheritance and class hierarchies with polymorphic associations, sparse tables, and the like. There’s been some discussion of stuff like composite keys, searching, indexes, unique constraints – and it’s pretty clear that when it comes to object-relational mapping, it’s the object folks that are doing all the work, and the relational side of things is really not doing anything much to make their lives easier.

There’s two things that stand out as being particular painful when it comes to object-relational mapping – composite keys and inheritance. Inheritance because it has no analogue in the relational model so you have to somehow “fake” it – which makes no difference to the code, but it can leave the DB in a bit of a mess. Composite keys because they have built-in limitations regarding nested queries (WHERE foo IN (SELECT…)) – and complex equality and value semantics, which means most ORMs strongly advise against using composite keys if you can help it.

(From this point forward I’m making stuff up as a sort of thought experiment. Don’t take any of it too literally :))

What if we added ABSTRACT and EXTENDS keywords to SQL? What if you could do this in your database?

CREATE ABSTRACT TABLE Customer (
ID int identity(1,1) primary key
)

CREATE TABLE Company EXTENDS Customer (
CompanyName varchar(256)
)

CREATE TABLE Person EXTENDS Customer (
Forenames varchar(256),
Surname varchar(256)
)

Here’s the rules:

Tables must be abstract or concrete.
You can’t insert, update or delete records from an abstract table
Both abstract and concrete tables can participate in foreign key relationships
One abstract record must have exactly one concrete record (i.e. you can’t insert a Company whose ID already represents a Person)
Concrete tables behave like the union of their own columns and their abstract base table’s columns

SELECT * FROM Customer

Id

1

2

3

SELECT * FROM Company

Id CompanyName

1 Monkey Butlers Ltd

SELECT * FROM Person

Id Forenames Surname

2 Eddie Van Halen

3 Jack Sparrow

INSERT INTO Person(Id, Forenames, Surname) VALUES(4, “Ayende”, “Rahien”)

-- will work as expected

INSERT INTO Company(Id, Name) VALUES(4, “Rhino Ltd”)

-- will fail with “Distributed key violation” or some such thing, because ID4 is already taken by a Person and so can’t be used for a Company.

You can, e.g. associate Customer with Address (so every address is ‘owned’ by exactly one customer, regardless of whether the customer in question is a person or a company) – but you can also associate Employee with Company directly, so that your data schema enforces the business requirement that a Person cannot have Employees. As a database concept, it doesn’t really add anything – but with a suitably turbocharged ORM, think of what you could do. A simple Customer.ListAll() could return an array of Customer objects – each of which is ACTUALLY a strongly-typed Person or Company. You could do things like Customer.ListAll(typeof(Company)); you could switch on the type of the objects – and all without the compromise of sparse tables or child table inheritance.

Ok, what about composite keys? Imagine we’re booking flights for an airline. There is a real-world business constraint that the same person cannot be on the same flight twice, and composite keys are tailor-made for modelling this sort of “unique combination” scenario:

CREATE TABLE SeatReservation (
   Key COMPOSITE (
      PassengerId int,
      FlightNumber varchar(5)
   ) primary key,
   SeatRow char,
   SeatNumber int
)

CREATE TABLE SpecialMeal (

-- Notice that the “type” of this column is a SeatReservation.Key – i.e. a reference
-- to the composite type defined as Key in the SeatReservation table.
   SeatReservation.Key SeatReservation primary key,
   IsVegetarian bit,
   IsVegan bit,
   IsHalal bit
)

SELECT * FROM SeatReservation WHERE Key.PassengerId = 12

SELECT * FROM SeatReservation WHERE Key IN (SELECT SeatReservationKey FROM SpecialMeal)

Think of this like adding a struct type to SQL, where COMPOSITE defines a collection of columns just as in C# a struct defines a collection of fields. Composites carry field-value semantics for comparison and equality; if all the columns are equal, the struct is equal. Composite literals are defined inline like:

select * from SpecialMeal WHERE SeatReservation.Key = (178189, ‘VS207’)

- using similar syntax to SQL’s familiar INSERT INTO (Column1, Column2) VALUES(Value1, Value2)

This would allow ORMs such as NHibernate to explicitly map a composite key as a struct, benefiting from intrinsic value-object semantics. The resulting database queries could use composite keys in exactly the same way as primary keys – because the database explicitly allows a composite ‘type’ to be used wherever a primitive type is currently supported – and, most importantly, the business meaning of a composite key is explicit in both the database schema and the object model.

Offsite Backup Recommendations for Really, Really Big Files?

2009-05-09T18:55:00+00:00

When I’m not coding and being all nerdy, I play guitar and mess around with digital audio recording. Uncompressed digital audio is stored in the same format as old-fashioned audio CDs – 74 mins of raw audio takes up 740Mb or thereabouts. This means that a good afternoon’s recording can easily create 2-3Gb of files – and they’re not like MP3s or movies that are easy enough to replace; these are often the only copy in the world of that one killer solo or that bass riff that I’ve been practising for weeks and just happened to nail it this afternoon.

Stuff like Carbonite or Mozy is all well and good for documents and holiday snaps, but given I have consumer ADSL (512Kb upstream), is there even any point thinking about offsite backup for my audio projects, or am I better off just copying stuff to a USB hard drive and maybe taking a drive into work every couple of months? What do people do with video / post-processing stuff? What’s your backup strategy when the actual work you’re doing is creating gigabytes of files on a daily basis?

[photo from john_a_ward via flickr – thanks for sharing, John.]

Unsubscribe (aka ‘the Internet Way of Saying “It’s Not You, It’s Me.”’)

2009-05-09T09:42:00+00:00

My personal inbox is full of opt-in marketing… I receive newsletters about hotels, concerts, travel insurance, car insurance, DVDs, cheap flights… seems you can’t buy anything these days without ending up on their list. Spam is one thing, but most of this is stuff I have actually signed up for – albeit unwittingly – over the years. You know the deal - “by purchasing this motor insurance you agree to receive blah blah blah from time to time” and you never get around to opting-out?

So I spent a happy Saturday morning unsubscribing from every piece of legitimate-but-unwanted marketing material in my inbox… and it’s remarkable how diverse something as simple as the unsubscribe experience can get. Here is how unsubscribe should work. I want to click an unsubscribe link. I do not want to log in and update my preferences. I want to click one link. I want to get a page that’s looks like it’s somehow connected to the organisation who are sending me this stuff – even if it’s just a logo at the top of the page. I want a message informing me that my e-mail address has been removed. I see that, I feel all happy, like you care about me, and you care about your brand, and I’ll come back and do business with you again, because I know you’re happy to let me walk away if I want to.

It’s amazing how many unsubscribe links go to a page called unsubscribe.htm – htm! - on unrelated domain (some third-party mailing list provider) with no branding or connection to the company you’re dealing with, and give you some completely impersonal “Your request has been acknowledged” message. Don’t acknowledge my request – take me off the damn list already. “Acknowledging my request” is right up there with my call being important to you, your being an equal opportunities employer, and your product helping weight loss as part of a calorie controlled diet.

Oh – and when I click the same link again, I’d love a message saying “you’re already unsubscribed”. Unsubscribe should not be idempotent. If they really removed you from the list the first time, what exactly are they doing the second time?

PS: Whilst we’re talking usability and logging in and stuff, how many times has this happened to you?

Go to website login form
Enter your e-mail address and password… “sorry, incorrect password”
Click “Forgotten password”…
…and have to enter your e-mail address again?

I just told you my e-mail address… is it too much to ask that you remember it for fifteen seconds?

A Weird Little Quirk with Windows 7, Outlook 2007 and NormalEmail.dotm

2009-04-29T17:50:00+00:00

One of those “get this fix into Google” blog posts… running Outlook 2007 on Windows 7 Beta, whenever i close Outlook I get this error message:

Outlook cannot save or create this file. Make sure that the disk you want to save the file on is not full, write-protected, or damaged. (C:\NormalEmail.dotm)

It’s actually trying to create NormalEmail.dotm under

C:\Users\user.name\AppData\Roaming\Microsoft\Templates\

but for some reason I don’t understand, after installing Office 2007 on Windows 7, there’s a weird 100Kb file called simply “Templates” – no extension - in C:\Users\user.name\AppData\Roaming\Microsoft\. It doesn’t do anything as far as I can tell, but it means Outlook can’t create the Templates\ folder because there’s already a file by that name in the AppData\Roaming\Microsoft\ folder. Deleting this file doesn’t appear to hurt anything, and once it’s gone, Outlook happily shuts down without complaint.

That’s been bugging me for weeks, so I’ll bet you it’s fixed in yesterday’s Office 2007 Service Pack 2, or the Windows 7 Release Candidate that’s due out tomorrow… but if it’s not, then this is how I fixed it :)

Hey, Apple. DVD + iPod - Make It Happen. Here’s How.

2009-04-13T23:52:00+00:00

Two good things happened this Easter weekend. One – I joined the gym. Two – I bought the West Wing complete boxed set. The West Wing is good because it’s intelligent, gripping, well-written, brilliantly acted television. The gym is good because it will probably stop me dying of a heart attack aged 35. If I can work out some way of combining these two awesome positive forces, it’ll probably improve my life in a whole lot of ways.

I don’t currently own a portable video player. Yeah, I know. Luddite freak. It’s ok. Point, stare, laugh if you must. What I do have is a cupboard full of things (ankle weights, rollerblades, squash gear, yoga mats, that kind of thing) acquired through thinking “hey, if I buy that, it’ll make me fit”. Since I’ve now run out of cupboard space, I have a rule that I must actually try these things before I spend any money. So, before running off to Tottenham Court Road to drool over portable video players, I decided I had to try out this idea to see if it worked. My better half has a 5G video iPod that she offered to lend me for this experiment, so I figured all I had to do was import a couple of West Wing episodes into iTunes, sync them onto her iPod, take it to the gym, and see how it goes.

Oh, if only it was that easy. iTunes is blissfully ignorant of DVDs. Presumably because they’d much rather you paid for a second copy of everything from the iTunes video store. I should mention that they do now have a thing called the iTunes Digital Copy – where you get a DVD, and a second disc with a digital copy of the same movie in MP4 format that you can transfer once onto one iTunes computer. Which is frankly completely stupid. There’s already a perfectly good digital copy of the movie on the first disc – that’s what DVD is. Next thing you know, they’ll be trying to sell you three copies of Timecop for $18

I eventually managed to transfer one disc’s worth of shows, via a convoluted process of trial-and-error involving VLC, Handbrake and lots of tedious waiting around to see if this time it would actually work. Long story short. Yes, it’s possible, (and yes, watching West Wing in the gym is frikkin’ awesome). No, it’s not easy. On Mac OS X, Handbrake does a reasonable job. On a PC, it’s even worse because the PC version of Handbrake won’t read DVDs directly. There are many extremely dodgy-looking DVD-to-iPod convertors on Google; there’s also some wonderfully powerful open-source tools around that can read, write and transcode the various formats, but none of them are particularly intuitive. Encoding a DVD file for the iPhone using ffmpeg is apparently as easy as:

ffmpeg -threads 4 -i %1 -r 29.97 -vcodec libx264 -s 480x272 -flags +loop -cmp +chroma -deblockalpha 0 -deblockbeta 0 -crf 24 -bt 256k -refs 1 -coder 0 -me umh -me_range 16 -subq 5 -partitions +parti4x4+parti8x8+partp8x8 -g 250 -keyint_min 25 -level 30 -qmin 10 -qmax 51 -trellis 2 -sc_threshold 40 -i_qfactor 0.71 -acodec libfaac -ab 128k -ar 48000 -ac 2 %~n1.mp4

- see? It’s that easy! Now, I love ffmpeg. For doing weird and wonderful things with video, it’s unsurpassed – but this? This should be easy. I bought a DVD in a store, and I want to watch it on a consumer video playback appliance. I’m not stealing anything, sharing anything, or doing anything even remotely anarchist – I’m just being a good little consumer who spent a lot of money on shiny things and wants them to play nicely together.

Let me say right now that I will give a huge amount of love and cash to the first company who can come up with a nice way of making video as convenient as MP3 is right now.

I can buy a CD in a store, rip it at work, listen to the MP3s on the way home, copy them off my MP3 player onto my media server and listen to them again while I’m making dinner. I can buy MP3s online from pretty much anywhere, play them on anything, and even burn them back onto CD to listen to in the car. (And for the record, I’m quite happy paying more for high-quality MP3s than I do for physical CDs, because you save me the hassle of ripping it myself.)

Well, I want the same with video. I’ve already spent a fortune on DVDs. I’m not going to buy those movies and TV shows again – there is nothing wrong with the quality or content of my existing purchases, and I know it’s technically possible to do what I want, because those catch little ffmpeg incantations like the one above seem to do a pretty good job. And yeah, I know it’s probably not going to happen because of copyright, piracy, the DMCA, and the usual bureaucratic reasons that only really matter to people who don’t have time to circumvent them.

But until that happens - Apple, how about this, just for starters?

Allow each iTunes Music Store account to make a single DRM-encoded copy of a particular DVD. I can download The Dark Knight from the iTunes Music Store, and watch it on my five authorized devices, and neither Apple nor the MPAA nor Warner Bros seem to have a problem with this. They get their money, I get my movie, end of story. So why not just provide a way of doing the same thing, but instead of downloading it from you, I can buy The Dark Knight in HMV and rip it myself? The file’s still encrypted, still DRM-protected, and if nothing else, it’ll save you 8Gb worth of bandwidth.

Apple have proved with iTunes that most consumers don’t actually care about DRM, ideology, freedom or any of that stuff, because they’d rather just play with the shiny things. They have proved this astonishingly well, and created such very lovely shiny things into the bargain, that I’m verging on just ditching any pretence of principled consumerism and buying an iPod Touch. They’re pretty, and they work, and apparently they now play Scrabble as well. Bonus.

But – it looks like if I want to watch West Wing in the gym, I’m resigned to using DVD rippers, ffmpeg.exe and command-line encoding hacks anyway. So I’m thinking I should sod Apple and buy an Archos, since they have bigger drives, bigger screens, Linux under the hood, and no DRM. If I end up spending a couple of hours a week hacking ffmpeg instead of working out, then that’s what I was doing last week anyway. Plus ça change, plus c'est la même chose

Exifacto – a Nicer EXIF Library for .NET

2009-03-21T23:37:00+00:00

I’ve been looking at reading and parsing EXIF data for a project I’m working on; the project itself is still a long way off, but – as is so often the case in software - the EXIF code I’m working on is starting to show some real promise, so I’ve cleaned up the (very basic!) implementation I’ve currently got and put it up on Google Code for the world to play with.

The System.Drawing classes in .NET expose EXIF data as key/value pairs, but it’s about as unfriendly an API as I’ve ever seen. Each EXIF tag is exposed as a numeric ID (which is one of about a hundred magic numbers defined in the EXIF spec), and an array of bytes – it’s up to you to work out which item is which, extract, parse and decode the byte arrays, and make sense of the result. Just to make things interesting, most numeric data in EXIF is stored as ratios – pairs of signed or unsigned ints representing the numerator/denominator of a rational number. Which makes perfect sense if you’re a photographer - always talking in F-stops and hundredths of a second and fractions of an inch – but makes things just a bit harder if you’re trying to get this stuff out into your .NET application.

For example - to get the date a photograph was taken, using .NET code, you need to do this:

DateTime myImageTaken = DateTime.MinValue;

Bitmap bitmap = new Bitmap(@"image001.jpg");
foreach(PropertyItem item in bitmap.PropertyItems) {
   if (item.Id == 0x9003) {
       string exifDate = Encoding.ASCII.GetString(item.Value);
       if (DateTime.TryParseExact(exifDate,
           exifDateFormats,
           CultureInfo.InvariantCulture,
           DateTimeStyles.AllowInnerWhite,
           out myImageTaken)
       ) {
           break;
       }
   }
}
Console.WriteLine("Image taken at: " + myImageTaken.ToString());

With Exifacto, you do this:

ExifData exifData = new ExifData(“image001.jpg”);
Console.WriteLine(“Image taken at: “ + exifData.DateTimePhotoCreated)

Right now, it’s got support for the basic string and DateTime information, and a couple of the lookup properties wrapped in .NET enumerations; next thing is parsing and arithmetic for signed and unsigned rational numbers. The actual implementation isn’t really the point, though - the idea is to get as much descriptive documentation into the Intellisense comments as possible, ending up with a discoverable API that takes a lot of the confusion out of working with EXIF data.

It’s being hosted at http://code.google.com/p/exifacto/ – Subversion only for now, but source code & DLLs to follow once I’ve tied up a few loose ends.

Just One More Reason to Love Red Gate Software

2009-03-05T15:43:00+00:00

Red Gate make really great software – if you work with SQL Server or .NET and haven’t checked out Red Gate’s offerings, I highly recommend them. They also sponsor the UK Alt.Net conferences, they’re frequently voted one of the top ten workplaces in the UK – and I just stumbled across this, on the 11th page of their (legally binding) license agreements:

SCHEDULE 3
BEERWARE
The following Beerware is licensed on a free basis:*

DTS Package Compare
SQL Log Rescue

* subject to a non-binding obligation on the Licensee (or their representative) to buy a beer for
a representative of the Licensor at any trade conference at which the Licensee (or their
representative) and the Licensor's representative are both present, provided, however, that
the Licensee (or their representative) should not attempt to purchase a beer where to do so
would be unlawful in the relevant jurisdiction.

Even their lawyers are awesome. I love Red Gate.

Critical Mass

2009-02-28T00:50:00+00:00

I’m pretty much a control-freak nerd with antisocial tendencies and an aversion to physical exercise. Critical Mass involves joining 200 total strangers on a bicycle ride around London with no plan, no fixed route and no organisers… so I really have no idea why I enjoy it so much. But I do.

It might be that whole snow-day thing, of witnessing a phenomenon that you couldn’t recreate no matter how hard you tried so you really, really try to absorb and remember as much detail as you can. Or it might be the sheer joy of cycling up Whitehall and down Oxford Street without wondering if you’re going to die horribly every time you go near a bus. It’s the last Friday of every month, a whole crowd of cyclists appear under Waterloo Bridge by the BFI and cycle around London. It’s not a protest, it’s not organised, there’s no destination or agenda; it’s just a bunch of people who love cycling and happen to be following one another. Thing is, when the group of cyclists reaches a certain size (the “critical mass” they’re named after), the normal rules of cycling in London are turned upside down, and the traffic has to get out of your way for a change.

The London rides have been happening since 1994, but I only found out about them last year. I’d been following the coverage of the legal dispute between the police and the cyclists about Critical Mass, which was eventually resolved in favour of the cyclists late last November. Two days later, I was in the café at Foyles bookstore on Charing Cross Road and saw the Critical Mass procession going past – two bikes with sound systems, a couple of unicyclists, a penny-farthing or two, and several hundred regular cyclists – and I just thought “yeah, I want to do that!” I went along to the next ride – which was Boxing Day, and absolutely freezing, but really good fun nonetheless – and tonight was my second ride. 15 miles or so, from Waterloo to Kings Cross, Euston, then along Hyde Park into Kensington and back through Knightsbridge.

Every day, I try to ask myself “why will I remember today? What makes today stand out from all the others?” Well, today’s was cycling down Oxford Street, cyclists as far as the eye can see, shoppers applauding and cheering, AC/DC blasting out from the bike next to me… awesome. But don’t take my word for it – grab your bike and join us for the next one, wherever you are.

Blog woes…

2009-02-27T22:58:00+00:00

Windows Live Writer on Windows 7 is a little… unstable. "The “Save” button crashes it. So I was playing around with a couple of other blog clients today… two non-starters and one that helpfully ignored the “post as draft” button. So ignore the post about nulls being null, ‘cos it’s not finished, and bear with me until I get this thing sorted. (Other than the Live Writer problems, Windows 7 is really looking quite nice. Like Vista but quicker and less… opinionated. I like it.)

A Post About Nothing, or, How I Learned To Stop Worrying And Love Nullable

2009-02-19T17:29:00+00:00

Iain Holder has an interesting post about Nullable<bool> over on his blog, about the pitfalls of using nullable boolean values:

What does null mean? It's got to mean something. You're either pregnant or you're not. You can't have a third state. A light switch is either on or off. If a light switch doesn't exist then it's potentially very dangerous. Almost as dangerous as having a nullable bool.

Personally, I like null; I think it’s a wonderfully useful and often-misunderstood concept. Null is not dangerous per se, but used correctly, it can be a very effective warning sign.

Null means we don’t know – or alternatively, that question doesn’t make sense in this context.

To run with Iain’s example for a moment, let’s write a patient tracking system for a hospital; every time a patient arrives, we store their medical details in a database – blood type, next of kin, whether they’re pregnant, that kind of thing – so that the doctors can recommend appropriate treatment. All fields are required – no nulls allowed. It’s nice, simple, saves time, saves lives.

All very well, until an ambulance brings in an unconscious woman who might need X-rays. Is she pregnant? She might be – you don’t know; she came in alone, and she’s unconscious. The clock is ticking… an X-ray might save her life, but if she’s pregnant, it could harm her unborn child. If our patient system makes us choose true/false with no third option, we’re potentially making a very dangerous assumption either way. We need a third option, so when a doctor gets that patient’s chart, they see that we don’t know, and they can make sure they do a pregnancy test before sending the patient for X-rays or administering potentially dangerous medication.

I’m not arguing that NULL is the only solution to this problem. The problem is universal, and nullable fields is just one of many possible solutions. It happens to be a solution that’s natively supported in most databases and platforms, and I personally think the semantics of “null = don’t know” are rather nice. You may disagree – but you still need something to indicate when data in your model is potentially inaccurate or missing.

Yes, null has no place in a perfect model…

If your model (whether it’s an OO domain model or a relational database model) is complete, perfect, accurate and consistent, then you’re laughing. You will never have null values because your model maps perfectly onto the problem domain you’re working in, and you know every detail of every entity in that problem domain. You know every single customer’s date of birth; you have detailed records of the marketing preferences of every person in your database, and your model is so perfectly tuned to your business that there’s no sparse tables, no outer joins -

…but there’s no such thing as a perfect model

A model is an abstraction, and abstractions always leak. The real world isn’t domain-driven, or relational, or object-oriented – these paradigms are just ways of slicing and storing information about the real world that help us solve a problem or do a job. A lot of the time we’re making decisions based on the value of the information, but sometimes we need to make decisions based on whether that information is present or not.

A slightly less contrived real-life example might help. In one of the systems I work on, we have a nullable bit field in our main customer database for e-mail opt-in preferences. We interpret the true/false/null values as follows:

True	This customer has agreed to received marketing information via e-mail
False	This customer has chosen not to receive marketing information via e-mail
Null	We have no record of this customer's preferences.

When we send out e-mail newsletters, we only include the customers whose field is actually true – that’s the point of opt-in marketing, right? Any customer can log in and change their preferences at any time, so the customers who have opted-out (i.e. their value is false), we leave them alone – they’ve said they don’t want to get hassled, so we don’t hassle them, and if they change their mind, they can log in and reactivate their newsletter any time.

So what about null? Well, when a customer logs in, they get a personalized welcome page. If we see that their opt-in field is null, we add a message to this page saying “Hey, we have this e-mail newsletter you can subscribe to - could you take a moment to let us know whether you’re interested?” The point is, as soon as they’ve expressed a preference one way or the other, we’ll stop showing them this message, so a customer should only see this message the first few times they’ve logged in. We end up with better data; they don’t get unwanted e-mail, and everyone’s happy.

The weird swirly picture there turned up in a Google Images search for “null” – I don’t know enough about magnetic topology to have the faintest idea what it is, but it’s quite beautiful, don’t you think? Image © Colin Beveridge

Specs, Bugs & Rock’n’Roll

2009-02-19T01:44:00+00:00

I have probably seen Back to the Future over 100 times. It’s one of my favourite movies, and it’s also astonishing how a movie that I loved when I was seven years old has retained it’s appeal for over twenty years - it’s like it somehow evolved from a time-travel movie into a 50s/80s period time-travel movie without ever really looking dated in between. (That, or I’m just a huge nerd with an 80s obsession who never really grew up.)

Anyway, you know the scene where Marty’s playing at his parents’ high-school dance, and he’s about to leave, and the band ask if he’ll play one more – “c’mon, something that really cooks” – and Marty steps up to the mic, jokes about “this one’s an oldie… at least, it’s an oldie where I come from” – and then he plays a storming version of Johnny B. Goode, and invents rock’n’roll? Well, as a kid watching this movie over and over, that’s one of the things that never, ever made any sense to me. Time travel, fine – I mean, they have a nuclear DeLorean, right? – but the band jamming along, in perfect time, to a song they’d never, ever heard before? Inconceivable!

A couple of years later I took up the guitar, probably inspired in no small part by Marty McFly, and along the way I started picking up odd bits of music theory. I remember watching Back to the Future one day around this time, and noticing something I’d never noticed before… just before Marty starts his whole Johnny B. Goode riff, he says to the band “Ok, guys, this is a blues riff in B; watch me for the changes, and try to keep up, OK?” Right up until that point, I’d been teaching myself guitar, on my own, by playing along to tapes and stuff. I’d never been in a band and never really taken any formal music classes. I had some music theory books, and I knew vaguely what a "blues riff" was, but I’d never stopped to think why that particular chord sequence had a special name. See, when you’re playing guitar by yourself in your bedroom, you don’t need to communicate with other musicians, so a lot of things – sheet music, sight reading, key changes and music theory – just seem like a complete waste of time. Of course, to seasoned pros like the musicians in Marvin Berry’s Starlighters, those two simple bits of information - “blues riff” and “in B” – tell them exactly what’s about to happen – at least, in enough detail that they can join in at the right point, hit the right notes, and work the rest out as they go along.

What’s this got to do with software? Well, blues riff refers to a 12-bar blues chord progression, which is a pattern - in both senses of the word. Trivially, it’s a repeating sequence of things; in this case musical chords. It’s also a pattern in the software/architecture sense - a recurring ‘solution’ that’s implemented in many different contexts. Whilst the implementation details may vary wildly from one implementation to the next, the underlying structure doesn’t. Whether it’s jazz, thrash, funk or fusion, you’ll find twelve-bar patterns cropping up all over the place in contemporary music, and

This is why design patterns are important in software development. Knowing them doesn’t necessarily make you a better programmer. Just as someone can play a wonderful tune without knowing what all the notes are called, it’s quite possible to implement design patterns without being aware that’s what you’re doing. I built some code years ago that used an ‘active record’ design. At the time, I had never worked with design patterns. I had no idea that my “objects-based-on-tables” solution had a name, or that anyone else was doing the same thing. Without that shared vocabulary, though, collaborating on that code was painful. I had to explain that code line-by-line, explain what it does, how it works, how all the pieces fit together. It’s time-consuming, it’s error-prone, and it’s probably really, really boring for the person listening to the explanation.

Imagine Marty McFly turning to the band and saying “OK, guys, this song has this chord here (plays a B chord) for four bars, then two bars of a this one (plays an A chord) – got that? – and then two bars of that first chord again…” and so on for about ten minutes; and the band are working it out and making notes as he goes along, so they can remember what goes where, and by the time he’s done, the crowd’s got bored and lost interest, Lorraine’s gone home with Biff, and George is out in the parking lot crying quietly into a copy of Amazing Stories.

On the other hand, you can learn the patterns. Not because they’ll make you a better coder, but because they’ll transform your ability to communicate with other pattern-literate developers. You’ll learn them, and you’ll practice them at home, on your own little projects, and then when Lead Architect Martin McFowler turns round at the start of your next project and says “OK, this is a domain model with data access via a repository, we’ll manage references via an identity map and use concrete table inheritance to map the subclasses”, you can sit down, work out how to apply those patterns to your particular project, and bam! – you’ve just invented rock’n’roll.

OCTV: Coming Soon to a Screen Near You?

2009-02-19T00:48:00+00:00

Video surveillance is in the news again; the police insisted that an Islington pub landlord install CCTV cameras as a condition of granting his license application – and the Information Commissioner subsequently pointed out that they’re not really allowed to do that, and now the usual suspects are weighing in on both sides of the same old privacy vs. security debate. I personally have two contrasting experiences with video surveillance, that make me wonder whether we’re getting this whole thing completely backwards and missing a great opportunity here.

Several years ago, a friend and I were in a pub in central London, and her phone vanished. One moment it was out on the table in front of us; a few minutes later, it was gone – nowhere to be found. We hadn’t left the table, but at some point either it got knocked onto the floor and someone picked it up, or some light-fingered individual just grabbed it whilst we were looking the other way. This happened in a well-lit bar, full of witnesses - and right underneath a CCTV camera. It didn’t do the slightest bit of good. Nobody saw anything – or if they did, they weren’t telling. The CCTV tapes would have shown clearly exactly what happened, but the bar staff point-blank refused to let us see the footage, citing the usual data protection excuses.¹

As a counter-example, when I commuted from Southampton to Winchester years ago, the drive home took anywhere from 20 minutes to 2 hours, depending on traffic – so I used to check out the BBC’s traffic webcams before leaving. Even a blurry 320x240 picture that’s five minutes old was good enough to see whether there was a tailback on the M3 or not – sure, it meant once in a while I’d sit at work for an extra hour or two waiting for the traffic to clear, but that was way better than spending that time sat in the traffic jam itself.

Personally, I don’t really object to video surveillance in public places. You’re in a public place, your behaviour and actions can directly affect the people around you. If you’re doing something that you’d rather wasn’t captured on videotape, then I’d probably rather you weren’t doing it in the same pub as me. The problem is that in most cases the only people who can review the resulting footage are the CCTV owners and the police - and whether you question their motives or not, they clearly have better things to do. The problem isn’t the TV bit of CCTV – it’s the CC. Closing that circuit creates an imbalance of power and feeds an “us and them” mentality that really doesn’t do anybody any favours.

So what if we turned the whole thing upside down? Instead of CCTV, let’s put up signs saying “This area is covered by open circuit television” – and websites where you, or anyone else, can see the footage? Suddenly, all that information is accessible by somebody who actually cares enough to look at it – i.e. you. Even if we’re just dealing with real-time feeds, you can pull up an OCTV feed and see if any of your friends in the pub yet, or whether it’s raining in Camden right now, or how big the queue is outside the cinema.

What about if we made the archived footage available? 24 hours worth would let you pull up the day’s coverage of the street outside your house – so you can prove that the phone company are lying to you about the engineer who they claim “waited outside for an hour”. You can see who – or what - caused that scratch on the side of your car. A weeks’ worth of archives could show you how many people are out & about on the streets in that neighbourhood where you’re considering buying a house. A year’s worth will show you how many sunny days they’ve had, and how busy it gets when there’s a football game on up the road, and how often the council ‘forget’ to pick up the recycling.

The Metropolitan Police don’t have time to find CCTV evidence for every stolen car or domestic burglary that takes place… because there are too many crimes, too much footage, and not enough surveillance officers. So share it. A police officer can’t justify spending seven hours reviewing footage of a single car theft… but what if it was your car? Would you be prepared to put in those hours? Would your insurance company?

OK, it’s not quite that simple. It’ll make extramarital affairs, gambling problems and skiving off work a bit difficult… sorry about that. On a more serious note, it’ll bring things out into the open that some people would rather stayed hidden… your employer can see where you spend your evenings, your family can see where you really work. Technology’s never going to solve that kind of problem, though - unless by “solve it” you mean “drag it out into the open and mess up everybody’s lives until they learn how to deal with the truth”.

The technology’s not quite there yet, but I think it’s probably pretty close. Solar-powered webcams with built-in GPS and 16Gb of online storage, IPv6 wi-fi connectivity, capturing a HD image every few seconds and uploading the results to some big store in that cloud thingy that all the cool kids are talking about… not cheap (yet), but by no means impossible – and technology just keeps getting cheaper. Our capacity to record and store information is increasing exponentially, and I really don’t think that’s going to stop. When that information ends up locked in vaults and government servers, it’s basically useless – the cost of retrieval and analysis is prohibitive, and short of serious investigations or legal proceedings, that data never comes out again. The internet has demonstrated time and time again that if you share your stuff – images, stories, information – then people will find remarkable, innovative, practical and beautiful things to do with it – things you’d never have imagined. To a government, company or police force, picking valuable data out of literally millions of digital images is an expensive and daunting prospect… but on the web, it’s the kind of thing people do all day, every day, for fun. So what is everyone so afraid of?

1. I discovered recently – far too late to be of any use – that we could have made a formal request under the Data Protection Act, in writing, and paid the £10 administration charge, and received a copy of any footage that we appeared in. Oh well. I know for next time.

SQL Server 2005 Performance Dashboard

2009-02-10T18:10:00+00:00

One of those quick "this helped me, it might help you" posts. I've been trying to run SQL Server 2005 Performance Dashboard on one of our live servers, and hit a couple of snags along the way.

Having installed the dashboard reports and run the setup.sql script included with the download, I tried to view the report on one of our databases and got the following error:

Error: Index (zero based) must be greater than or equal to zero and less than the size of the argument list

Whoa. Bizarre - and rather unhelpful - error message. Clearly this Performance Dashboard thing hasn't been tested at all... and then it occurred to me that I was using SQL2008 client tools to talk to a SQL2005 server, but that the actual catalog (database) was still running in SQL2000 compatibility mode.

Trying exactly the same thing (right-click the database, Reports, performance_dashboard_main) using SQL Server 2005 Management Studio (instead of 2008 - same server, different version of the client tools) produces a far more helpful error message:

Error: Unable to display report because the database has a compatibility level of 80. To view this report, you need to use the Database Properties dialog to change the compatibility level to SQL Server 2005 (90).

So - over to the test server, bring up a test instance of the database, and change the compatibility level to 90. Database and apps still appear to be running fine, but trying to run the performance dashboard now produced the following error:

Error: Difference of two datetime columns caused overflow at runtime.

This time Google had the answer (thanks David) although it's not immediately clear what the solution is referring to. What you need to do is open the Setup.SQL script (which is installed along with the performance dashboard - you'll find it at C:\Program Files\Microsoft SQL Server\90\Tools\PerformanceDashboard\). Find line 276, which says:

sum(convert(bigint, datediff(ms, login_time, getdate()))) - sum(convert(bigint, s.total_elapsed_time)) as idle_connection_time,

and replace this line with

sum(convert(bigint, CAST ( DATEDIFF ( minute, login_time, getdate()) AS BIGINT)*60000 + DATEDIFF ( millisecond, DATEADD ( minute, DATEDIFF ( minute, login_time, getdate() ), login_time ),getdate() ))) - sum(convert(bigint, s.total_elapsed_time)) as idle_connection_time,

As David explains on the linked thread, DATEDIFF is returning an int (here, it's calculating the number of milliseconds between two DATETIME instances) - so any connection that's been active for longer than ~24 days will overflow an INT when you try and convert the connection time to milliseconds.

Anyway, change that line, run setup.sql again, and it works - acres of lovely statistical goodness at my fingertips:

Next step - make sure the SQL 2005 compatibility hasn't broken anything, then modify the compatibility level on the live server, and then I'll be able to get some real live performance stats based on actual web traffic, which should make for interesting reading.

Videos from SkillsMatter Open Source .NET Exchange

2009-02-08T13:36:00+00:00

Reviews and videos from the SkillsMatter Open Source .NET event last month, including my jQuery lecture, are now online at the SkillsMatter site.

(Note to self: Don’t say “um” so much next time.)

Is a Crisp a Value Object?

2009-02-07T13:29:00+00:00

I’m sat in a railway station in the French Alps, waiting for the TGV to Paris, with nothing much to do except eat crisps and watch Oggy and the Cockroaches on French TV. They say good writing draws inspiration from the world around it, so – because Oggy and the Cockroaches defies all rational discussion – let’s talk about crisps. Ed left a comment on my “lazy loading lunchbox” post that got me thinking about the semantics of value objects – what exactly is a value object, and what role would it play in the whole rucksack/lunchbox scenario.

Ruck sack = Aggregate Root. RuckSackRepository is the only mechanism for retrieving the object graph. The CrispBag is an Entity and of course has no repository. The Crisp Bag is just an array / collection of Crisps, which are in turn Value Objects.

On careful reflection, I don't think a crisp is a value object – at least, not in the playground scenario I was dealing with - because I'd argue that every crisp has a distinct identity and lifecycle. Don't believe me? Here, have some crisps. No, it's OK, go on, have the whole bag... <crunch, crunch, crunch> Now that you're happily tucking into your crisps - guess what? Colin licked one of those crisps earlier when you weren't looking. Suddenly the question of "which crisp?" becomes extremely important. Is the contaminated trouser-crisp one of the handful left safely in the bag - or have you already eaten it? If we model crisps as value objects, we have no way of telling. We can tell how many crisps are left, sure, but we can’t distinguish between them, and we can’t model any operation that would affect the state of a single crisp whilst leaving the other crisps unaffected.

OK, so if a crisp isn’t a value object, then what is? I think that’s a much harder question to address, because as with so many patterns, the answer depends on the scenario you’re modelling.

The value object pattern is typically used to model quantities, descriptions and measurements, where two objects can be considered equal if they have the same value (state). Two classic examples are dates and money. In most domain models, “January 2nd, 1978” will always refer to exactly the same thing – a particular 24-hour period sometime in the late 1970s – so you can model your date/time fields as value objects (remembering to include time zone information if necessary). If you and I are both born on 2nd January 1978, then it’s valid to assume we have the same birthday, because that date can only refer to one thing. (By way of comparison, if your father and my father are both called Dave Beattie, we’re not necessarily related, because there could be any number of people with that name in our domain model)

Money is interesting, because money is a value object by law in most economies - that's how it's treated in business and in court. In legal terms, this is known as fungibility. Point is, when we ask “does this credit equal that debt?”, we’re only interested in the amount. You can repay a £10 debt with any ten pounds as long as the numbers add up; you don't need to track the specific ten-pound-note that was originally borrowed. When you deposit £50 at the bank, you can't go back a week later and ask if that particular fifty pounds is still there - as soon as you deposit it, it becomes part of your account balance and ceases to exist as £50 in its own right.

Remember, though, that value objects are just one modelling pattern, and you need to be sure that you’ve chosen the right pattern for your particular scenario. If you're writing a system for the CIA to track marked dollar bills, you'll need to model those bills as entities so you can track individual bills (e.g. using their serial number). Point is, when you do this, you're not treating them as currency, you're treating them as artifacts. The same would apply to an antiques house dealing in rare coins - it's not the face value of the coins that matters, it's their age, rarity and historical significance that's important. If you’re writing a game where the player can travel through an infinite number of parallel universes, you might need to track an infinite number of different instances of “January 12th, 1978”.

If it’s not clear how to model a particular element in your model, try asking “which one?” If the question makes sense within your own scenario:

Me: “Colin licked one of those crisps!”
You : “Which one?”

Me: “Hey, says here this house was once owned by Chris Columbus!”
You : “Which one?”

- then you’re probably dealing with entities. If the question “which one” is meaningless in the context of your domain:

Me: “I only paid £1 in income tax last year!”
You: “Oh yeah? Which one?”

Me: “I was born on January 12th, 1978”
You: “Really? Which one?”

- then you’re probably better off modelling the subject of the question as a value object.

jQuery at WebTech Exchange 2009

2009-01-23T00:30:00+00:00

So, just got back from doing my jQuery bit at the SkillsMatter Open Source .NET Exchange, which was great fun. Thanks to Gojko and SkillsMatter for putting the whole thing together, to the other speakers for their excellent sessions, and to everyone who came up to me afterwards with feedback and questions; it’s really inspiring to see so many people giving up their own time to learn about writing better software.

I’ve been asked to do a full session at the WebTech Exchange conference in May; it’ll be talking about jQuery in one form or another, but I thought I’d throw the exact topic open to see if there’s anything people particularly want to see. A couple of ideas I’m already playing around with:

A detailed analysis of how jQuery actually works. How have they actually implemented the selectors, live events, event handler syntax, fluent method chaining and that all-powerful jQuery object? (which, by the way, is a functional object that behaves like an array, in case you were wondering…)
Implementing a jQuery plug end-to-end – starting with the basics of jQuery’s plugin architecture and actually implementing a full jQuery plug-in module from scratch during the session.
A “top 10 plug-ins” – showcasing some of the third-party modules that really show off the extensibility and power of the jQuery framework, and how you can mix' and match modules within your own pages to deliver amazingly rich UI experiences with very little code.
What’s new in jQuery 1.3 – the new Sizzle selector engine, live events (which allow you to attach handlers to elements that don’t exist yet), the new .support API, and so on.

Any other ideas you think might be worth a look – either leave a comment on here, e-mail me on dylan@dylanbeattie.net, or you can find me on twitter as @dylanbeattie – and if you’re at all interested in progressive .NET and web technology, I really recommend you check out the conference programme. Hamilton “Hammett” Verissimo de Oliviera, project leader on the Castle Project, will be delivering the keynote and talking about plans for Castle 2.0, and there’s great sessions lined up on Prototype, Dojo, cloud computing. BBC iPlayer, REST and OpenRasta, and a whole lot more. Oh, and there’s an early-bird discount if you register before 28th February.

jQuery at SkillsMatter Open Source .NET Exchange

2009-01-22T01:25:00+00:00

Here's the slides, sample code and links from my jQuery talk at the SkillsMatter Open Source .NET Exchange here in London on 22nd January.

Downloads

Slides : Powerpoint 2007, PDF, PNG
Code: jquery_examples.zip - requires Visual Studio 2008 and the ASP.NET MVC Beta

Links

jQuery is available from www.jquery.com

jQuery Intellisense in Visual Studio 2008 requires Service Pack 1 for Visual Studio 2008 and VS2008 Patch KB958502, which causes Visual Studio to check for the "-vsdoc.js" annotated Javascript file. More info on these is available from ScottGu's blog.

You'll find up-to-date -vsdoc.js files, including support for jQuery 1.3, on James Hart's blog.

Since ASP.NET MVC is still in beta, your best bet for working code and examples integrating it with other libraries like jQuery is probably Stack Overflow, which has a fairly active and enthusiastic ASP.NET MVC contingent.

The Story of the Lazy-Loading Lunchbox

2009-01-14T01:03:00+00:00

Many years ago when I was a lot smaller and still thought 20 GOTO 10 was pretty cool, my long-suffering mother would pack me off to school every day, with a lovingly-packed rucksack containing my homework, gym kit, packed lunch, pencil case and various other educational essentials. At lunchtime, all the packed-lunch kids would go outside and play at being spies and Thundercats until the sandwich-bell rang, which meant it was time to go inside and eat. We'd all troop into the dinner-hall and get out our plastic MASK and He-Man lunchboxes and unwrap our squashed-looking sandwiches and warm bananas and little plastic Thermos flasks of Ribena. Ham sandwiches, jam sandwiches, or - for special occasions - ham and jam, all the better for spending three hours in a warm airtight plastic box alongside a banana.

Once in a while, something would liven up the lunch break. I'll never forget what happened when Jeremy's parents went on holiday and left the babysitter in charge. Next day, Jeremy opened his lunchbox and instead of sandwiches, there was a pizza menu, and a £10 note in there. He had to ask for special permission to use the phone in the office... good thing Chris had an older sister who worked at Speedy Pizza otherwise I don't think they'd have believed us. Next day the headmistress shouted at everyone in assembly, and there were letters sent home to parents. (My dad laughed about it all weekend.)

Then Jim joined our class. I happened to be lunch monitor on his first day, which meant I got to hand out all the lunchboxes when the sandwich bell rang. As I lined them up to hand out, I realized that Jim's was empty. Light as a feather, no noise when you shook it - clearly there was nothing inside. Poor kid, I thought, forgetting his packed lunch on his very first day! I figured I could give him half of my sandwich, and if Chris had bacon crisps he'd probably give them to Jim 'cos Chris hated bacon crisps... it was OK, I told myself. We'd work something out.

Then Jim sat down with his empty lunchbox... and before we knew it, he was tucking into a full three-course roast lunch, piping hot - with gravy, and a glass of milk, and ice-cream for afters. We sat around with envious eyes, eating our banana-scented ham sandwiches, aware that something funny was clearly going on ("why doesn't it melt?") Next day - same thing. On Fridays, he'd get roast pork and dumplings (his favourite, apparently). We asked Jim one day how he did this, but he didn't know. "My mum always does my lunchbox", he said, between mouthfuls of potatoes and gravy, and so the mystery went on. Finally, just before the end of term, Jim asked if I wanted to go over and play after school one afternoon. I jumped at the chance - finally, I could ask his mum about those amazing packed lunches.

The long-awaited day came, and after Jim and I were all done playing Thundercats vs. Ghostbusters in the garden, Jim's mum called us in for tea. We sat around their big dining table and we ate raspberry jelly and ice-cream, and eventually I plucked up the courage to ask about the lunches.

"Mrs. Fowler", I said politely (for that was her name), "me'n'the other children were wondering about Jim's packed lunch, please... how come the ice-cream doesn't melt from being next to the roast potatoes all day? An' how does Jim get all that food into such a tiny box?"

"Oh, that's easy", she said. "See, we bought Jim a lazy-loaded lunchbox. He's a growing lad, and he needs plenty to eat, but I think carrying all that food around all day would give him a bad back. Besides, most days he's off out the door before I've even put the potatoes in the oven - and if he kept turning up late because I didn't have his packed lunch ready for him, I'm sure his teacher would complain. So I get his lunch ready for him and keep it warm in the oven, and then when it's time to eat it, I just pop it into his lunchbox - easy peasy! Now, who'd like more ice-cream?"

I went home even more confused than when I arrived. She must be winding me up, I thought - that's impossible - but Jim's amazing lunches just kept on going. One by one, we all tried to get OUR mothers to do us roast-beef-and-ice-cream packed lunches, but it never worked - from the day Pete's bag dripped gravy and raspberry ripple all over the nature corner, to the day Rob's mum got in trouble for trying to break into the school kitchen, none of us ever got a three-course roast packed lunch except Jim.

Hang on... you're making this up!

OK, ok, busted. It's not a true story. Except the bit about Thundercats. And Chris getting gravy in the nature corner, come to think of it. But there's a couple of details in here that might help explain a few things if you're trying to get your head round OO architecture and enterprise design patterns.

Separation of Concerns - aka "Why is a Eight-Year-Old Ordering Pizza?"

The various parts of an object-oriented application should have their own distinct responsibilities, and should depend on each other for help when they need to do something outside their immediate remit. Web pages shouldn't send e-mail. Database queries shouldn't return HTML. When objects end up doing things they're not supposed to, that's a warning sign - just like an eight-year-old ordering a pizza.

If you're relying on a eight-year-old to arrange their own lunch, you're not separating your concerns properly. To make lunch, we need to go shopping, handle money - and get the Penguin bars from their secret hiding place on top of the fridge. Kids don't know where the food comes from - and they shouldn't have to; if they were in charge of this, it would lead to all sorts of chaos.

Kids have clearly defined responsibilities. They torment the guinea pig, throw up during maths, eat lunch, play Thundercats, have nap time and go home. They shouldn't be shopping, they shouldn't be cooking, and they get really funny looks if they tell the teacher they're just popping out for falafel and a skinny latté. (Incidentally, the the teachers, Mr. Namespace and Mrs. Interface, understand all this, and the reason they don't let kids go out for falafel is that they're enforcing this separation of concerns.)

Aggregates and Repositories - aka "Mum, Did You Pack My Gym Kit?"

That rucksack that my mum would pack for me before sending me off for a day in the big wide world was an aggregate - a top-level container, containing a whole collection of objects that make sense when you put them inside each other, as in the following example:

When I was growing up, kids didn't carry money, and they wouldn't let us out of school to go to the shops. I had no way of getting another pencil or a bag of crisps if I realized I'd forgotten it halfway through the day. I was completely dependent on mum making sure all of these things were in my bag to start with.

This is one of the distinguishing features of the repository, as opposed to just cherry-picking bits of data using ad-hoc queries. When using a repository, you are retrieving deep object graphs. Once I had "retrieved" my rucksack from the repository, I could assume that I also had all the rucksack contents - the lunchbox, the pencil tin, the curiosities and the yo-yo. (The tantrums I'd throw when I got to school and my yo-yo was missing were really just an early form of NullReferenceException.)

So, assuming our repository pattern is returning nicely-populated object graphs, we now have a whole different set of problems.

Memory consumption. I had to carry everything, all day - and there was only so much space in the locker room at school. The day everyone brought their skateboards in, there wasn't enough space for them all.
Long-running queries. If I'd lost my gym shoes or we'd forgotten to defrost any bread to make sandwiches, I'd always end up being late for school. Even though I didn't need the gym shoes until after morning break and the bread until lunchtime, I couldn't leave home without them. Some days it would rain and games would be cancelled and I wouldn't even need the gym shoes in the end - I'd just carry them around all day for no reason.
Stale objects. By the time I actually got around to eating lunch, it had been knocking around in my bag for a couple of hours and invariably smelled of bananas.

Obviously Jim's lazy-loading lunchbox - and presumably the lazy-loading gym-bag, the lazy-loading pencil-case and the lazy-loading gym kit - could solve all these problems, but first let's look at another approach.

Value Holders - aka "Forget lunch, just give 'em a pizza menu"

The day Jeremy's parents were away and the babysitter sent him to school with a pizza menu - that's what PoEAA calls a Value Holder. That particular lunchbox didn't contain any lunch, but it did contain a very simple method that Jeremy could use to get lunch when he needed it. This works, sure - but asking Jeremy to order pizza and pay for it is violating our separation of concerns. Sometimes, this is OK - but be aware you're doing it. You'd should have a damn good reason for letting a eight-year-old order pizza. I suspect Jeremy's babysitter has no such reason; maybe OO principles aren't required reading at babysitter school any more.

In OO, this will typically show up as a method like Customer.GetOrders() - the customer doesn't actually contain the orders, but it knows how to get them. This obviously requires that GetOrders() knows something about the data store (repository, data access layer, whatever) - so our object is not persistence ignorant.

Virtual Proxy - aka "The Magic Lazy-Loading Lunchbox"

Jim's magic lunchbox is something much smarter, though. Jim's lunchbox is a virtual proxy - a lightweight, portable "pretend lunchbox" that, as long as we don't open it, looks and behaves just like a regular lunchbox. Jim's lunch isn't in there. It's at home, keeping warm (or cold, ), but - and this is the kicker - the virtual proxy holds a reference back to it. Jim can't go home and get his lunch - because lunch-retrieval is not his responsibility - but the magic lunchbox can not only fetch lunches on demand, it can do so completely seamlessly, without Jim being aware that it's even happening. All Jim knows is that every day his mum gives him a rucksack, and when he takes his lunchbox out at lunchtime and opens it, he gets a three-course dinner.

This is much harder to implement seamlessly, but once it's working, it's an absolutely joy to use. Just fire up Customer, get back an apparently fully-populated object, and then do something with Customer.Orders and watch as the database quietly retrieves the exact records you need - no more, no less - and seamlessly weaves the new data into your existing object graph. Almost as cool as watching someone get a three-course roast lunch out of a plastic lunchbox.

Finally... Eager Loading - aka "Colin"

Colin carried a huge sports bag with him, everywhere he ever went, with half-a-dozen paperback books, drinks, snacks, sweets, a Spiderman outfit, poster paints, his father's old harmonica... the kid was like a walking junkyard. He'd carry absolutely everything he could think of with him, all the time, just in case he needed it.

Colin's a great example of what we call eager loading - where you actually load more than you need at initialization time, because (for whatever reason) it's going to be really, really hard to get it later on. If you've ever run Google Reader in offline mode, where it uses the Google Gears engine to get all your blog posts now so you can read them later on the train - that's eager loading. As with all these techniques, it's worth knowing that it exists (so you know when it might help you), and what it's called (so you know what to Google when you need to use it for real).

Two things I'd love to see in my next mobile phone

2009-01-03T09:55:00+00:00

When the phone's first switched on, it's locked. When the phone is locked, it display's the owner's name and number and - here's the twist - it can call certain predefined numbers. So if I drop it in the street and you pick it up, you can use it to call my house and tell me you've found it - but you can't use it for anything else.
I want to be able to call my own phone (e.g. from a landline), enter a PIN, and switch it to Loud Ringing remotely - so next time I lose it somewhere around my house whilst it's on silent, I can reactivate the ringer from another phone.

Both are basic software features you can probably implement in any modern handset - they don't use any new hardware or infrastructure. I guess since they make it easier to find missing phones, they make it less likely you'll end up buying another one, though...

Mocking the QueryString collection in ASP.NET

2008-12-17T13:47:00+00:00

One of the hardest parts of building testable web applications using ASP.NET is the HttpContext object, which encapsulates access to the HTTP request and response, server state like the Session and Application objects, and ASP.NET's implementation of various other bits of the HTTP specification.

HttpContext has a God complex. It's all-seeing, all-knowing, ever-present, and most WebForms apps just call HttpContext.Current and work with whatever comes back. This approach really doesn't lend itself to test-driven designs, though, so the ASP.NET MVC team have implemented a collection of virtual base classes - HttpContextBase, HttpRequestBase, etc. - which gives us the ability to isolate elements of the HttpContext for testing purposes, either using a mocking framework or by writing our own test classes that inherit from those base classes. On the whole, this approach works fairly well - especially once you start explicitly passing an HttpContextBase into your controllers instead of letting them run amok with HttpContext.Current - but there's still some legacy implementation details inherited from ASP.NET that can cause a bit of confusion with your isolation tests.

In ASP.NET - both MVC and WebForms - the QueryString property of the HttpContext.Request claims to be a NameValueCollection. It isn't - which becomes immediately apparent if you're trying to test a controller method that handles IIS 404 errors. In classic mode, IIS will invoke a custom error handler as follows. Let's say you've mapped 404 errors to /MyMvcApp/Error/NotFound - where MyMvcApp is a virtual directory containing an ASP.NET MVC application, which contains an ErrorController with a NotFound() method.

When your browser requests http://myserver/page/is/not/here.aspx; IIS doesn't find anything, so it invokes your configured handler by effectively requesting the following URL:

http://myserver/MyMvcApp/Error/NotFound?404;http://myserver:80 /page/is/not/here.aspx

Notice that there's no key/value pairs in that query string. The code in my controller that parses it is using HttpContext.Request.QueryString.ToString() to extract the raw query string - but here's where it gets a bit weird. The framework claims that Request.QueryString is a NameValueCollection, but at runtime, it's actually a System.Web.HttpValueCollection. The difference is significant because HttpValueCollection.ToString() returns the URL-encoded raw query string, but NameValueCollection.ToString() returns the default Object.ToString() result - in this case "System.Collections.Specialized.NameValueCollection" - which really isn't much use to our URL parsing code.

So - to test our parsing code, we need our mock to return an HttpValueCollection. Problem is - this class is internal, so we can't see it or create new instances of it. The trick is to use System.Web.HttpUtility.ParseQueryString(), which will take the raw query string and return something that claims to be a NameValueCollection but is actually an HttpValueCollection. Pass in the URL you need to test, and it'll give you back a querystring object you can pass into your tests.

Putting it all together gives us something along these lines - this is using NUnit and Moq, but the query string technique should work with any test framework.

[Test]
public void Verify_Page_Is_Parsed_Correctly_From_IIS_Error_String() {

	// Here, we inject a test query string similar to that created
	// by the IIS custom error handling system.
	var iisQueryString = "404;http://myserver:80/i/like/chutney.html";
	var testQueryString = HttpUtility.ParseQueryString(iisQueryString);

	Mock<HttpRequestBase> request = new Mock<HttpRequestBase>();
	request.ExpectGet(req => req.QueryString).Returns(testQueryString);

	Mock<HttpContextBase> context = new Mock<HttpContextBase>();
	context.Expect(ctx => ctx.Request).Returns(request.Object);

	// Note that we're injecting an HttpContextBase into ErrorController
	// In the real app, this dependency is resolved using Castle Windsor.
	ErrorController controller = new ErrorController(context.Object);

	ActionResult result = controller.NotFound();

	// TODO: inspect ActionResult to check it's looked up the requested page
	// or whatever other behaviour we're expecting.
}

Open Source .NET Exchange

2008-12-15T23:20:00+00:00

I’ll be presenting a short session on jQuery at the SkillsMatter Open Source .NET Exchange here in London on January 22nd.

If you’re a .NET developer of any kind, you’ve probably seen or heard people talking about stuff like web UI frameworks, object-relational mapping, fluent APIs, asynchronous messaging, aspect-oriented programming – and you might well be wondering what they are, and why they’re relevant. These events are designed as a sort of “tasting menu” of open source frameworks and techniques – six fifteen-minute sessions that’ll give you some idea of what these technologies can do, why you might want to consider using them, and where you can find more information if you’re interested.

In the jQuery session, I’ll be showing you how jQuery’s CSS-based selector syntax and flexible “chaining” API let you add rich, cross-browser behaviour and effects to your web pages. I’ll demonstrate how to add animation, dynamic content and AJAX callbacks to your web pages, and hopefully include a few examples from the multitude of freely-available plug-ins and libraries built on top of the jQuery framework. Yes, all that in fifteen minutes. Like I said, jQuery makes things easy.

You can see the full programme at SkillsMatter’s site. I’m really pleased that I’m speaking first, because it means I get to relax and listen to the rest of the speakers afterwards – in particular, Mike Hadlow’s session on the repository pattern. Mike’s code (and help!) were invaluable on one of my projects earlier this year, and in particular his Linq-to-SQL repository – part of the Suteki Shop project - was a great example of how this pattern can make your code cleaner and your life easier.

If any or all of this sounds interesting (or if you just fancy an evening of beer, pizza and geek chat) then please sign up and come along - especially if you’ve not come along to an event like this before.

Stuff You'll Wish You'd Known When You Switched To 64-bit Windows

2008-12-12T11:30:00+00:00

64-bit Windows is great. I've been running on XP 64 and Vista 64 for about a year now. My extremely old Canon USB scanner isn't supported, and I had to wait a long time for 64-bit drivers for my Line6 Pod XT, but otherwise everything works very nicely - and for running virtual PCs, the extra memory is really worth it.

That said, there's a couple of underlying differences that result in some very odd behaviour in day-to-day usage. It's important to realize that 32-bit and 64-bit Windows subsystems exist as parallel but separate environments. Confusingly, the 64-bit tools and utilities live in C:\Windows\System32, and their 32-bit counterparts live in C:\Windows\SysWOW64. 32-bit processes on x64 Windows are actually running inside a virtual environment, which redirects requests for underlying system resources for 32-bit processes.

64-bit Windows ships with 32-bit and 64-bit versions of lots of common applications - including Internet Explorer and the Windows Script Host. Check out the links below for some more detailed discussion of the architecture and reasoning behind this.

Internet Explorer

You'll find Internet Explorer and Internet Explorer (64 bit) in your Start menu. The 64-bit version can't see any 32-bit components - so no Flash player, no Java, no plugins, no ActiveX controls, nothing. This is useful for testing, but not much else.

Windows Scripting Host

If you run a Windows script (myscript.vbs) from cmd.exe or directly from Explorer, it'll run as a 64-bit process, which means it can't see any 32-bit COM objects like ADO connections and datasets. If you explicitly invoke it using C:\Windows\SysWOW64\cscript.exe, it'll run as a 32-bit process.

The Windows\System32 folder

64-bit apps - like the notepad.exe that ships with Windows - can see C:\Windows\System32\ and it's various subfolders. 32-bit apps - like TextPad - can't see this folder because Windows is "hiding" the system folders from the 32-bit process. This is completely baffling when you try to edit your /etc/hosts file using your normal editor and it appears to be completely missing - even though you had it open in Notepad a second ago. There's a thing called the sysnative file system redirector that you'll need to set up to be able to see these folders from 32-bit apps.

The Registry

The same caveat applies to the registry. When a 32-bit app asks for a value stored under, say,

HKEY_LOCAL_MACHINE\Software\MyCompany\MyProject,

64-bit Windows will actually return the value from

HKEY_LOCAL_MACHINE\Software\Wow6432Node\MyCompany\MyProject.

This is normally fine, because most 32-bit apps are installed by a 32-bit installer, so the redirection is in place both during install (when the keys are created) and at runtime (when they're used). If you're manually importing registry keys - e.g. by doubleclicking a .reg file - they'll import into the locations specified in the file, and then your 32-bit apps won't be able to find them. You'll need to manually copy the keys and values into the Wow6432Node subtree (or edit the original .reg file and re-import)

References

http://blogs.msdn.com/helloworld/archive/2007/12/12/activex-component-can-t-create-object-when-creating-a-32-com-object-in-a-64-bit-machine.aspx

http://blogs.sepago.de/helge/2008/03/11/windows-x64-all-the-same-yet-very-different-part-5/

Fun with Server.GetLastError() in classic ASP on Windows Server 2008

2008-12-04T13:17:00+00:00

One of our sites, written many moons ago in classic ASP using JScript, uses a bunch of custom error pages to handle 404 errors, scripting errors, and so on.

Our error handling code looks like this:

var error=Server.GetLastError(); 
var errorMessage=""; 
errorMessage +=Server.HTMLEncode(error.Category); 
if (error.ASPCode) errorMessage +=Server.HTMLEncode(", " +  error.ASPCode); 
var errorNumber=error.number;
errorNumber = (errorNumber < 0 ? errorNumber + 0x100000000 : errorNumber).toString(16);
errorMessage +=" error 0x" + errorNumber + " (from " + Request.ServerVariables("SCRIPT_NAME").Item + ")\r\n\r\n";
if (error.ASPDescription) errorMessage += "ASPDescription: " + error.ASPDescription + "\r\n"; 
if (error.Description) errorMessage +="Description: " + error.Description + "\r\n"; 
// and then we can display/log the error message 

On our old server, this worked because the HTTP 500 error page was mapped to a custom URL, /errors/500.asp, which included the code above.

When we migrated our site onto IIS7 recently, this stopped working - the custom page was still executing, but Server.GetLastError() wasn’t returning any information about what had gone wrong.

There was a very similar known bug in Vista which was supposedly fixed in SP1, but it looks like the same fix isn’t part of Windows 2008 Server yet. There is a workaround, though - if you set the site’s default error property (under IIS settings > Error Pages > Edit Feature Settings…)to the custom page (see below), IIS will invoke this page whenever an error is not handled by an explicitly configured status-code handler (so your 404, etc. handlers will still work) - but for some reason, handling the error this way means Server.GetLastError() still works properly.

Lies, Damned Lies, and Statistics

2008-11-27T20:13:00+00:00

One of our big customer contracts is up for renegotiation next month. This involves pulling a list of all the search & site activity that originated from that customer over the last year, and then negotiating based on whether usage is up or down. Over the last few years we've seen 10%-15% increases from this particular account, year-on-year, which is good. Yesterday morning I ran the stats report and got this:

Not good. In fact, very very worrying indeed. Whilst the marketing team went into crisis mode to work out what the hell we were going to do if this was real, I started double-checking to make sure this was genuine. It certainly looked genuine. The graph is horribly organic, the way the decline is gradual, occasional peaks and troughs, but with a very, very definite downward trend. In my experience, when software fails, it tends to fail in big straight lines - everything just stops working completely and stays there.

Turns out the stats were wrong - huge sigh of relief all round - but the reason why they were wrong is, I think, quite interesting. These statistics are calculated using some custom logging routines in our (legacy ASP) web code. When a user first hits the site, we create a record in the UserSession table in our database that stores their IP address, user agent string, user ID, and so on. There's some counter fields in that table that are incremented over the course of the session as the user accesses particular resources, so we can build up a fairly accurate picture of which resources get accessed heavily, by whom, and at what times throughout the day.

Well, it turns out our CreateUserSession() routine was failing if the browser's UserAgent string was more than 127 characters. Historically, this was never a problem, but at some point last year Microsoft started putting all sorts of information about .NET framework versions and plugins into the HTTP_USER_AGENT header sent by Internet Explorer (Scott Hanselman has a great post about this if you're interested) As various updates were pushed out to our users via Windows Update and corporate rollouts, the user agent strings were getting longer and longer, until one day they'd exceed 127 characters - and that particular PC would stop showing up in our logs. Whenever they'd roll out new hardware, we'd see the stats increase temporarily, until those new boxes were upgraded and the same thing happened. Hence the gradual decline and the fact that non-IE users were unaffected.

We would have noticed this a long time ago, of course - but the CreateUserSession() call was wrapped in a try/catch block that called a notification function when it caught an exception, and somewhere along the line, the notification mechanism for this particular system had been commented out. I'd love to blame someone else for this, but Subversion has a commit with my name on it sometime last year with the relevant line mysteriously commented out.

I believe the kids are calling that an "epic fail". I believe they have a point.

HQL-lo World

2008-11-19T23:58:00+00:00

I've been playing with Castle ActiveRecord for a project I'm working on, and hit a brick wall earlier tonight that left me completely stuck for a couple of hours... and turned out to be incredibly simple and obvious. Turns out I'd refactored one of my business objects - from Page to CmsPage - and hadn't noticed that in one particular place in the code, I was doing this:

var rootPages = new SimpleQuery<CmsPage>(@"from Page p where p.Parent is null");
return (rootPages.Execute());

The Execute() call there was throwing an ActiveRecordException that just said {"Could not perform ExecuteQuery for CmsPage"} - no InnerException, nothing showing up in SQL Profiler, nothing except a bunch of query strings that all looked fine to me:

Even enabling ActiveRecord logging (which was wonderfully easy, by the way) didn't help - I couldn't see anything obviously amiss in the NHibernate logs.

Turns out I'd not yet got my head around a fundamental concept of object-relational mapping, namely that you are querying your objects, not your database. The string literal in the SimpleQuery definition that looks a bit like LINQ is HQL - Hibernate Query Language. I'd used the [ActiveRecord(Table="Page")] attribute to map the renamed class to the underlying DB table, which is still called Page, and it just completely didn't occur to me that the HQL query needs to be changed to reflect the new class name. Change that query to

var rootPages = new SimpleQuery<CmsPage>(@"from CmsPage p where p.Parent is null");

and it works as intended. I fear this ORM stuff is going to take some getting used to...

Usability Tip of the Day: Label your Form Elements, Dammit.

2008-11-08T23:28:00+00:00

I see high-profile, expensive, shiny, corporate websites all the time that don’t label their form inputs. It’s easy. It’s accessible. And – in the case of checkboxes and radio buttons, where the form inputs themselves are about this big :, it’s massively helpful, because in almost every modern browser, you can click the label instead of having to click the actual form element. It’s staggering that so-called professional web developers don’t label their form elements properly. Here’s how you do it:

<input type=”radio” id=”beerRadioButton” name=”beverage” value=”beer” />
<label for=”beerRadioButton”>Beer</label>
<input type=”radio” id=”wineRadioButton” name=”beverage” value=”wine” />
<label for=”wineRadioButton”>Wine</label>

(In ASP.NET WebForms, if you set the AssociatedControlId of an <asp:Label /> control, it’ll render an HTML label element with the correct for=”” attribute; if you omit the AssociatedControlId attribute, it won’t even render as a label...)

Here's a form example without the labels wired up properly:

Beer Wine

And here's the same radio buttons, with the labels wired up properly. See how in the this example, clicking the labels will select their associated radio-buttons, but in the previous form, you have to actually click the radio-button itself?

Beer Wine

Just do it, OK? It helps people using screen readers. It helps mobile browsers. And it helps people downloading Lego instructions after a couple of beers. Trust me.

Working on ASP.NET MVC Beta and Preview code side-by-side

2008-11-08T15:53:00+00:00

I have an app built against ASP.NET MVC Preview 3 that needs some tweaking, and I'm also working on a couple of projects using ASP.NET MVC Beta, so I'm in the slightly odd situation of trying to build and run preview 3 and beta projects side-by-side. (Yes, I will be updating this code to run against the beta version. I don't have time to do that this weekend, though, and I need some changes live before Monday afternoon.)

I've just checked-out the preview 3 project to make some changes, and although it builds absolutely fine, I'm seeing the lovely Yellow Screen of Death when I try and run it:

Server Error in '/' Application.

Method not found: 'Void System.Web.Mvc.RouteCollectionExtensions.IgnoreRoute (System.Web.Routing.RouteCollection, System.String)'.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.MissingMethodException: Method not found: 'Void System.Web.Mvc.RouteCollectionExtensions.IgnoreRoute(System.Web.Routing.RouteCollection, System.String)'.

This is weird, because this code is deployed and running live on a box that doesn't have any versions of MVC installed; in theory, the project is entirely self-contained and XCOPY-deployable. First thing I tried was to shut down Visual Studio, uninstall ASP.NET MVC Beta, reinstall Preview 3, reload VS2008. That worked, so it's definitely the beta doing something strange. This project has hard-wired references to copies of the MVC assemblies in the \Dependencies folder of the solution, which are copied to the \bin folder during the build. It looks like the beta is installing something that's interfering with this process. Frustratingly, the installers also set up the MVC Web Application project type in Visual Studio, so although I can run the site without any versions of MVC installed, I can't open it in VS2008 because of the "project type is not supported" error.

Ok, first thing to realize is that, according to ScottGu's beta release blog post, the beta installs System.Web.Mvc, System.Web.Routing and System.Web.Abstractions to the GAC to allow them to be automatically updated. The preview versions of MVC would only install them to C:\Program Files\Microsoft ASP.NET\.

Given this particular chunk of web.config code:

<system.web>
 <compilation debug="true">
<assemblies>
    <add assembly="System.Web.Mvc, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>
   </assemblies>
</compilation>
</system.web>

the runtime is going to use the first version of System.Web.Mvc matching the specified culture, version number and public key token. This is significant because the CLR checks the GAC first when resolving assembly references - and if it finds a matching assembly in the GAC, it won't look anywhere else. The ASP.NET MVC previews and beta release all use the same assembly version, culture and public keys, so the CLR has no way of distinguishing between the preview 3 version of System.Web.Mvc and the beta version of the same assembly. They're different DLLs with different file versions, but because the assembly version is the same, the CLR regards them as the same assembly.

There are techniques you can use to override this behaviour, but, according to this thread on StackOverflow, these techniques only work if the assembly in the GAC has a different version to the assembly that's deployed with your application.

Ok - no problem, we'll just remove System.Web.Mvc from the GAC, by running gacutil.exe /u to uninstall it.

C:\Documents and Settings\dylan.beattie>gacutil /u system.web.mvc
Microsoft (R) .NET Global Assembly Cache Utility. Version 3.5.30729.1
Copyright (c) Microsoft Corporation. All rights reserved.

Assembly: system.web.mvc, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL
Unable to uninstall: assembly is required by one or more applications
Pending references:
SCHEME: <windows_installer> ID: <msi> DESCRIPTION : <windows installer>
Number of assemblies uninstalled = 0
Number of failures = 0

C:\Documents and Settings\dylan.beattie>

OK, that didn't work. Because we installed the ASP.NET MVC beta using Windows Installer, it's registered a dependency on System.Web.Mvc that means we can't uninstall it. So... registry hack time. This is the bit that might kill your PC, wife, cat, whatever. Editing the registry is dangerous and can cause all kinds of problems, so read this stuff first, and if it sounds like a good idea, proceed at your own risk.

Fire up regedit and navigate to HKEY_CLASSES_ROOT\Installer\Assemblies\Global, and you should find a key in there called

System.Web.Mvc,version="1.0.0.0",culture="neutral",publicKeyToken="31BF3856AD364E35",processorArchitecture="MSIL"

I deleted this key. I also got a bit carried away and deleted the key

System.Web.Mvc,1.0.0.0,,31bf3856ad364e35,MSIL from

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\GACChangeNotification\Default

as well... but I forgot to try gacutil /u first, so I don't know whether this second step is necessary or not. It seemed like a good idea, though, and doesn't appear to have broken anything, so you may or may not need to delete this second key as well.

Having removed those keys, I could run gacutil /u and remove System.Web.Mvc quite happily:

C:\>gacutil /u System.Web.Mvc
Microsoft (R) .NET Global Assembly Cache Utility. Version 3.5.30729.1
Copyright (c) Microsoft Corporation. All rights reserved.

Assembly: System.Web.Mvc, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL
Uninstalled: System.Web.Mvc, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL
Number of assemblies uninstalled = 1
Number of failures = 0

C:\>

My preview 3 project now builds and runs quite happily against the System.Web.Mvc DLLs installed as part of the website, and the VS2008 MVC Project template still works just like it did before.

A Rant about RAID, with a Bad Metaphor about Eggs, and No Happy Ending.

2008-11-03T21:53:00+00:00

I went in to work this morning and my main workstation had died over the weekend. Bluescreen on boot, no safe mode, nothing. Windows Update gone bad? We'l l probably never know, given I don't think it's coming back any time soon... but, as with previous overnight machine suicides, it looks like a problem with SATA RAID - specifically, two WD Velociraptors in a RAID-1 (mirror) array controlled by an Intel ICH10R chipset on an Asus P5Q motherboard.

You know your whole eggs & baskets thing, right? SATA RAID is like carefully dividing your eggs into two really good baskets, then tying them together with six feet of wet spaghetti and hanging them off a ceiling fan.

Long story short, I lost a day, and counting. I had to split the mirror into individual drives, switch the BIOS back to IDE, which gave me a bootable OS but - seriously - no text. No captions, no icon labels, no button text, nothing. Just these weird, ghostly empty buttons. Running a repair off the WinXP x64 CD got my labels back, but somehow left Windows on drive D. Another half-hour of registry hacks to get it back to drive C: where it belongs, and I had a creaking but functional system - VS2008 and Outlook are working, but most of my beloved little apps are complaining that someone's moved their cheese. Reinstalling is probably inevitable, along with the deep, deep joy that is reinstalling Adobe Creative Suite when your last remaining "activation" is bound to a PC that now refuses to deactivate it. Even Adobe's support team don't understand activation. Best they could come up with was "yes, that means there's no activations on that system." Err, no, Mr. Adobe, there are. It was very clear on that point. Wouldn't let me run Photoshop without it, you see. "Oh... then you'd better just reformat, and when you reinstall, you'll need to phone us for an activation override". Thanks, guys. I feel the love.

Sorry, I digress. This whole experience is all the more frustrating because RAID mirrors are supposed to be a Good Thing. If you believe the theory, RAID-1 will let you keep on working in the event of a single drive failure. Well... In the last 5 years or so, I haven't had a single workstation die because of a failed hard drive, but I've lost count of the number of times an Intel SATA RAID controller has suddenly thrown a hissy-fit under Windows XP and taken the system down with it. Every time it starts with a bit of instability, ends up a week or two later with bluescreens on boot and general wailing and gnashing of teeth, and every time, running drive diagnostics on the physical disks shows them to be absolutely fine.

This is across four different Intel motherboards - two Abit, one Asus, and a Dell Precision workstation - running both the ICH9R (P35) and ICH10R (P45) chipsets, and various matched pairs of WD Caviar, WD Raptor, WD Velociraptor and Seagate drives. One system was a normal Dell Precision workstation, the others are various home-built combinations, all thoroughly memtest86'ed and burned-in before being put into production doing anything important.

Am I doing something wrong here? I feel like I've invested enough of both my and my employer's time and money in "disaster-proofing" my working environment, and just ended up shooting myself in the foot. I'm beginning to think that having two identical workstations, with a completely non-RAID-related disk-mirroring strategy, is the only way to actually guarantee any sort of continuity - if something goes wrong, you just stick the spare disk in the spare PC and keep on coding. Or hey, just keep stuff backed up and whenever you lose a day or two to HD failure, tell yourself it's nothing compared to the 5-10 days you'd have lost if you'd done something sensible like using desktop RAID in the first place.

[Photo from bartmaguire via Flickr, used under Creative Commons license. Thanks Bart.]

The Roadcraft of Programming

2008-11-02T18:02:00+00:00

I was chatting with Jason "Argos" Hughes after the Skillsmatter event last week, and he said something I think is really quite brilliant, so I hope he doesn't mind if I quote him here and expand on his ideas a little.

We were discussing the merits of various different platforms and programing languages, and he said "knowing a language inside-out doesn't make you a better programmer, any more than knowing a lot about a particular car makes you a better driver".

That comment has been going round and round my head ever since, and I think that's one of the most insightful metaphors about programming languages that I've heard. Anyone who's owned a car will know that every make and model - and every individual example of a particular model - has its idiosyncrasies and quirks. I drive a slightly knackered Vauxhall Tigra. On this particular car, I know that I need to replace the cam-belt every 40,000 miles or Really Bad Things might happen. I know that I need to clean the gunk out of the frame around the back window otherwise it fills up with rainwater; I know where the little lever to adjust the seats is, and where all the various controls and switches are, and how to check the oil and change the headlamp bulbs.

None of this makes me a good driver. In fact, it has absolutely nothing to do with my driving ability. Beyond a basic familiarity with a vehicle's controls and signals, the Highway Code has very little to say about the quirks and idiosyncrasies of particular cars. On the other hand, it has rather a lot to say about stopping distances, speed limits, lane discipline, the importance of maintaining awareness of your surroundings and communicating your intentions clearly to other road uses. In other words, being a good driver boils down to discipline, restraint, awareness and communication - your choice of vehicle is largely irrelevant. Good drivers are good whatever they're driving, and the choice of car alone can't turn a poor driver into a good one.

I think there are strong parallels here with software development. Good coders are like good drivers; they'll work within the safe parameters of whatever technology they're using, exercise restraint and discipline in the application of that technology, and rely on awareness and communication to make sure that they're doing doesn't create problems for other people.

Programming interviews can easily degenerate into a pop-quiz about the characteristics of a particular language or platform, but maybe we should be approaching them more like a driving test - even to the extent of letting the candidate demonstrate their problem-solving capabilities using whatever languages and tools they're comfortable with, and then discussing the results in terms of clarity, effective communication, restraint and awareness. Even though we're a .NET shop, I can see how a developer who can create elegant solutions in Ruby or Java and explain clearly what they've done might be a better .NET programmer than somebody who knows every quirk of C# and ASP.NET but can't demonstrate those core qualities of discipline, restraint, awareness and communication.

Huagati DBML Tools for Linq-to-SQL

2008-11-02T17:37:00+00:00

I haven't had a chance to use them yet, but the Huagati DBML/EDMX tools look interesting - a set of extensions to the DBML designer in Visual Studio 2008 that provide some additional functionality, including the much-needed ability to update your DBML to reflect changes in your database schema. It's a commercial package costing $119.95 per user, but a free trial license is available.

With Microsoft effectively abandoning Linq-to-SQL, it's good to see tools like this in the wild. Of course, it'd be really good to see Microsoft open-source Linq-to-SQL and let the community develop it as they see fit... but failing that, these tools can make things easier if you're maintaining an existing Linq-to-SQL system.

Help! I have Multiple Internet Personality Disorder!

2008-10-20T00:13:00+00:00

Update - looks like you can already do this. chrome.exe will accept a --user-data-dir="" switch, so you can set up shortcuts with different profiles - and it works, really quite well. I now have three Chrome shortcuts that bring up different homepages with different sets of persisted cookies. No colour-coding or cool icons, though...

I have too many Google accounts. Or rather, I have the right number of Google accounts for me, but that's too many for Google, who would seemingly be much happier if I only had the one.

I'll explain. I have a Gmail mailbox, which I forward copies of stuff to, so I can get hold of it from anywhere. My main mailbox runs on a Linux box so old I think it's actually running Redhat instead of Fedora, so Gmail acts as a second-level backup strategy as well. There's a couple of calendars and things in this Google account as well. I also have a Google account linked to my 'real' e-mail address, which I use to sign in to Blogger and various other online services that have ended up under the Google umbrella. Then I have another set of credentials, which are the accounts we've set up at work for access to stuff like Google Webmaster Tools and Google Analytics. I'm not entirely happy with Google linking my e-mail or my blog to my employer's website statistics, so I keep this separate as well.

Basically - I want multiple, independently-persisted identities, with their own history, their own cookies and their own shortcuts, so when I say "remember me", I'm actually saying "remember who I'm pretending to be right now" Google Chrome already has 'incognito mode' (and we all know what that means, right?). Can we have work mode, home mode, geek mode, pretending-to-be-a-client-so-I-can-test-my-own-website mode, and as many other modes as we want? With their own colours? And icons? And desktop / start menu shortcuts?

Actually, it doesn't have to be Google Chrome at all, it's just that their little "secret agent" icon guy worked really well for the screen mockup. Firefox could do this. Or even Internet Explorer. I know there are cookie-switcher add-ons for Firefox et al, but what none of these solutions offer, as far as I can tell, is the ability to use multiple identities simultaneously - and since Google's made such a big thing of Chrome's separate-processes-for-each-tab technology, it seems like it couldn't be too hard to give those processes their own profiles and history.

Adding a work-in-progress to Subversion

2008-10-18T13:35:00+00:00

I love Subversion, but from time to time I'll stumble across a bit of SVN behaviour that just doesn't feel quite right. Case in point - you've created 10-15 files, set up a folder structure for a new project, made rather more progress than you were expecting to, and now you want to check the whole thing into revision control.

The 'proper' way of adding existing code to a repository is via the svn import command, but that doesn't turn your local folder into a Subversion working copy. Having completed the import, you'll then need to move/rename/delete your work in progress, and then do an svn checkout to download the version of your project that's now under revision control. This can take a while if you're working on big files and your repository is on the far end of a slow connection... and even when that's not applicable, it's still frustrating.

So, here's how you can add a new project to Subversion without having to do the import-checkout shuffle.

Use the repo-browser to create a new empty folder in the repository - this will form the root folder of your new project, so call this folder /myproject/trunk or whatever you'd normally use.
Check out the empty folder into the folder containing your work-in-progress project. You'll get this warning - which is fine, because what you're doing is 'wrapping' an empty SVN folder around your existing work.
You'll check out a single folder, and you'll see that your project now consists of a root folder with the happy green SVN icon, containing a bunch of folders with the question-mark overlay that means "Subversion doesn't know about this folder yet..."
Now you can do an svn commit in the usual way, and it's trivial to add the 'new' files (i.e. all of them) that should be added to the repository. On the first commit, you'll need to uncheck the bin/obj folders for .NET projects, and then on the subsequent commit, you'll be able to add them to the SVN ignore list (you can only ignore a folder whose parent is already under version control)

Googling the Zeitgeist

2008-10-17T14:36:00+00:00

Just for fun, I googled "website", and got a little glimpse into the internet zeitgeist as Googlebot sees it. With such a generic query, it's basically comparing websites based on a lowest common denominator and, presumably, the sites with the greatest number of incoming links and highest page-rank bubble to the top.

www.google.com tells us the most important website-related websites in the world, right now, are

Website - Wikipedia, the free encyclopedia
Welcome to Obama for America - Barack Obama's presidential campaign website
Microsoft
Website.com (who are presumably here because they talk about websites a lot)
Adobe
Apple
The IRS (US tax and revenue agency)
Starbucks
McDonalds
Subway (restaurants)

That's interesting... Wikipedia, Barack Obama, tech companies, coffee, taxes and fast food. It's like a little summary of the daily lives of hi-tech America. (Worth noting that of the sites in that list, Microsoft, Apple and Adobe have a Pagerank of 9/10, while website.com has a fairly unremarkable 6/10)

Then Ben Taylor tried the same thing on Google UK, which gives us

The BBC
Banksy (the "street artist")
en.wikipedia.org/wiki/Website
Oasis (the band)
The Secret Intelligence Service
The British royal family
Bloc Party (the band)
The National Trust (an organisation that works to protect historic buildings and sites of natural beauty in the United Kingdom)
Number 10 Downing Street (official residence of the Prime Minister)
Shakespeare's "Romeo & Juliet" on Google Books.

That's us... Graffiti, Shakespeare, history, and rock'n'roll... how very British.

It'll be interesting to see how those lists change over time... graphing the progress of specific topics up and down the Google "website" results over time would make for interesting viewing. Watch this space. Or rather, come back in about six months when I've got the data, and then watch this space.