It’s been a little quiet on the blog recently, and the reason for that is either perfectly valid or profoundly unnecessary, depending on your viewpoint. Even I am not entirely sure which one it is.

Over the past couple of weekends (and too many late weeknights) I have used all my spare time to build a mini-site for my obsession with music. It started as a small idea to just show the current track I’m listening to, and a list of recent physical albums I added to my collection. But then it snowballed into something much more. You can view the site at music.elezea.com, or by clicking on the link in the top navigation. If you want to know a bit more about how it works, read on!

It all started when I came across Andy Bell’s mini-site for his music collection. He uses a Notion database and Last.fm to show all the music he has in his collection, and what he’s listening to. Since I also still use Last.fm (yes, it’s still around!), and all my physical music is documented on Discogs, I wanted to build a small site that uses the Last.fm and Discogs APIs to show some of that information.

But once I got started and got stuck into all the information available via those APIs, I just couldn’t stop. I still have so much more I want to do, but I know it’s time to take a break. All in all this has been such a fun and rewarding thing to spend my time on. I know the site has pretty much zero value to the world at large. But I love checking it to get more information about something I’m listening to—and it helped me take quite a few steps forward in my technical skills. So I’m choosing to call it a win.

Below are some notes about what the site does, how it works, and also what the experience was like for me (as a non-developer trying to learn).

Now playing

• Get the most recently played track from the Last.fm API, and check if the song is currently playing or not.
• If it’s currently playing, show the current time and a message that what you’re seeing is what I’m listening to in real-time.
• If it’s not currently playing, mention that and show the time it was played.
• Pull in the cover art and other data about the song from Last.fm.
• Do a lookup for the artist and if Last.fm has data about them, show the first two tags (genres), first 3 related artists, and their bio.
• Do another lookup to a different API endpoint for the artist’s top albums, and display data about their two most popular albums.
• Use the YouTube API to do a lookup for the song, and embed the most relevant result in the page so that you can listen to it right there.

Top albums and artists

• Show the top albums I listened to in the last 7 days, including play counts.
• Show the top artists I listened to in the last 7 days, including play counts.
• Make a separate API call for each artist to get their genres and similar artists, if that data exists on Last.fm.
• Make another API call to get each artist’s most popular albums.

Recent purchases

• Pull the last 6 releases I added to my Discogs collection.
• Also pull in data about the genre, label, and release date.
• ⏳ The Discogs API is really great, so I want to add a bunch more stuff here, but that’s also for the mythical v2 of this thing.

Random thoughts, complaints, and what’s next

• The site is deployed with Netlify via a Github repo, and it just works. Netlify is so great.
• I don’t care what you “real developers” say, the two biggest problems in programming are environmental variables and formatting dates. I am thankful for ChatGPT for helping me with the date formatting piece, and my colleague and friend Derek for helping to get the environmental variables to work.
• Last.fm’s API clearly hasn’t been touched in years and the documentation isn’t great, so it’s been a bit of a mission to figure all that out. Postman has been a life-saver here to test the API calls and see what data comes back.
• YouTube’s API has a limit of 100 search lookups per day, which feels ridiculously low. I hit that within an hour while I was building and testing it. Oops! On the upside: I am now much better at error handling. If the site hits that limit it will now show a message to that effect, and link to a direct search on YouTube for the song.
• ⏳ I’m using YouTube only because the Spotify API makes it incredibly difficult to get an auth token. Auth tokens expire after 1 hour, and refreshing that token every hour is currently beyond my limited skills. I might come back to this as well because the Spotify API has sooo much interesting data.
• ⏳ Another huge data source is the MusicBrainz API. I plan to spend some time wading through those docs as well to see what else I could add.
• If you can think of any other cool things I might want to add to this, please reach out on Mastodon!

The FAA outage in January that caused the first nationwide ground stop of all flights in the US since 9/11 is kind of old news now, but there’s one detail that I can’t stop thinking about. In the aftermath of the incident the cause was determined to be a database sync issue:

The F.A.A. said in a statement that the workers had been trying to “correct synchronization” between the main database for the Notice to Air Missions alerts and a backup database when the files were mistakenly deleted, causing the outage that snarled air traffic throughout the day on Jan. 11.

CNN added a little more detail:

A contractor working for the Federal Aviation Administration unintentionally deleted files related to a key pilot safety system, leading to a nationwide ground stop and thousands of delayed and canceled flights last week, the FAA said Thursday.

The FAA determined the issue with the Notice to Air Missions (NOTAM) system occurred when the contractor was “working to correct synchronization between the live primary database and a backup database.”

The unsurprising narrative that came out of the tech world following the incident can basically be summarized as “ha ha, silly contractors!” But that feels like a lazy response to me. I didn’t see anyone ask what I believe is the more important question: How do we improve the system (people, processes, technology) that enables one person to inadvertently take down all air traffic in the US?

Let’s remember that this kind of thing can happen to absolutely anyone. Etsy even hands out a “three-armed sweater” award to the engineer who had the most spectacular mishap in any given year:

Kate’s story is a nail-biter, involving a tiny code change that unexpectedly brought down Etsy.com. All of her coworkers rallied around her to help get the site back online, while offering words of encouragement and reassurance.

So it might be really convenient to blame the FAA outage on “contractor error” and then just keep going. But that’s not going to prevent the next incident from happening.

It is further also tempting to blame the entire issue on “tech debt” and call it a day. And, fair enough, there’s certainly plenty of that going around in FAA systems. Ars Technica has a good overview of some of the major issues and how the FAA wants to fix them. But like all giant “replatforming” projects (this one is called NextGen, because of course it is) things are… not going great:

FAA tech problems were previously described in a March 2021 report by the US Department of Transportation Office of Inspector General. The report discusses the FAA’s Next Generation Air Transportation System (NextGen), “a multibillion dollar infrastructure project aimed at modernizing our Nation’s aging air traffic system to provide safer and more efficient air traffic management.”

“NextGen’s actual and projected benefits have not kept pace with initial projections due to implementation challenges, optimistic assumptions, and other factors,”1 the report said.

But blaming tech debt—and especially blaming individuals—is not going to get us very far. Tech debt will always be there (although I have some thoughts on how to prioritize it), and individual mistakes are not going to go away. What we can do is examine the system that enables, in this case, a database sync to corrupt the primary live db, and figure out how to prevent that from happening in the first place.

Almost 30 years ago Jakob Nielsen published his 10 Usability Heuristics for User Interface Design, and “error prevention” is still as true today as it was then:

Good error messages are important, but the best designs carefully prevent problems from occurring in the first place. Either eliminate error-prone conditions, or check for them and present users with a confirmation option before they commit to the action.

The example I always think of here is how you often seen battery packs shaped in a certain way so that it’s impossible to insert them incorrectly (contrast that with the terrors of trying to insert a USB cable the correct way the first time!).

In a situation like the one the FAA experienced, yes it’s important to acknowledge human error, and talk about the underlying tech issues, but that’s not enough. We have to figure out how to add preventative measures to our systems and pipelines2. To put it another way, they might not be able to replace their battery packs with NextGen solar yet, but they can certainly change the shape of the battery to prevent contractors from blowing up the camera.

---

1. My emphasis added because who among us have not heard those words before… ↩

2. For further reading on what to do after a major incident, check out Will Larson’s Move past incident response to reliability. ↩

Mike Fisher argues that we should rebrand technical debt as “product debt”, and I think it’s a good argument! That said, I’d like to add some considerations to two of his points. First:

We usually think of refactoring as “cleaning up” code, where we change the code to be more easily understood, perform better (faster or more efficiently), or follow current conventions/standards. The goal of refactoring is to change the code without changing its functionality; it should continue to pass all unit and functional tests. 

I take a slightly different approach to refactoring, and how to prioritize the work. I believe it’s important for teams to have a stated and agreed-upon value of “leave the code better than I found it.” This means that refactoring shouldn’t be a separate activity, for its own sake, that needs to be scheduled. It should be a natural part of feature development.

If you’re creating a mechanism for add-ons on the product, spend a few extra days to refactor the billing code you’re already working on. If you are adding metrics to the dashboard in your UI, take the time needed to refactor the front-end code to make it more performant. Whatever code you’re touching while you’re working on a project, leave it better than you found it. It is way more efficient to extend a project by a week to refactor code you’re already working on than it is to create a separate project that needs to be planned, prioritized, and worked into the roadmap.

Second point:

So, how do we ensure we are paying down technical debt when there is so much pressure to ignore it until things really break? I think one part of the answer is to use a different term. Instead of tech debt, which implies it is the responsibility of the tech team, let’s call it product debt.

I think this is a good first step to getting more teams to care about technical debt—but it’s not enough. One of the issues with getting technical/product debt prioritized is that often “the business” doesn’t see the value in statements like “we’re going to clean up the code so that it doesn’t break a few months from now”. Instead, we need to frame the work in terms of the benefits to customers and/or the business.

For instance, we could make the case that refactoring this piece of code would significantly increase our deployment speed, which would mean faster time to market. Or we could argue that fixing our slow staging environments would result in happier, more productive engineering teams.

With technical debt—as with most things in software development—the thing you do is never the main thing. The main thing is what the thing you do enables. What value it brings to customers and the business. That’s the framing we need for working on technical debt.