Error'd: April is Special, and so are you
"April is special," writes Elwin. It is, but take heart May, every month is special at TDWTF.
"Admiral Ackbar is pinterested," punned The Beast in Black
Manuel H. clocked something off on this website. "Noon seems to be very late in Lithuania, or maybe only in this hotel restaurant in Vilnius." 15H AM must be on some planet with a 32H day.
"Amazon can't make up its mind!" ranted an anon. "Do I need to wait 2 business days or 3? Make up your mind Amazon!"
Duston decided to close us out with a pun. "Looks like they have a problem, but it's trivial." Well done.
CodeSOD: In the Know
Delilah works in a Python shop. Despite Python's "batteries included" design, that doesn't stop people from trying to make their own batteries from potatoes. For example, her co-worker wrote this function:
def key_exists(element, key): if isinstance(element, dict): try: element = element[key] except KeyError: return False return TruePython, of course, has an in operator. key in dictionary is an extremely common idiom. There's no reason to implement your own. Certainly, there's no reason to re-implement it by catching and throwing exceptions.
This is ugly, stupid, and bad. It gets worse, though, when you see how it gets used.
for key in old_yaml_data: if key in new_yaml_data: if old_yaml_data[key] != new_yaml_data[key]: temp = new_yaml_data[key] new_yaml_data[key] = merge(new_yaml_data[key], old_yaml_data[key]) if key_exists(new_yaml_data[key], 'image') and key_exists(old_yaml_data[key], 'image'): new_yaml_data[key]['image'] = temp['image'] elif key == "databases": revert_db_tags(new_yaml_data[key], temp)This code is attempting to upgrade "old" YAML data with "new" data. So it's basically merging dictionaries, which is a great case for the in operator.
And they use the correct idiom on the second line there! This was written by one developer! They do the standard key in new_yaml_data check. And they also use key_exists. I can only assume that they had a stroke between starting and finishing this script, which I'll note is, in total, 48 lines long.
Here's the whole short script, which is just generally a mess. Slapped together Python code that's trying to be a "smarter" shell script, but is definitely written with the elegance of hacked-together-bash.
import sys import yaml from jsonmerge import merge appHomePath = sys.argv[1] oldValuesYAML = appHomePath + "values.yaml" newValuesYAML = appHomePath + "/upgrade_version/values.yaml" with open(newValuesYAML, 'r') as f: new_yaml_data = yaml.load(f, Loader=yaml.loader.FullLoader) with open(oldValuesYAML, 'r') as f: old_yaml_data = yaml.load(f, Loader=yaml.loader.FullLoader) def key_exists(element, key): if isinstance(element, dict): try: element = element[key] except KeyError: return False return True def revert_db_tags(old_yaml_data, new_yaml_data): dbList = ["mongoDB", "postgresDB"] mongoDbTagsToRevert = ["mongoRestore"] mongodbKeysToDelete = [] postgresDbTagsToRevert = [] for db in dbList: old_yaml_data[db]['image'] = new_yaml_data[db]['image'] for mongoDbTag in mongoDbTagsToRevert: old_yaml_data['mongoDB'][mongoDbTag]['image'] = new_yaml_data['mongoDB'][mongoDbTag]['image'] for mongoDbTag in mongoKeysToDelete: del old_yaml_data['mongoDB'][mongoDbTag] for postgresDbTag in postgresDbTagsToRevert: old_yaml_data['postgresDB'][postgresDbTag]['image'] = new_yaml_data['postgresDB'][postgresDbTag]['image'] for key in old_yaml_data: if key in new_yaml_data: if old_yaml_data[key] != new_yaml_data[key]: temp = new_yaml_data[key] new_yaml_data[key] = merge(new_yaml_data[key], old_yaml_data[key]) if key_exists(new_yaml_data[key], 'image') and key_exists(old_yaml_data[key], 'image'): new_yaml_data[key]['image'] = temp['image'] elif key == "databases": revert_db_tags(new_yaml_data[key], temp) with open(newValuesYAML, 'w') as f: data = yaml.dump(new_yaml_data, f, sort_keys=False) [Advertisement] Plan Your .NET 9 Migration with ConfidenceYour journey to .NET 9 is more than just one decision.Avoid migration migraines with the advice in this free guide. Download Free Guide Now!
CodeSOD: Find a Bar for This One
A depressing quantity of software is what I would call a "data pump". I have some data over here, and I need it over there. Maybe I'm integrating into a legacy app. Or into an ERP. Or into a 3rd party API. At the end of the day, I have data in one place, and I want it in another place.
Sally has a Java application written in the Quarkus framework, which has a nightly batch that works to keep a table of Bar entities in sync with a table of Foo entities. (This anonymization comes from Sally) These exist in the same database. There is also a Bar webservice, which provides information about the Bar entities. The workflow, such as it is, is that the software needs to find all of the Foo entities that do not currently have associated Bar entities, and then call the Bar webservice to get the required information to create those Bar entities.
Let's see how that works.
@Inject UserTransaction transaction // If this is annotated with @Transaction the usage in the Message function down below will have some Thread exception public List<FooData> getAllFoos() { try{ return fooDataRepository.findAllFoos(); } catch (Exception e) { throw new RuntimeException(e); } }We'll worry about that comment in a second, but this function returns a list of all of the Foo objects in the database. It does not return a list of all the Foo objects without associated Bar entities. It's just the whole giant list of everything. The underlying database is a standard relational database; it'd be trivially easy to write that query, even going through the ORM.
Well, that's bad, but it's all pretty minor. How does the actual update go?
// Can't be annotated with @Transaction because Oracle DB can handle the given Amount of dataEntities in one Transaction '\._./' Message updateBarsWithFoos() { List<FooData> foos = getAllFoos(); if(!foos.isEmpty()){ foos.forEach(foo -> { try{ transaction.begin(); if(barRepository.findByName(foo.getName()) == null){ if(barDataService.searchByName(foo.getName()) != null && barDataService.searchByName(foo.getName()).marker() != null){ barRepository.createBar(barDataService.searchByName(foo.getName())); } } transaction.commit(); } catch (Exception e) { try { transaction.rollback(); } catch (Exception ex) { throw new RuntimeException(ex); } } }); } return new Message(MessageLevel.INFO, "Created bars") };Ah, the real WTF is that it's an Oracle database. That's always a WTF.
But let's trace through this code.
We get all of our Foo entities. We check for emptiness and then do a forEach, which seems to make the empty check superfluous: a forEach on an empty list would be a no-op anyway.
We start a transaction, then check the database: if there are no Bar objects that link to Foo, then we call into the barDataService to find data. If there is, we call into the service again, to see if the marker property is not null. If it is, we call into the service again to get the actual data we're putting into the database. Then we close the transaction. If anything goes wrong, we rollback the transaction and chuck an exception up the chain.
That is three web service calls inside of a database transaction. Three calls which could easily be one, and that call could easily also happen outside of a transaction if you're mindful about confirming your constraints. And of course, because they're not mindful at all, they need to manage the transaction directly, and can't use the @Transaction annotation provided by their framework, which would at least cut down on some of the boilerplate.
Now, I'm sure you'll be shocked - shocked - to learn that the webservice is actually a bit flaky, and thus times out from time to time. And this isn't the only batch job running, which means the long-lived transactions cause all sorts of contention and terrible performance across the various batches. And this app doesn't have its connection pool properly configured, so the entire software stack can exhaust all of its database connections surprisingly quickly, causing yet more failures.
The root of the WTF, of course, is doing this as a batch job. A well engineered application would do everything it could to not create data in the database that isn't referentially sound. There, Sally gives us the one bit of good news:
My current project will do away with the batch processing altogether, so we can say, "RIP, transactional wholesale triple caller!"
Three Digit Acronyms
JB has a database table that, at first glance, looks like one of those data warehouse tables that exists to make queries performant. You know the sort, the table that contains every date between 1979 and 2050, or every number out to 1,000,000 or something. It looks dumb, but it helps make certain joins and queries performant.
The database table is called three_alpha_numerics. It has two columns: digit, which contains three characters, and is_numeric, which is a a single character: 'Y' or 'N'. It looks roughly like this:
+-------+------------+ | digit | is_numeric | +-------+------------+ | 009 | Y | +-------+------------+ | 00A | N | +-------+------------+So, for example, if you wanted all the possible numeric triples, you could SELECT digit FROM three_alpha_numerics WHERE is_numeric = 'Y', which is obviously the easiest thing one can imagine.
So what is this for? Well, it's used by a stored procedure that generates unique IDs. That stored procedure does a left join against another table to find all the unused digits. And here's the real gotcha: that stored procedure only ever uses the rows where is_numeric is Y, meaning the vast majority of the data in this table is never used.
Unique IDs, of course, are an incredibly difficult task for databases to do, so it absolutely makes sense that we create a system that allows us to only have 1,000 unique IDs. That's more than 640, which should be enough for anyone. Having many thousands of unusable alphanumeric triplets is just the cost we have to pay.
Representative Line: Dating Backwards
Another representative line, and this one comes from an Excel spreadsheet. But, per Remy's Law of Requirements gathering ("No matter what the requirements doc says, what your users wanted was Excel"), this one was actually written by a developer. A developer who didn't understand how Excel works, but more important, didn't understand how dates worked either.
This comes from Ulysse J.
=CONCATENER(SI(MOIS($A18)>9;ANNEE($A18)-2000;(ANNEE($A18)-2000)*10);SI(JOUR($A18)>9;MOIS($A18);MOIS($A18)*10);JOUR($A18))Now, the first thing: Excel function names are locale specific. This was written in France, so the functions are French. CONCATENER is "concatenate", SI is "if", MOIS is "month", and so on.
The purpose of this function is to convert a field (cell A18) in DD/MM/YYYY into YYMMDD. So how does it do this?
Well, we check the month. If it's greater than 9, we output the year minus 2000. If it's less than 9, then, we output the year minus 2000, multiplied by 10. That is to say, August, 2026 would start by outputting 260. We repeat this logic for the days: if the day is larger than 9, we output the month, otherwise we output the month times 10. Finally, we output the day.
This is attempting to do padding. There's just a problem. Imagine February 1st, 2009- an actual date in the document. We convert the year into 90, the month into 20, rendering the date as 90210. That's incorrect. And once we get to 2100, if there is still an Excel in 2100 (I joke: of course Excel will still exist in 2100. Humanity won't, but the robots will use Excel), this will also break. Not that it matters- I mean, YYMMDD doesn't make sense by that point.
Obviously, the correct solution is to use Excel's rich, built-in formatting functions to convert between date formats. It's easy! But Ulysse raises another point:
Extra points: even if you do not know how to do proper [formatting], the input format is guaranteed to have correct padding. I would just concatenate parts of it (treating dates as text is bad, but still less bad than treating them as integer triplets).
I will say this: I know a software developer wrote this, because your average Excel user could easily write bad formulas, but never bad in this kind of convoluted way. You need a real expert to do something this bad.
Error'd: Balmenach Bad Gateway Single Malt
"Winner ad placement!" snarked our Peter G.
Errors on this website are always a shoo-in for the weekly column. An anonymous reader wrote "I got error 500 when I tried to submit an Error'd. Please make the file uploader check if the attached file is within the file upload limit, which I think is less than 4 MB." They shared an audio error'd which may be coming along next week.
"Give us feedback - wait, did it work at all?" confused poor I_Absolutely_Want_To_Give F. "As every good service management company, ServiceNow wants feedback, above all."
"0 minutes does not equal 0 seconds..." sagely summarized Daniel D. "Claude like floors. I mean floor. But maybe ceil would be better applicable to this calculation, right?"
Finally, this one is a real novelty, from Adam R. Is the label actually 27 years old? It certainly could be; Error 502 is a good bit older. But I think this would be our oldest Error'd yet. Adam explained: "This appears to be a real auction for a whiskey bottle whose label does, in fact, say Error 502 Bad Gateway on it. The winning bid: £130. Source: https://www.scotchwhiskyauctions.com/auctions/228-the-179th-auction/876095-balmenach-1998-27-year-old-error-502-bad-gateway-thompson-bros/"
[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.
The Pride Goeth
Janči, a master's student of bioinformatics, was seated near the back of a large classroom. This was a simple compulsory elective course geared toward biologists. The professor was currently walking the class through their latest assignment. "We'll need to connect to some Linux servers," he announced.
The other students seated nearby traded blank stares. They were all Mac and Windows users with no IT background. Meanwhile Janči, a veteran Linux user, started feeling a little smug. An easy A was at hand.
"First," the professor continued, "you'll need a private key."
After the professor had explained a few details, the first WTF came in the form of a bulk email sent to the entire class. The private key was attached. The username was the email address it was sent to.
What do you call the exact opposite of a private key? Janči wondered, bemused.
"You'll also need to download an application to help you log in," the professor said. "I recommend MobaXterm."
As he detailed the process of visiting the SSH client website to download the software, Janči tuned out. He didn't need such hand-holding. He accessed OpenSSH, tried connecting ...
... and failed.
Meanwhile, everyone around him was logging in no problem.
Janči's face burned with embarrassment at this second WTF. His first instinct was to blame the deprecated cryptography of the server. He spent most of the remaining lecture time searching for a way to allow his SSH to use SSH-DSS. (It turned out to be supported the whole time, despite the warnings he received.)
Janči then tried to re-download the "private" key and adjust the SSH config file several times. He cycled through different possible usernames associated with his university email account.
No dice.
He was the only person in the class who hadn't yet logged into the server. Not even the professor was able to help him, since he was using Linux.
Embarrassment and frustration mounted. An hour later, out of ideas, Janči fell back to downloading MobaXterm and running it inside Wine.
It didn't work.
The professor offered him a spare Windows box. "Here, try this one."
Janči booted it up, copied the "private" key to the new machine ... and still couldn't sign in.
Now, this was getting suspicious.
The lecture ended. A friend of Janči's hung back while the rest of the students filed out. "Why don't you try logging in with my credentials instead of yours?" she asked.
Janči was up for anything at that point.
It worked. On his own machine, on the Windows box, everywhere.
With that lead in mind, Janči opened the server's /etc/passwd file to look at all the usernames. He noticed that, unlike everyone else, his username and email address didn't match.
His university used Microsoft emails. Everyone had several address aliases, and they could also use whatever email address they liked in the system, even a personal one.
Janči had chosen to use a school email in the form of <number>@uni.uni. Unfortunately, the Ubuntu server didn't like the idea of user being named just <number>, so it had renamed it to user<number>. Some script for generating SSH configuration had probably failed from there, because Janči also discovered that his user home directory was missing a .ssh directory and known_hosts file.
Unfortunately, due to restricted access, he wasn't able to copy them from any of his classmates. In the end, he could connect to the server as any of his classmates, but not as himself.
[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.CodeSOD: Over and Under Reaction
Today's anonymous submitter sends us two blocks. The first is a perfectly normal line of React code:
const [width, setWidth] = useState(false)This creates a width variable, defaulting it to false, and a setWidth function, which lets React detect when you change the variable, and trigger a re-render. Importantly, this mutation only happens on the next render, which means if you call setWidth and then check width, you won't see your change happen.
As I said, this is perfectly normal React code. Well, almost. First, I have to ask: why on Earth is width being set to a boolean value? "How wide are you?" "Yes." It's possible that there's a good reason for this, though I suspect that it's unlikely.
The second issue, however, is that the linter complained that the setter was never actually used. That was odd, because if our submitter grepped the codebase, there were two calls to setWidth. Let's see what that looked like:
const show = (show) => { setWidth(show) setWidth(!show) }We create a function show, where we expect a boolean value, and then we setWidth with that value, and then with the negation of that value. So show(true) will set width to be false. To make matters more confusing, we set width both ways, and I assume this is someone trying to get around React's state management. React won't trigger a re-render if you set the state to a value it already has. So I suspect they're twiddling to try and force it to re-render, and I also suspect that this might not work? Even if it does, this isn't how you should be using React. As I said, I'm no React expert, but as the saying goes: "I don't have to be a helicopter pilot to know that when I see a helicopter hanging upside down from a tree someone messed up."
Our submitter writes:
Got hired to cleanup a mission critical website for a company that had just learned that offshore teams might not be worth the cost saving measures.
"Pay me now or pay me later."
Representative Line: Underscore Its Unimportance
Frequent submitter Argle (previously), sends us a short little representative line. The good news is that this line of code came across Argle's screen during a code review: it was being removed. The bad news is that it was sitting in the code base for ages.
_ = len / 8.0f;Argle writes:
In a code review today. A co-worker wisely removed the line. Dunno the logic that made anyone write it in the first place.
This is C#, though it could be basically any language. Using _ is one of those little conventions that we use to tell the linter to ignore the fact that this variable isn't used. And this variable was not being used. Of course, in addition to being unused, it's also a puzzle: where does the 8.0f come from? No one knows. Why would we even want the length divided by eight? No one knows. There's nothing about this code that gives any indication that it was a meaningful operation at any point.
No one knows what it does, or why it was there in the first place, but someone put the time into making sure the linter didn't complain about its uselessness by using _ as the variable.
Representative Line: A Solid Reference
Today's anonymous submitter works for a large company. It's one of those sorts of companies which has piles, and piles, and piles of paperwork and bureaucracy. It also means that much of their portfolio of software is basic CRUD applications. "Here's a database for managing invoices." "Here's a database for managing desk assignments." "Here's a pile of databases which link our legacy applications to our new ERP system."
Which brings us to our representative line. It is not a representative line of code, but a representative line of the design specification. This is the design specification for yet another database-driven application.
7.7 REFERENTIAL INTEGRITY CONSTRAINTS Referential integrity constraints are not applicable for [REDACTED] Application.Upon seeing this, our submitter predicted that they'd be having a lot of TDWTF submissions in their future.
The worst part? This isn't the only time this has been included in the design spec. Several database driven applications have had this line in their spec. No one is able to explain exactly why referential integrity constraints are not applicable. At best, there are a few batch jobs that don't define a schema themselves, though they need to comply with it. Maybe someone is just copying and pasting from an old design spec and hoping no one notices or cares?
Good news: it's likely that no one will notice, or care. At least not until something breaks in production.
Error'd: Null Null Null
The single most common category of entries for this column is failed handling of NaN, null and undefined. Almost exclusively from javascript in web pages, sometimes in node servers, and almost never any other languages or frameworks. They're getting a bit repetitive but it's our solemn duty to call out failure where we find it. So if you send us one of these, make sure it identifies the source!
"If you want something you've never had, do something you've never done" exhorted Ben.
"Dashed Hope for Jennifer Null," titled an entry from some guy[sic]. "As recently linked from TDWTF article "Not for Nullthing", not only names can break computer systems, but also article content." Stretching, but we'll allow it.
"Where does Batman go on holiday?" asked Morgan. "Nananananana... Nowhere!"
"UBER is ready for driverless vehicles..." Bruce C. "Uber is getting so big, they can't even keep track of their driver's names."
"Well at least the reason wasn't null or NaN," wrote Steve W. regarding CenturyLink. "I've been trying for weeks to use their web page to change my (incorrect billing address). Such progress."
Additional entries on the topic from
Dan
: "we're fresh out of null"
Henrik
: "What is this null music streaming service"
Mike
: "Name: undefined"
Laks
: "In this app, every new user defaults to a nullptr."
and
Jim
: "Think I'll buy $NaCar with this refund!"
and many others were all appreciated and noted.
CodeSOD: Failing to Fail
Russell F (previously) sends us a small one today. It's not just a representative line, it's a representative comment. More than that, it's a true confession. Russell wrote some code, you see, and the logic was confusing. So, a co-worker added a comment to explain what the code was doing:
'This is *supposed* to fail. If it fails to fail, it throws a failure messageRussell writes:
I have to confess that this one is my fault. The comment was added by one of my coworkers to clarify what I was doing, and made me realize how stupid I'd been.
"Failing to plan is planning to fail" becomes "failing to fail is failure message".
CodeSOD: Please Find, Rewind
As previously discussed, C++ took a surprisingly long time to get a "starts with" function for strings. It took even longer to get a function called "contains". In part, that's simply because string::find solves that problem.
Nancy sends us a… different approach to solving this problem.
bool substringInString(string str, string::iterator &it) { string tmp; bool result = false; int size = str.length(); int count = 0; while (count < size) { tmp += *it; it++; count++; if (tmp.find(str) != string::npos) { result = true; it -= size; break; } } if ( !result) { it -= size; } return result; }This function iterates across a string, character by character. In this iteration, we copy one character at a time into tmp. Then we see if tmp contains our search str. If it does, we break out of the loop after rewinding the iterator. Outside of the loop, we check if we found the substring, and if we did, we rewind the iterator. Then we return true or false based on whether on not we found the substring.
So wait a second. str is our search string. it is where we're searching. And we copy from it up to our search string's length into a temporary string. We then do a find in that temporary string- hey! This is just a startsWith check written in the most insane way possible.
Why even bother with the while loop? While tmp is shorter than the search string, the answer is always "no, we haven't found it". And the developers knew that- that's why they always rewind size characters on the iterator. They're always searching exactly that many characters. Of course, since we always rewind the same amount, we can also just move the it -= size statement out of the loop and out of the if statement and do it once.
Nancy calls this "a little gem" in a "large codebase". Yeah, a real gem.
[Advertisement] Plan Your .NET 9 Migration with ConfidenceYour journey to .NET 9 is more than just one decision.Avoid migration migraines with the advice in this free guide. Download Free Guide Now!
CodeSOD: Not for Nullthing
Today's anonymous submitter sends us some code that just makes your mind go… blank when you look at it.
public static boolean isNull(String value) { return StringUtils.isBlank(value); }StringUtils.isBlank comes from the Apache Commons library. It's a helper function for Java which returns true if a string is, well, blank. "Blank" in this case is: empty, null, or only whitespace. So it's important to note that isBlank may return true on a null, but it isn't truly a null-check, so wrapping it in isNull is just confusing.
But imagine I've got another problem. Let's say I have a database that's been poorly normalized and maintained. And so I have a bunch of fields that maybe are null, but some also maybe contain the string "null". What am I going to do then? I need another function.
public static boolean isNullAndNull(String value) { return isNull(value) && "null".equalsIgnoreCase(value); }Ah yes, isNullAndNull, the clearest and easiest name I could imagine for this. It tells me exactly what the function is checking: is it null, and is it also null? We add a second check to our isNull call- we check if the input value matches the string "null". Except we're &&ing the conditions together. So this function will always return false. It can't both be blank and contain the string "null".
Which means Jennifer Null, who is a real person, can breathe easy. This version of a null check won't think she's nothing.
[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.Empty Pockets
If you've seen one developer recounting how their AI agent deleted production, you've seen them all. They're mostly not interesting stories. It's like watching someone speeding through traffic on a motorcycle without a helmet: the eventual tragedy is sad, but it's unsurprising and not an interesting story to tell. It's not even interesting as a warning: the kind of person who speeds on a motorcycle without a helmet isn't doing so because they don't understand the danger. They've just decided it doesn't apply to them.
But the founder of PocketOS, Jer, recently shared how- whoopsie!- their AI agent deleted production. There's a lot of ingredients that go into this particular disaster, which I think makes it interesting, because the use of a poorly supervised AI agent is only one ingredient in this absolute trainwreck of a story.
PocketOS is a small company that makes software for rental companies to manage reservations. Car rentals are a big customer, but the tool is more general than that. They manage all of their infrastructure via a service called Railway. Railway is a pretty-looking GUI tool for automating your deployments and the target environments.
PocketOS also is heavily adopting Cursor wrapping around the Claude model. They've paid big bucks for the top-end model offered. Many of their components, like Railway, offer MCP services so that their LLM can do useful things. They're using the Claude LLM to automate as much as they can.
So far, this is all a pretty typical setup. They pointed Claude at their code and gave it a "routine" task, and sent it to work. It toddled through the problem and encountered a credential issue. It "decided" that the fix for this issue was to delete a storage volume and recreate it. It scanned through the code to find a file containing an API key, found it, and then sent a POST request via cURL to delete the volume in question.
Jer writes:
To execute the deletion, the agent went looking for an API token. It found one in a file completely unrelated to the task it was working on. That token had been created for one purpose: to add and remove custom domains via the Railway CLI for our services. We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete. Had we known a CLI token created for routine domain operations could also delete production volumes, we would never have stored it.
Wait, the tokens you create in Railway all have god-level privileges? That sounds like a terrible idea. And you were storing the token in your code? We'll come back to this in a moment, but sure, this is bad, but you can just restore from backup, right?
The volume was deleted. Because Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it. Our most recent recoverable backup was three months old.
Oh. Oh no.
Now, I don't think it's literally true that Railway is storing your backups literally in the same volume as the thing they're backing up. I certainly hope not. But they do apparently delete your backups when you delete the volume associated with them. Which is a choice, certainly. A bad one. And one that they documented, according to Jer. It was, in his words, "buried" in the docs.
But let's go back to the tokens for a moment. I am not a Railway user, but I checked out the tool and went through the process of creating a project token. And while no, Railway does not give you big red flags warning you "Hey, this token can do ABSOLUTELY ANYTHING", it also never gives you an opportunity to scope the token. Which, I don't know about you, but the first thing I do when I create an authentication entity is try and figure out how to control its authorizations, because I assume at the start it doesn't have any. That'd be sane.
The scoping happens when you create the token, depending on what context you're in when you do it. It's only a handful of scopes, and no fine grained permissions on API keys at all. The lowest level is "Project" which can do anything to a single environment- which does mean that even if you, like Jer's team, wanted to have a script that changed some DNS settings in production, that same key could be used to delete volumes in production. Which means you really really want to take care of that key, and you certainly don't want to leave it where some junior developer or bumbling AI agent can find it.
Jer also complains that Railway shouldn't allow an API call to take destructive actions without more protections, like forcing someone to type in the name of the thing being deleted or sending a confirmation email, or something. This, I'm more skeptical of. Most cloud providers don't offer anything like this in their APIs, at least that I've seen, because on a certain level, if you're invoking the API with the proper credentials, that's a big enough hill to climb that we can assume you've intended your action. The correct way to protect against this is properly scoped keys and keeping those keys secure and not just lying around in plain text. There's a certain aspect of understanding that you're using a potentially dangerous tool and need to take the responsibility for safety into your own hands; while a table saw can easily take some fingers off, it's perfectly safe when used correctly.
This is all bad, but how can we make it worse? Well, Jer demanded that Claude "explain itself". In a section called "The Agent's Confession", Jer highlights that the agent is able to identify the explict rules that it failed to follow.
Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing.
No, it is not the agent on record. I see this kind of thing a lot when people talk about LLMs. An LLM cannot explain its reasoning. It cannot go on "the record". It cannot confess to anything. While what it plops out when asked might be interesting, it is not an explanation. The only explanation is that it's a powerful statistical model trying to create a plausible string of tokens! It's simply looking at its context window and your prompt and trying to predict what it should say. It can tell you what rules it violated not because it understands the rules or knows it violated any rules, but because those rules are in its context window. If you ask it right, it'll confess to killing JFK and framing Oswald for the crime.
Jer then tries to ensure that Cursor takes some of the blame, pointing to Cursor's "guardrails" documentation. Except, here, the documentation is actually quite explicit about what those guardrails guarantee. If you're using a first-party tool, it will prohibit unsafe operations. When using 3rd party MCPs, like Railway's, the only guardrail is that it requires human approval for every action- unless you update your allowlist for that MCP. If you put them in your allowlist, the guardrails go away. Jer argues that tools should enforce more protection against LLM behaviors, but the problem with that is people- like the PocketOS team- turn those protections off. And like a lot of safety mistakes, they can get away with it all the way up until the point where they can't.
Jer follows this by listing off a pile of other times using Cursor has caused disasters, which isn't making the argument he thinks it is: yes, Cursor is dangerous, but those dangers are well known. It makes the choice to turn Cursor loose without strict supervision seem even more foolish.
Jer writes:
For now I want this incident understood on its own terms: as a Cursor failure, a Railway failure, and a backup-architecture failure that all happened to one company in one Friday afternoon.
It's also a PocketOS failure. It's a failure to properly assess the tools and environments you chose to use for your product. A failure to read and understand the docs for vital features, like *backups*. A failure to employ even the most basic safeguards. A failure to put a second's thought into key management- even if that key was only for DNS entries, you still shouldn't chuck it in source control. A failure to have a competent backup strategy. It's worth noting that they did restore from a three month old backup, which means they were at one point taking backups outside of Railway's volume setup. That was a wise decision. That they stopped is a failure.
The first rule of disaster retrospectives is that it's never one piece that's the failure. It's never one person's fault, one tool's fault, one vendor's fault. It's a systemic failure. Railway's keys should be finer grained. But also, you shouldn't leave keys lying around. Deleting backups when you delete the volume is a terrible idea, but having only one service for backups (that's also your primary site) is a terrible idea. Claude's ability to enforce its own guardrails should be better, but LLMs are notoriously dangerous about this: you should know better, and by your own words you did.
This is not an anti-AI post, or even a "get a load of this asshole" post. It is a "understand the damn tools you're using" post. Be critical of them. Don't trust them. Ever. Especially LLMs, because the worst part of an LLM is that it takes away the one thing computers used to be good at: predictable, deterministic behavior. But not just LLMs: don't trust your cloud provider, don't trust your infrastructure manager. Dig into them and understand how they work, and if they seem to complicated to understand, than they may be too complicated to trust.
Update: As pointed out in the featured comment below, Railway did finally get a backup restored. So they got their data back. Yay? From the post, Jer remains committed to making this a Railway issue and not a PocketOS issue.
[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.Error'd: Parametric Projection
"AWS Spellcheck Fail!" alerts Peter "If only someone at AWS knew the correct paramters to activate the spellcheck."
"How long is too long for a job to be open? " wonders Lincoln K. "I didn't even know LinkedIn existed 61 years ago, let alone was accepting postings... Though only 81 applicants in that time is hardly an impressive turn-out." For a "Vice President Operations and Quality Control", no less.
An anonymous Richard reports "This came through my door. On a card that, in order to get to my door, had my full address printed on it, including my ."
Oenophile Abroad Michael R. shares "My Macbook broke after being "exposed" to red wine. As a German in London it pleases me so see that the repair shop offers this time granularity."
CodeSOD: Cancel Catch
"This WTF is in Matlab" almost feels like cheating. At one place I worked, somebody's job was struggling through a mountain of Matlab code and porting it into C. "This Matlab code looks like it was written by an alien," also doesn't really get much traction- all Matlab code looks like it was written by an alien. This falls into the realm of "Researchers use Matlab, researchers may be very smart about their domain, but generally don't know the first thing about writing maintainable code, because that's not their job."
But let's take a look at some MatLab Carl W found:
try if (~isempty(fieldnames(bigStruct)) && isfield(bigStruct,'pathName')) [FileName, PathName] = uigetfile(bigStruct.pathName); else [FileName, PathName] = uigetfile(lastPath); %lastPath holds previous path end catch bigStruct = struct; endThe uigetfile function opens a file dialog box. When the user selects a file, FileName holds the filename, PathName holds the containing path. If the user doesn't select a valid file, or clicks "Cancel", both of those variables get set to 0. It's then up to the caller to check the return value and decide what happens next.
Which is not what happens here, obviously. The developer responsible seems to believe that it maybe throws an exception? And they can just catch it? Carl's best guess is that this is a "weird" way to catch the cancel button. But it does mean that FileName and PathName get set to 0, and those zeros propagate until something finally tries to open those files, at which point everything blows up and the user doesn't know why.
[Advertisement] Plan Your .NET 9 Migration with ConfidenceYour journey to .NET 9 is more than just one decision.Avoid migration migraines with the advice in this free guide. Download Free Guide Now!
A Whale of a Problem
From our Anonymous submitter:
Our company creates graphs to visualize data. We have many small fish customers, but we have one whale who uses our product that is 90% of company revenue. (WTF number 1.)
So if he is not happy, it's all-hands-on deck-mode.
He complained that our APIs and charts are loading slowly for him. For 3 weeks, we've tried a TON of optimizations, including WTF 2: spinning up a special server he alone can hit.
Today, we found out that he's always complaining when he's in his car, driving from home to the office. But since he "totally has the best wifi money can buy," that isn't worth investigating.
WTF 3: thinking wifi and data are always 100% reliable in a car driving around.
Our submitter highlights one of the major pitfalls of the so-called whale client: if they're a bad client, you're in for an extra-bad time.
As I lean harder into freelancing, I'm learning to scan the waters ahead of me for potential whales. My goal is to build up multiple small, diverse income streams, because I've had my own dangerous encounters with whales in the past.
At one employer of mine, there was Facebook, who acted as if they were our new owners rather than a new customer. They'd already produced flashy marketing videos of the sorts of solutions they planned to implement with our software, showing people delighted with the results. In meetings, these things were talked up as amazing game-changers. Meanwhile, I found all the things Facebook wanted to do horribly creepy and invasive.
Even worse, Facebook began dictating how our award-winning technical support should change to accommodate their whims, up to and including having a dedicated toady—er, support rep—who did nothing but field Facebook-related tickets, similar to a technical account manager (TAM).
That was the last straw for me. I left that company before I was forced to deal with any of Facebook's crap.
My second whale sighting occurred at a startup that'd landed Porsche, far and away their biggest client ever. All of a sudden, our timeline for adding new features and fixing bugs became Porsche's honey-do list. All of a sudden, the platform frequently crashed and became unusable for everyone because it couldn't handle the amount of traffic Porsche (and their clients) hurled at it.
On the other hand, there were several times in that startup's existence when a big wad of promised funding failed to materialize. Porsche kept the business afloat and literally kept my lights on.
I find it less than ideal to be at any company's mercy. I want a world that would neither spawn whales nor millions of startups named Sploink, Dink, and Twangle that promise to bring the power of AI to your dinner fork.
Have your own epic whaling adventures? Share with us in the comments!
CodeSOD: Lint Brush Off
A few years back, C# added the concept of "primary constructors". Instead of declaring the storage for class members and then initializing them in the constructor, you can annotate the class itself with the required fields, and C# automatically generates a constructor for you. It's all very TypeScript and very Microsoft, and certainly cuts down on some boilerplate.
Esben B's team isn't really using them in many places, but they are using a linter which is opinionated about them. So this in-line constructor causes the linter to complain:
public DocumentNetworkController(ILookupClient service)The linter wants you to switch this to a primary constructor. Esben didn't want to do that, and didn't want to change the global linter configuration, and so added a pragma to disable that particular warning:
#pragma warning disable IDE0290 // Use primary constructor public DocumentNetworkController(ILookupClient service) #pragma warning restore IDE0290The linter didn't like this. It threw a new warning: that this suppression wasn't needed. Which was news to Esben, as clearly the suppression was needed if you wanted to make the warnings go away. The obvious solution was to disable the warning that you didn't need to disable the warning:
#pragma warning disable IDE0079, IDE0290 // Use primary constructor public DocumentNetworkController(ILookupClient service) #pragma warning restore IDE0290, IDE0079Except this doesn't work. These pragmas take effect on the next line, which means you can't disable IDE0079 on the same line as IDE0290 and expect it to work. Which means the final version of the code looked like this:
#pragma warning disable IDE0079 // Disable warning about not needed supression #pragma warning disable IDE0290 // Use primary constructor public DocumentNetworkController(ILookupClient service) #pragma warning restore IDE0290, IDE0079Esben writes:
So the nice recommendation to use a primary ctor ended up with 3 lines of annoying boilerplate code. Good times \o/
While yes, this is frustrating, I will say there's an element of "when the table saw keeps taking fingers off, that may be more of a you problem." I don't know the details, so I can't say, "just change the linter config or adopt its recommendation" and claim that the problem goes away, but when the tool hurts you, it's a definite sign of one of two things: it's either the wrong tool, or you're using it wrong.
CodeSOD: The JSON Template
We rip on PHP a lot, but I am willing to admit that the language and ecosystem have evolved over the years. What started as an ugly templating language is now just an ugly regular language.
But what happens when you still really want to do things with templates? Allison has inherited a Python-based, WSGI application which rejects any sort of formal routing or basic web development best practices. Their way of routing requests is simply long chains of "if condition then invokeA elif otherCondition then invokeB". Sometimes, those conditions will directly set the MIME type on the HTTP response.
They do use a templating library called Mako for generating their responses. They use it for their HTML responses, obviously. They also use it for their JSON responses, generating code like this:
{ "success": true, "items": { %for item in items_available.keys(): "${item}": ${items_available[item]}${',' if not loop.last else ''} %endfor } }The %for and matching %endfor mark the Python code off, which generates JSON via string-munging, complete with the check to make sure we're not on the last iteration of the loop.
Like so much bad code, this offers a degree of fractal wrongness. Instead of iterating over the keys and fetching the items inside the loop, you could iterate for key,value in items_available.items()- and according to the Mako docs, that for is just a regular Python for loop. That we're just outputting the contents of the dictionary is itself potentially a problem- sure, if we know the types of the dictionary, we'll know that whatever it is can be output in the body of a JSON document, but do we really think this code is using type annotations? I don't. And for a RESTful web service, I'm always going to feel weird about using a success field when ideally the HTTP status code could convey most of that information (and yes, I know there are reasons to still put status in the body, I just hate it).
Of course, the real issue is just: Python's built in JSON serialization is actually pretty advanced. And performant! You don't need any of this, you could just do something like:
return json.dumps({"success": true, "items": items_available})No templates. No formatting. No worries about how the data gets represented. Well, still worries, because JSON serialier will throw exceptions if it doesn't know what to do with a type. But then at least you get that exception on the server side and aren't sending the client a malformed document.
In any case, this is a good demonstration that you can write bad PHP in any language.