Like ghosts, security vulnerabilities are the result of us going about our lives, doing the best we can, and then experiencing things going awry just because that’s usually what happens. We plan systems, we build software, we try to create the best teams we can, and yet there will always be echoes in the world from our suboptimal choices. They can come from lots of places: one team member’s work not being double checked when a large decision is at stake, a committee losing the thread, or – worst of all – absolutely nothing out of the ordinary. Alas, it’s only true: security issues are a natural side effect of creating and using technology.
In this post (and the talk it accompanies; video to come when it’s up), we’re going to talk about the energy signatures, cryptids, orbs, poltergeists, strange sounds, code smells, and other things that go bump in the night. That’s right: we’re talking about security horror stories.
Before we get started, here’s a little disclaimer. Everything I’m about to talk about it is something I’ve encountered, but I’ve put a light veil of fiction on everything. It’s just professionalism and good manners, a dash of fear of NDAs, and a little superstition too. The universe likes to punish arrogance, so I’m not pretending I’m immune to any of this. No one is – it’s why appsec exists!
Now: let’s get to the haunts.
Automatic Updates
I’m starting with one that’s probably fresh in everyone’s minds, and I’m mentioning it first because it’s such a dramatic betrayal. You think you’re doing the right thing. You’re doing the thing your security team TOLD you to do by keeping your software up to date. And then suddenly, you hear a familiar name in the news, and you have an incident on your hands.
This one is like getting stabbed with your own stake when you’re fending off a vampire; I thought we had a good thing going!
Ideally, you’ll learn about it via responsible disclosure from the company. Realistically, it might be CNN. Such is the way of supply chain attacks.
You invite it in by doing the right thing. Using components, tools, and libraries with known vulnerabilities is in the most current version of the OWASP top ten for a reason, so there’s endless incentive to keep your stuff up to date. The problem comes from the tricky balance of updates vs. stability: the old ops vs. security argument.
A couple of jobs ago, we were suddenly having trouble after our hourly server update. Ansible was behaving itself, but a couple of our server types weren’t working right. After an intense afternoon, some of my coworkers narrowed it down to a dependency that had been updated after a long time with no new releases. It wasn’t even what you’d call a load-bearing library, but it was important enough that there’d been a breaking change. The problem was solved by pinning the requirement to an earlier version and reverting the update.
My own sad little contribution toward this not being the state of things forever was to make a ticket in our deep, frozen, iceboxy backlog saying to revisit it in six months. I was an SRE then, but I was leaning toward security, and I was a little perturbed by the resolution – though I couldn’t have suggested a better solution for keeping business going that day.
(It did get fixed later, by the way. My coworkers reached out to tell me, which was very kind of them.)
This has become one of those stories that stays with me when I review code or a web UI and find that an old version of server software or a dependency has a known issue. Even if it doesn’t have a documented vulnerability, not throwing the parking brake on until it’s updated to the most recent major version feels like doing badly by our future selves, even if it’s what makes sense that day.
The best way to fix this is to pay attention to changes, check hashes on new packages, and generally pay close attention to what’s flowing into your environment. It isn’t easy, but it’s what we’ve got.
Chrome extensions and other free software
The horrors that lurk in free software are akin to the risks of bringing a Ouija board in the house that you found on the street
Behold: my favorite piece of art I made for this talk/post.
You can identify it by listening for you or someone around you saying, “Look at this cool free thing! Thanks, Captain Howdy, I feel so much more efficient now!” Question everything that’s free, especially if it executes on your own computer. That’s how we invite it in: sometimes it’s really hard to resist something useful and free, even if you know better.
A particular problem with Chrome extensions is that you can’t publish them without automatic updates enabled, so someone has a direct line to change code in your browser, so long as you keep it installed.
Last fall, The Great Suspender, a popular extension for suspending tabs and reducing Chrome’s memory usage, was taken over by malicious maintainers. As a result, people who had, once upon a time, done their due diligence were still sometimes unpleasantly surprised.
That’s the tough thing about evaluating some of these: you have to consider both the current risks (permissions used, things lurking in code) and the possible future risks. What happens if this program installed on 500 computers with access to your company’s shared drive goes rogue? It makes it difficult to be something other than that stereotype of security, the endless purveyors of NO. But a small risk across a big enough attack surface ends up being a much larger risk.
In short: it’s the Facebook principle, where if you get a service for free, you might consider the possibility that you’re the product. Security isn’t necessarily handled as carefully as it should be for the product rather than paying customers. Pick your conveniences carefully and prune them now and then. (Or get fancy, if you control it within your company, and consider an allowlist model rather than a blocklist. Make conveniences prove themselves before you bring them in the house.)
unsafe-eval and unsafe-inline in CSP
Our next monster: an overly permissive content security policy. I’d compare it to a particular episode of Buffy the Vampire Slayer or any movie that uses that trope of people reading an old book out loud without understanding what they’re saying.
Fortunately, a content security policy is a lot easier to read than inspecting every old leather book someone might drag into your house
We invite it in because sometimes,the risky choice just seems easier. You just need to run a little code from a CDN that you need to be able to update easily. You know.
For fending it off, I would gently urge you not to put anything in your CSP that literally contains the word “unsafe.” I honestly understand that there are workarounds that can make sense in the moment, when you’re dealing with a tricky problem, when you just need a little flexibility to make the thing work.
In that case, I urge you to follow my quarantine problem-solving process, for moments where you need to figure something out or finish a task, but the brain won’t cooperate.
- Have you eaten recently? (If not, fix that.)
- Does this just feel impossible right now? Set a timer, work on it for a bit, then stop.
- Can you not think around this problem usefully? Write out your questions.
I would suggest starting with “why do I think unsafe-eval is the best option right now, and what might I google to persuade myself otherwise?”
Sometimes you want to keep things flexible: “I’ll just allow scripts from this wide-open CDN URL wildcard pattern.” But what can this enable? What if the CDN gets compromised? Use partners you trust, sure, but it’s also a good idea to have a failsafe in place, rather than having something in place that allows any old code from a certain subdomain.
Look, it’s hard to make things work. You have to be a glutton for punishment to do this work, even if you love it. (And I usually do.) But you can’t just say yes to everything because of future-proofing or whatever feels like a good reason today, influenced by your last month or so of work pains. You can’t do it. I’m telling you, as an internet professional, not to do it. Because your work might go through my team, and I will say NOT TODAY, INTERNET SATAN, and then you’re back at square one anyway.
Stacktraces that tell on you
“The demon is a liar. He will lie to confuse us; but he will also mix lies with the truth to attack us.”
William Peter Blatty, eminent appsec engineer and author of The Exorcist
Logs and stacktraces contain the truth. Right? They’re there to collect context and provide you information to make things better. Logs: they’re great! Except…
Exception in thread "oh_no" java.lang.RuntimeException: AHHHHHH!!!
at com.theentirecompany.module.MyProject.superProprietaryMethod(MyActualLivelihood.java:50)
at com.theentirecompany.module.MyProject.catOutOfBagMethod(MyActualLivelihood.java:34)
at com.theentirecompany.module.MyProject.underNDAMethod(MyActualLivelihood.java:27)
at com.theentirecompany.module.MyProject.sensitiveSecretMethod(MyActualLivelihood.java:11)
at com.theentirecompany.module.MyProject.oh_no(MyActualLivelihood.java:6)…except for when they get out of their cage. Or when they contain information that can be used to hurt you or your users.
We invite it in by adding values to debug statements that don’t need to be there. Or maybe by writing endpoints so that errors might spill big old stacktraces that tell on you. Maybe you space and leave debugging mode on anywhere once you’ve deployed. Or you just haven’t read up OWASP’s cheat sheet on security misconfiguration.
How to fend it off: conjure a good culturally shared sense of safe log construction and remember what shouldn’t be in logs or errors:
- Secrets
- PII
- PHI
- Anything that could hurt your users
- Anything that could get you sued.
Make a commit hook that checks for debug anything.
Secrets in code
“To speak the name is to control the thing.”
Ursula K. Le Guin
The monster I’d compare this to is when the fae (or a wizard, depending on your taste in fantasy) has your true name and can control you.
API_KEY = "86fbd8bf-deadbeef-ae69-01b26ddb4b22"How to identify it: read your code. Is there something in there you wouldn’t want any internet random to see? Yank it! Use less-sensitive identifiers like internal account numbers if you need to, just not the thing that lets someone pretend to be you of one of your users if they have it.
You know you’re summoning this one if you find yourself saying, “Oh, it’s a private repo, it’ll never matter.” Or maybe, “It’s a key that’s not a big deal, it’s fine if it’s out there.”
We fend this one off with safe secret storage and not depending on GitHub as a critical layer of security.
We all want to do it. Sometimes, it’s way easier than environment variables or figuring out a legit secret storage system. Who wants to spend time caring for and feeding a Vault cluster? Come on, it’s just on our servers, it’s not a big deal. It’s only deployed within our private network.
It is a big deal. It is my job to tell people not to do this. Today I will tell you for free: don’t do this!
I’ve argued with people about this one sometimes. It surprises me when it happens, but the thing is that people really just want to get their job done, which can mean the temptation of doing things in the way that initially seems simpler. As a security professional, it becomes your job to help them understand that saving time now can create so any time-consuming problems later.
It’s great to have a commit check that looks for secrets, but even better is never ever doing that. Best of all is both!
A single layer of input validation
Our next monster: a single layer of input validation for your web UI. I quote Zombieland: always double-tap.
Our comparison for this one is anything tenacious. Let’s say zombies, vampires, hydras. Anything that travels in a pack. And we identify it by siccing Burp Suite on it. Maybe the form doesn’t let you put a li’l <script> tag in, but the HTTP request might be only too happy to oblige. We invite it in by getting a little complacent.
The best way to fend it off is to remember that regular users aren’t your real threat (and you’re probably just irritating people with names that don’t meet the definition of “normal” some shoddier validation will catch, which can do some awful racist things). There’s a reason injection, especially SQL injection, is a perennial member of the OWASP top ten.
Say it with me: client-side and server-side validation and sanitation! And then add appropriate encoding too, depending on what you’re doing with this input.
Most people do only interact with your server via your front end, and bless them. But the world is filled with jerks like me, both professional jerks and people who are jerks for fun, and we will bypass your front end to send requests directly to your server. Never take only one measure when you can take two.
Your big-mouthed server
curl -I big-mouthed-server.comHow to identify this one? curl -I thyself
We invite this one in by not changing defaults, which aren’t always helpful. This also relates to security misconfiguration from the OWASP top ten. The early internet sometimes defaulted too much toward trust (read The Cuckoo’s Egg by Cliff Stoll for a great demonstration of the practices and perils of this mindset), and we can still see this in defaults for some software configs.
Protect yourself from this one by changing your server defaults so your server isn’t saying anything you wouldn’t want to broadcast. The primary concern here is about telling someone you’re using a vulnerable version of your server software, so keep your stuff updated and muffle your server.
Let’s go X-Files about it: trust no one, including your computers.
ESPECIALLY your computers.
Don’t trust computers! They just puke your details out everywhere if someone sends them a special combination of a few characters.
In conclusion, sometimes I wish I had been born in the neolithic.
Laissez-faire auth (or yolosec, if you prefer)
Our next monster: authentication that accepts any old key and authorization that assumes that, if you’re logged in, you’re good to go. Its horror movie counterpart is that member of your zombie-hunting group that cares more about being a chill dude than being alive.
You can identify it by doing a little token swapping. Can you complete operations for one user using another’s token? The monster is in your house.
Here we are again at the OWASP top ten: broken authentication is a common and very serious issue. The problem compounds because we need to do better than “Are you allowed in?” We also need to ask, “Are you allowed to do THIS operation?”
We invite this one in by assuming no one will check to see if this is an issue or by assuming the only people who interact with your software have a user mindset. Or, sometimes worst of all, just not giving your devs enough time and resources to do this right.
We can fend it off by using a trusted authentication solution (OAuth is popular for a reason) and ensuring there are permissions checks, especially on state-changing operations – ones that go beyond “if you’re not an admin, you don’t have certain links in your view of things.”
I’ve seen a fair amount of “any token will do” stuff: tokens that don’t invalidate on logout, tokens that work if you fudge a few characters, things like that. It’s like giving a red paper square as a movie ticket: anyone can come in anytime. Our systems and users need us to do better.
The elder gods of technology
Ah, yes, the technology that will never leave us. Think ancient WordPress, old versions of Windows running on an untold number of servers, a certain outdated version of Gunicorn, and other software with published CVEs.
“That is not dead which can eternal lie.”
Which monster would I compare this to? Well, I’m not going to say his name, am I?
You know you’re at risk of summoning it when you hear someone say “legacy software,” though newer projects are NOT immune to this. Saying “we’re really focusing our budget on new features right now” is another great way to find… you know… on your doorstep.
We can fend it off by allocating budgets to review and update dependencies and fix problems when they’re found. And they should be sought out on a regular schedule.
No tool, framework, or language is bad. They all have potential issues that need to be considered when. For instance, there are still valid reasons to use XML, and PHP is a legitimate language that just needs to be lovingly tended.
Yep, we’re back at the risk of using components with known vulnerabilities from the OWASP top ten. No tool, framework, or language is bad, but some versions have known problems, and some tools have common issues if you don’t work around them. It’s not on them; it’s on you and how you use and tend your tools.
The real incantation to keep this one away is understanding why we keep these things around. No engineering team keeps their resident technical debt nightmare because they like it. They do it because it’s worked so far, and rewriting things is expensive and finicky, particularly if outside entities depend on your software or services.
I’ve never been on an engineering team that didn’t have lots of other things to do rather than address the thing that mostly hasn’t given them problems… even if none of the engineers without vivid memories of the 90s understand it very well.
Sometimes the risk of breaking changes is scarier than whatever might be waiting down the road, once your old dependencies cause problems… or someone on the outside realizes what your house of cards is built on.
Javascript :D
Speaking of no tool, framework, or language being bad: let’s talk about Javascript. Specifically Javascript used incorrectly.
Our horror comparison? It’s a little 12 Monkeys, a little Cabin Fever. Anything where they realize the contagion is in the water or has gone airborne. “Oh god, it’s everywhere, it’s unavoidable, it’s a… global… pandemic…” Oh no.
The way to summon this one is simple. Say it with me:
internet
Anything… internet. It’ll be there, whether you think you invited it or not.
To fend it off? Just… be very careful, please.
I’m not going to make fun of Javascript. Once I was a child and enjoyed childish things, like making fun of Javascript. Ok, that was like three years ago, but now I am an adult and have quit my childish ways. In fact, I made myself do it somewhere between when I realized that mocking other people’s tech is not actually an interesting contribution to a conversation and when I became an appsec engineer, so roughly between 2016 and the end of 2019.
The internet and thus life and commerce as we know it runs largely on Javascript and its various frameworks. It’s just there! And there have been lots of leaps forward to make it less nightmarish, thanks to the really hard work of a lot of people, but still, things like doctor’s appointments and vaccination slots and other giant matters of safety and wellbeing hang out on top of the peculiarities of a programming language that was created in ten days.
Unfortunately, new dragons will just continue to appear, because that’s the side effect of making new things: new features, new problems. The great thing for appsec that’s an unfortunate thing about humanity is that we make the same mistakes over and over, because if something worked once, we want to think we have it sorted. And unfortunately, the internet and technology insist on continuing to evolve.
How to fix it? Invest in appsec.
Sorry. It’s a great and important goal to develop security-conscious software engineers and to talk security as early as possible in the development process. But there’s a reason writers don’t proofread their work – or shouldn’t. Writing and reviewing the work are two different jobs. It’s why healthy places don’t let people approve their own PRs. If we created it, we can’t see it objectively. And most of us become sharper when honed by other perspectives and ideas.
The most permissive of permissions
And finally, the monster that’s nearest and dearest to my heart: overly permissive roles, most specifically AWS permissions. I’d compare this to sleeping unprotected in the woods when you know very well they’re full of threats.
You can identify it by checking out your various IAM role permissions. And yes, this is totally a broken authentication issue too, a la OWASP.
The best way I know to invite this one in is by starting out as a smaller shop and then moving from a team of, say, three people who maybe even all work in the same room, doing everything, to thirty or fifty or more people, who have greater specialization… yet your permissions never got the memo.
And the best way I know to fend it off is to make more refined roles as early as you can. I know, it’s hard! It isn’t a change that enables a feature or earns more money, so it’s easy to ignore. Worst of all, as you refine the roles, reduce access, and iterate, you’re probably going to piss off a bunch of people as you work to get it right. IAM: the monster that doesn’t need a clever analogy because getting it right sucks so bad on its own sometimes!
It’s also the monster that lurks around basically every corner, because IAM is ubiquitous. So it’s everywhere, and it’s endlessly difficult: awesome! Alas, we have to put the work in, because messing it up the most efficient way I know to undermine so much other carefully done security work.
And yet I feel warmly toward it, because IAM was my gateway into security work. It’s the thing I tend to quietly recommend to the security-aspiring, because most people don’t seem to like doing it very much, yet the work always needs to be done. Just showing up and being like, “IAM? I’d love to!” is a highly distinctive professional posture. Get your crucifix ready and have some incantations at hand, and you’ll never run out of things to do. It’s never not useful. Sorry, it’s just like the rest of tech: if you’re willing to do the grunt stuff and get good at it, you’ll probably have work for as long as you want it.
Whew. That’s a lot of things that go bump in the night.
Let’s end with some positive things. I swear there are some. And if a security engineer tells you that there are still some beautiful, sparkly, pure things in the world, that’s probably worth listening to.
Strategies for vulnerability ghost hunting
Don’t be the security jerk
Be easy to work with. This doesn’t mean being in a perpetually good mood – I am NOT, and former coworkers will attest to this if you ask – but it means reliably not killing the messenger.
People outside of your security team – including and especially people who aren’t engineers – are your best resource for knowing what’s actually going on. Here’s the thing: if you’re a security engineer, people don’t act normally around you anymore. You probably don’t get to witness the full spectrum of people’s natural behavior. Unfortunately, it just comes with the territory. And it means we have to rely on people telling us the truth when they see something concerning.
Even people who are cooperative with security will hedge a little sometimes when dealing with us. I know this because I’ve done it. But you’ll reduce this problem if everyone knows that talking to security will be an easy, gracious thing where they’re thanked at the end. Make awards for people who are willing to bring you problems instead of creating a culture of fear and covering them up! Make people feel like they’re your ally and a valued resource, because they are!
Be an effective communicator
Being able to communicate in writing is so important in this work. Whether it’s vulnerability reports, responsible disclosure, blog posts warning others about haunted terrain, or corresponding with people affected by security poltergeists, being able to write clearly for a variety of audiences is one of our best tools.
If you think you’re losing your grip on how regular people talk about this stuff, bribe your best blunt, nontechnical friend to listen to you explain things. Then have that blunt friend tell you when what you said didn’t make a goddamn lick of sense… then revise and explain again. Do this until you’re able to explain things in plain language accessible to your users and spur them to action using motivations that make sense to them.
Now let’s leave this haunted house together and greet the coming dawn.
I hope you encounter none of these terrifying creatures and phenomena that I described, that your trails from server to server in our increasingly connected world are paved with good intentions and mortared together with only the best outcomes. But should you wander into dark woods full of glowing red eyes and skittering sounds belonging to creatures just out of sight… I hope you are better equipped to recognize and banish them than you were earlier. Thank you for reading.