This is the blog version of my BSides SF 2022 talk, “Read the Fantastic Manual: Writing Security Docs People Will Actually Read”. I’ll embed video here when it’s available. If you came here from my conference talk: hello! I’m glad you’re here.
Let’s talk documents. Every company needs them, and security in particular needs to be able to effectively educate at scale. We’re going to look at the ins and outs of creating (or reviving) a document repository that explains security needs to the rest of your company, though most of what’s discussed here is useful for anyone who needs to relay information to anyone else. What I’m describing is best for companies of up to perhaps 4,000 people and not so much your very large companies, which often have comms teams for this kind of work. The strategies here help people doing that work too, but as those companies can often afford a content strategist or two, it’s sometimes less critical for a security engineer to put the time into effective doc creation.
How do we figure out how to write?
Before we throw a bunch of work into this, we need a strategy. We need to work to understand what needs to be written and what needs we’re looking to meet with our documentation.
Learn from the present
The people we’re hoping to serve with documentation can give us a good idea of where to start. If it’s safe for you to be back in the office, this is a great time to eavesdrop. If you’re still remote, it’s time to read through the chat and probably a lot of it. If you don’t have a channel called something like #ask-security, a place where it’s clear people can ask questions about security and get answers, it’s time to make one. Not only does this tell people that you want to answer their questions, it also gives you a primary place to see what concepts and processes people are struggling with. What questions come up over and over? What security-adjacent work is making development proceed slowly or unsafely? See what people ask, and you’ll get your first ideas of what documents could benefit the people around you.
We also need to get a sense of what’s going on with the existing docs. In addition to understanding what’s already out there, check the views, opens, or whatever metric your document repository of choice offers. It can be gutting to see that some important doc has been opened all of 13 times (particularly if you wrote it!), but it’s information that we need. We also need to ask people what they think of the existing docs: are they written the right way? Do they cover the right material? What gaps are there? Getting honest feedback can be tricky in some work cultures, but we’ll talk about ways to work with that later.
Learn from the past
We can learn a lot about what to do and what to avoid by talking to the person responsible for existing documentation (though that might, of course, be you). If that person is no longer at the company, try to find someone who knows why that person is no longer there. If there are predictable frustrations, find out about them and figure out some ways to prevent them affecting your work.
Go back through chat histories, old all-hands decks, or other places the past lives and figure out what docs people have historically shared. What’s commonly passed around, and what does it seem like people are unaware of? If something important isn’t mentioned or looked at much, why do you think people might be unaware it exists? Keep at this until you have a good sense of behavior of olde and how well they’ve done.
Pace yourself
Documentation is a marathon, but there are things to keep in mind that can keep you going. For instance, one perk of a nonexistent set of documents (or docs that haven’t gotten the attention they need for a while) is that people will notice when you put some effort into it. A little love applied to something that’s been known to be subpar for a while will be appreciated.
Also reassuring? Perfection is impossible in this work. No, I swear that’s actually awesome. We’re looking for good enough, better, functional, and improved – not perfect. This frees us to aim to do good work without smothering it with an impossible quest.
And while you’re diving into this work and figuring out where your efforts are most needed, try to work on just a couple of questions at a time. We can’t boil nor drink the ocean, and we certainly can’t address all of an engineering org’s or company’s security questions at once. Looking at just a couple questions at a time – questions like “How does this company want to deal with phishing threats?” and “What are some simple ways to boost broad security awareness?” – keeps you from being overburdened and gives you the chance to find interesting connections that being single-minded about this work can obscure.
In short: how do we know what to write?
- Listen to conversations and read the chat to see what people know and need to know
- Check stats on existing documents so you don’t duplicate work
- Ask around to see what’s working with the existing documentation
- Work to answer two questions at a time as you proceed through this work
What makes a good document?
It’s a good document when it enables people to carry out the behavior you want. It’s not uncommon for docs to be like this well-intended and very cute cat feeding trough: you thought you were making one thing, but your users thought you made something else.
How do we build a better document?
First, we need to be thorough. There’s a tendency in some tech documentation to make a lot of unquestioned assumptions about the reader, leaving out vital details, specifics the writer deemed unnecessary, or other key pieces of information that can derail the reader as they go read something else to try to fill in the gaps.
Instead, add whatever you know about the subject. If you know specific commands, file locations, steps, links to background information, or anything else that can help the reader accomplish their goal, just include it. We want this to be a complete resource for finishing the task the doc is meant to illuminate. What if your entire team got food poisoning or, you know, Covid? We’re writing for the survivor. We’re writing to help Sigourney Weaver get to safety in Alien. Do it for Ripley.
Docs should also serve people having a bad day, which is not rarely the case when they’re trying to figure out something difficult that stands between them and finishing their work. Aim to write a friendly document – or, at the very least, not an actively adversarial one. We want the user to believe – to know – that the doc’s author is on their side.
To that end, the doc should speak the user’s language, and it should not use a different vocabulary than the intended reader. It’s fine and normal to introduce potentially new terms, but we need to explain them. This can mean tooltips, explaining acronyms before relying solely on abbreviations, and linking to explanations.
To best accomplish this, we need to have a specific audience in mind. When you’re writing a doc, complete this sentence before you start writing:
I am writing this doc for $audience to describe $information for $purpose.
Some examples:
I am writing this doc for front-end engineers to describe our preferred form libraries for XSS prevention.
I am writing this doc for everyone to describe common phishing techniques for thwarting recent spear phishing attacks.
I am writing this doc for that executive to describe why we need this tool for upcoming budget planning.
(We all know that executive, don’t we?)
Each of these gives you a sense of the reader’s context, what the content will be, and a definition of success or failure. Once you know this (and I suggest pasting your completed sentence at the top of your nascent doc for guidance you start to shape it), you’re ready to write. Once you’re finished, ask someone who’s a member of your stated audience to review it to make sure it does what you think it should.
How do we spot a good document?
- It includes everything we know that a reader might need to know about the subject or process
- It’s friendly and speaks the reader’s language
- It has an intended audience, information to be conveyed, and a purpose for existing
How should we write it?
Alas, for you, I have the perpetual security answer:
There are so many different kinds of documents we may be called to write. Here are a few of them:
- Process descriptions
- Reference
- Architectural decision records
- Where the bodies are hidden
- How to recover from an emergency
- What teams do
- How to reach people
- Lunch menus
- On-call schedules
- etc. etc. etc. etc.
Your purpose determines what you need to write.
Let’s look more closely at a few common possibilities.
Playbooks
You had a thing happen; here’s what to do. An example: you need to record a vulnerability. You need to get PII out of the logs. You need to create a new shard for your database.
These tend to be either frequently referred to or rarely looked at but still extremely critical. List out steps, be thorough, and expect to refine it a few times, especially if you’re writing for people who are just learning this stuff.
Other process
This can describe what a team does, how to do something essential but less pressing than what’s described in a playbook, or any other how-to.
This also includes process flowcharts, which I will confess I often hate. It’s not the flowcharts’ fault; it’s that I’ve most often had them lobbed at me by teams with a stronger interest in getting me to go away than in helping anyone. If it’s a team’s first offering when you ask for more information, I become immediately leery. They aren’t as accessible as people think, especially if you’re throwing out a rat’s nest of overlapping swim lanes.
I googled “terrible flow chart,” and the internet delivered. It’s funny, but it’s also not far off from experiences I’ve had with these from unhelpful teams. We can do better.
If you must use one of these – and yes, there are legitimate reasons for it, though fewer than some people think – ask someone outside of your team to review it before you throw it at someone who depends on it to get their work done. If you’ve only run it by people on your team, there will be opaque parts that need to be fixed before you call it done.
A grab bag of writing advice
Keep it friendly; informal is probably best, because security can be scary.
Remember, your doc may be someone’s best hope on a really bad day of work. We want to be friendly, and being informal where you can can pay dividends. Remember too that lots of people have had bad experiences with people on security teams. Getting to be the friendly helper on a bad day is an incredible opportunity to make the future different from the past.
Explain all abbreviations and acronyms.
Tech runs on acronyms, but we need to anchor them. When you first use an acronym, abbreviation, or other specialized vocabulary, explain it in some way. Write out what the abbreviation represents, and provide a resource to explain other terms the reader may not know yet.
Link generously, both internally and externally.
Linking to other internal docs is a great way to both show the user what other resources are available and make what they need available without them searching for it. Linking externally, to the resources you inevitably come across and rely on when creating a doc, gives the user who wants to learn more resources you know are good without making them search and vet on their own. You already did the work; let someone else benefit.
Make it clear what person is behind a doc; someone will need to know.
Sometimes, documents can seem to exist in a void, but a human always started it. Make it clear what person or team wrote the thing the user is reading. It humanizes things, and it makes it easier for them to reach out to ask for more, request changes, or just say thank you.
Chunk information by complexity with headers to enable skimming.
People read in lots of different ways, and we can support them by sprinkling a little information architecture on. Use headers liberally, bold critical information, and enable skimming wherever you can. Because we can…
Never assume anyone will read the whole doc.
I know, I know. I tend to read entire pieces of documentation. Maybe you do too. Most people, however, do not. Explain things concisely; some people may read just the paragraph they need, and that’s totally ok. In fact…
A doc must work in isolation from any supporting material.
We sometimes need to write lovingly crafted suites of documentation. I love that for us. However, we can’t assume anyone will actually consume all of it. Instead, if you’re detailing a process, assume someone will never follow the internal links you crafted or follow the divine path of information you’ve laid for them. Never make it necessary to read across three or four docs to get someone through a problem if you can help it.
If in doubt: gifs and animal pictures keep people’s attention.
It works for me, anyway. If you need to hold people’s attention through something dry, throw in fun things. Surprise can keep people engaged for longer.
Include how to give feedback in each doc: survey, chat, comments, etc.
If people want to ask for more, offer a correction, or compliment you, we want them to be able to find us. Have something at the bottom that tells them where to provide feedback. This is, you see, a gift.
How do we write our docs?
- Know what kind of doc you’re writing.
- Remember that your job is to explain AND be approachable.
- Act like each doc must exist in isolation.
- Make it skim-friendly for people in a hurry.
Where should our completed docs live?
First, we have to find all the options. There seem to always be at least two official doc repositories; usually one is Google Drive, but not always. It’s our job to figure out which one to use. Both may be in play (or more!), but usually only one is the right answer.
We also have to keep an eye on chat and what gets said there. Chat is not documentation, but it is an information repository. Watch how links are shared, what information is posted, and how people react to it.
When you choose your one true doc repository, figure out how it works. For instance, how are docs marked as drafts vs. completed?
We can learn a lot of this through asking questions and observing. We can also learn by choosing incorrectly and getting corrected (and probably will). This is normal and not a problem, so long as we absorb the information given to us.
We also have to understand how people look for information. Do they horde and share links, do they roll the dice with search, or do they graze until they find what they need?
As we figure this out, it’s important to remember that there is no perfect solution here. No, seriously, this is actually great. Documentation can be more or less effective, but it’s unavoidably a subjective exercise. We can aim for better, but we’ll never achieve perfection. And if you’re tempted to suggest importing your immaculate documentation storage solution from your previous job, thinking that will surely solve all the problems you’re facing, I invite you to remember the joke about how you now have 24 Javascript frameworks.
Instead, we’re looking for more like a 75 percent solution to our problem. You know how Cs get degrees? This is that situation, except our C is consistency. Here’s a secret: mostly it doesn’t matter where our documents live, so long as people know where to look. Whatever you do, just stick with it until you have a good reason to change course. Then iterate and communicate what you’re doing to the people who depend on you. No, really, that’s it. Yes, there are some really painful places to put documentation out there, but if we’re dealing with those, we’re probably fine so long as we stay the course.
How do we figure out where to put the docs?
- Figure out all the places docs can live
- Figure out where people most often search
- Figure out how they search
- Match it when you can, especially at first
- Stay the course and iterate deliberately and clearly
Circulating our work in the company
I know some people hide in documentation to avoid humans, but I regret to inform you that we need to talk to people so they know our work exists. I promise it can be rewarding, though. Here’s how.
Chat, especially in this more remote era, is one of our best tools. As before, we want to watch for opportunities to provide links to resources we’ve written at just the right time. Now, we’re looking for when questions come up, even if they’re not directly phrased as questions. We also want to make announcements when we’ve created something new.
Meetings, though painful in too great a number, remain a great tool for socializing our work. I like team meetings for this, especially if I’ve written something because of a need surfaced by a particular team. It’s genuinely fun when you know someone’s struggled with something and their life will become easier because of your work.
All-hands and other larger, higher-level meetings are great for this too. Less targeted, they make up for the broader aim with sheer numbers. Better still is if you have a regular segment about new documentation so that people know to expect these things monthly or quarterly.
A word on bribes and incentives
It can be enough to make written resources people need, in a way that works for them, provided in the right medium for the way they glean new information. That’s great. However, if we want to influence behavior, bribes and incentives are very useful.
Stickers
Stickers are a classic for a reason. Why do we tech types go apeshit for these? I don’t fully know. Not everyone who loves them was a Lisa Frank freak as a kid like I was, so I can’t account for it, but it works very well, especially if you do limited-edition ones for an event or other achievement. They’re inexpensive but fun, and people will do things you want of them if you provide stickers. I’ve gotten them from Gibson Print and Sticker Giant. My current company gets nice ones from Printfection.
Badges
People like flair. Give them flair. You may need to make friends with your company’s design department, but I swear this is actually a fun errand.
Rewarding ambassadors
There are lots of ways to do this. Having an organized kudos program where people nominate each other for doing good security work is one way; complimenting someone to their manager is another. Call out good work when you see it.
Coffee
Most offices have free coffee, and a lot of them (especially in the Bay Area) have legitimately good coffee. Despite all this, the $5 Starbucks card still feels special. Maybe it’s the permission to go get something covered in whip, I don’t know. But it works.
The common thread here is that effective incentives don’t need to be expensive. These are not bonuses; those are done by another team and likely above your paygrade. Instead, the goal of this kind of treat is to make a compliment more tangible. We want people to know we see and appreciate their efforts, which makes them more likely to do it the next time they get the opportunity.
Get comfortable repeating yourself
We rarely learn something on the first encounter with it. Instead, the journey to full documentation awareness for any given user is more likely chat + seeing it in the docs repo + a mention in a meeting + a link provided by a teammate at just the right time. This is to be expected; it’s normal and fine.
If you feel uncomfortable repeating yourself (as most non-tedious people do), make a calendar for when you’ll announce things in meetings, chat, and other places. This way, when you feel like you’ve said it too many times, you can look at your schedule and know exactly how many times you’ve said it. This also makes (yes) iterating easier, because you can see at a glance what you’ve done and decide whether more, less, or different would be a good idea.
How do we get our work out there?
- Promote where the people are: chat, meetings, all-hands.
- You are going to have to repeat yourself.
- You are going to have to repeat yourself.
- Bribe people toward the behavior you want.
How do we know if our efforts are working?
We have to evaluate what we’ve done.
We can ask people via regular check-ins and one-on-ones with people who gave us pointers on where to put our efforts. We can do surveys, though some people have had bad experiences with them. If you keep them short, to the point, and connected to a personal appeal, you’ll likely get some good information. We can look at metrics, going back to the views and opens we discussed earlier.
We can also find a goal person. This can be a persona, but if you can, try to find a real person whose needs you’re looking to meet. Once you’ve made the resources they need, you can find another goal person and start working to make other people’s lives easier.
We also need to review our body of work regularly. I like quarterly for this, but if that sounds punishing, biannual can work too, so long as you don’t skip it. We need to know what we’ve published, how it’s being used, and what it tells us about the future of our work.
It’s also a great time to review feedback, which ideally we’ve been collecting all along via our feedback call to action in every doc we’ve published. If you haven’t done that already, this is a great time to do it.
A note on soliciting feedback
If we are very lucky, we work at a company that prioritizes psychological safety. However, if you don’t, you may need to do some extra work to get feedback from people. At psychologically unsafe companies, what you know to be a friendly request for an opinion can sound like a trap to people who’ve gotten used to living in these cultures of no. Honest feedback depends on people feeling safe enough to be honest. Alas, if you’re in charge of security documentation, you probably don’t have the stature in the company to make a culture-wide change of this kind.
However, you can make a psychological safety bubble around you and your team. Like the rest of this work, it takes steady effort across time, but it can be done. When you ask for feedback, make it clear that you really want to hear the tough stuff, because you really want to improve things. Emphasize that you’re asking for opinions on a document, not on you or your team. And, when you hear difficult things, take that feedback and use it.
Across time, people will begin to trust you (if they didn’t at first) and be more immediately forthcoming; seeing their needs addressed warms people up, even if they’ve had reason to be withdrawn.
This can be tough to do, especially if you’re not used to it, but this work can’t go without honest reactions and opinions.
What next?
We get to start the next cycle. Ask around, do your research, determine new needs, and keep mapping the future.
How do we measure success?
- Do surveys, ask people
- Review regularly
- Create a psychological safety bubble if wider culture doesn’t support it
- Start the next cycle!
Keep with this cycle of listen, learn, implement, and iterate, and across time even the worst problems of documentation will begin to lessen. As documentarians, it’s our job to serve our colleagues. It’s tough work, but it scales better than almost anything we can do, and it’s available even when we’re not there to answer questions. I find it to always be worth the work, and I hope you think so too.