Pipefail Bail: When to Add (or Remove) set -u

The scene: I was going back to a set of 18-month-old Packer files to add set -eux -o pipefail to each file in the build. (If you’re not familiar with this command and its uses, here’s where I learned about it. Highly recommended.) I’d recently had a two-day time sink, wherein I couldn’t get LDAP access to work on our CI/CD, and eventually I found that the shell script that adds our LDAP certs had coughed and died midway through without Packer erroring out and letting me know that something was wrong. LDAP failure is a pretty common sign that something is wrong with our CI/CD, but in the past it’s been due to more exotic problems than Packer petering out. Pipefail isn’t necessarily the right tool for every job, but I wanted to spare my future self these issues, where VERY SIGNIFICANT PROBLEMS might otherwise be buried in a billion screens of Packer output.

(Yes, I’ll still look at the credentials folder fist next time.)

That was how I learned that Packer scripts can fail, but the build can still complete. This surprised me, considering how many failed builds I experienced when I was first working with Packer. So now I’m working through each script, finding quiet problems (such as unnecessary symbolic links being created during the installation of our version of Java) and other issues that perhaps aren’t problems today but may arise like the kraken later to take its accumulated revenge. Like I said, these scripts have been in use for about a year and a half, building AMIs at least once a month. Usually, only the base AMI changes, and the only other alterations have been additions – this version of Ruby that one dev team needs, this package for another group. Beyond that, it’s been pretty steady, which means a fair amount of time has passed since any kind of in-depth review of these files.

Pipefail is a great and rather educational way to work through your scripts, but on a recent day of this little side project, I encountered a surprising problem. In one of the scripts, PATH is augmented, followed by source /etc/bashrc. This is when the file errored out, with a gasp of  amazon-ebs: /etc/bashrc: line 12: PS1: unbound variable.

What in the what?

I did some googling for this unbound variable business, but the results didn’t apply to what I was doing. I wasn’t failing to create $youMessedUp. /etc/bashrc did indeed exist in the Packer Build instance, which I confirmed by, variously, touch /etc/bashrc, ls -a /etc | grep bashrc, and cat /etc/bashrc, at various times in my troubleshooting. The source command was being used correctly. And there were exactly no variables in that script.

Huh.

But /etc/bashrc was a robust file, quite lengthy compared to the most familiar file of its type in my life, the ~/.bashrc on my own machine. There was a lot going on in there… including variables. And because of the kinds of AMIs I use on this project – that is, AMIs built by a different team I have little contact with, issued every month without exhausting notes on what might have changed from the last version – any alteration I might have made that day might be useless or, worse, damaging when applied blindly next month.

Shit.

Beyond that, there was the issue of scope. This pipefail project was supposed to be about controlling my end of things. Faulty machine images and limited control are just part of my job. I’ve dealt with said images, but the dealing is not typically dont in shell scripts. Usually, if it’s something especially sticky, the job becomes one of communication, wherein I document what’s up and reach out to the agency in charge of regular base AMI creation so we can sort things out.

So that resolution and realization was where set +u came in.

I have an ongoing concern that shortcuts that I think are efficient might be unhelpful cheating, especially in this particular phase of my career. I ran my error and my situation by a few more senior engineers at my job. The idea of set +u came up. And said seniors confirmed that this was just wise and not laziness.

So:

set +u

source /etc/bashrc

set -u

That is, repeating the command at the top of the page. +u reverses that initial -u flag, which ends the script when an unbound variable happens. For that one line, only set -ex -o pipefail is in play, minus that situationally unfortunate -u.

This is useful if you have a weird situation like mine, where you need to run bash strict mode most of the time but have a line or a section of a script that deals with a resource that’s out of your control (but which you can still trust). Other times this is useful is if you’re activating a virtualenv in Python. In that case, set -u may be best set aside for that particular endeavor. In short, if your script is opening a big bucket of things out of your control (/etc/bashrc, the contents of an /env/bin/activate folder), and you want to go full set -eux -o pipefail otherwise, pop a little set +u in there.

But, this specific little situation aside, I’ve become a convert for set -eux -o pipefail on my Packer builds for sure and will probably keep the habit when I’m in a situation where I’m using AMIs not made by an outside team. The more you know, right? 

11 Lessons from My First Year in Software Engineering

Paper garlands against a twilit sky

I hit my one-year anniversary as a software engineer in October. It has been, professionally, one of the harder, stranger years of my life, but the challenges generally were exactly what I hoped they would be: complicated, but with clear questions, and answers that were a pleasure to seek. That said, there are a few things I wish I could whisper to my past self, either right when I was starting this job, just as I was starting Hackbright, or a couple of years ago when I wrote my first lines of Python. Here’s a bit of advice to my past self, to anyone who’s considering this journey, and to anyone who’s still fairly new and would like a little reassurance.

1. Everything I heard about learning this is true (or: no, really, just pick a project).

If you want to learn programming, you do need to just pick a project and proceed. I really disliked this advice when I first heard it, because I am all about context, and I couldn’t imagine picking an appropriate challenge without knowing the limits and possibilities out there. And that’s a legitimate concern – it’s crucial to pick the right-sized problem, so far as complexity and the number of tools it will require, if you’re going to learn without getting so frustrated that you quit prematurely. Even so, it’s those raw edges, that unpredictable stuff, that gives you the real learning, that can be the most educational (and most satisfying) to wrap your brain around.

My real, substantial learning on this job began when I was put on a project, which didn’t happen immediately after I was hired. I had learned things before then, self-studying along in the office, but it lived strictly within the realm of the hypothetical (something I consider likelier a limitation of my own beginner state than anything else). Learning within the context of a project can be kind of like memorizing a poem by hearing every fifth word, and out of order and occasionally in a different language to boot. However, what you do learn will be practical and actionable, and – perhaps most valuable of all – will provide the context around what happened and what you need to do. And eventually, you’ll know a lot of it – and be able to intuit or sleuth out the rest.

My suggestion to you: it’s annoying how much it’s true, but I’d suggest just giving in (and finding a good advisor for picking and shaping your project, if you can). Find a practical problem in your own life and decide a way to start addressing it. If you get stuck, it’s a big, generous internet out there, and some of the people in it will even have right answers. 

Bonus suggestion: consider making a command line utility. It has a delightfully low barrier to entry and gives you a great chance to make something useful to you without worrying about deploying or front-end work. If you’re a Python kid like me, start by looking at argparse and then let your imagination run away with you. 

Bonus bonus suggestion: many programming communities now have Slack networks that are open to the public, if you request access. If you know you’re interested in a particular language and want someone to ask questions to, see if there’s an active Slack channel for your area of interest. The availability of DMs and the more regulated, curated nature of most Slack communities can make them friendlier to beginners.

2. Learning is a skill. Learning this is a different skill.

Computer science’s history is relatively short, but it’s some dense archeology, if you’re trying to wrap your head around even the most essential central stuff. Some people get to be immersed in it for four years before they’re thrust into the workplace; the rest of us get to pick up on useful commonality when we start playing with our third programming language. (Though people like Gayle Laakmann MacDowell have said that this is far from an insurmountable hindrance.) The good thing is that each new skill you learn will require slightly less origination and effort and will build slightly more on things you’ve already learned.

However, this growing knowledge will never reach one hundred percent, regardless of your background. If you plan on staying in this field, you have to learn to love at least a little constant disorientation. If you aren’t confused on the regular in your first couple of years in this field, you’re not trying hard enough.

My suggestion to you: learn to love feeling like your feet aren’t quite firmly planted beneath you, because it means you’re in the learning space. Disorientation means you’re surrounded entirely by new things to learn. Eat it up.

3. Any dregs of self-consciousness and admitting ignorance will either go out the window fast – or you will remain bad at this.

My company is largely remote four days a week, and I was, for a time, the only engineer in our central office. This meant that, if I had a problem and my manager wasn’t available, I had to go into a public Slack channel to seek help. This eased in time, mostly as I got to know my coworkers better. But until I got to that point, every question I asked felt like broadcasting my ignorance to the company, who only knew me as the inquisitive little Slack avatar. HEY LOOK AT ME HERE’S THE THING I DON’T KNOW OF THE HOUR.

That is, until I stopped caring because I understood that no one else cares. And beyond that, it’s as true here as in any other field that the best time to ask basic-ass questions is toward the beginning, when they naturally occur, before you start eroding the foundation you’re trying to build. 

My suggestion to you: breathe deep and get over it – or pretend to until it’s true. Admitting you don’t know something is a vital part of being good at this job, because there’s no room to bullshit. Any fudging you do will be revealed later, and most likely at a really annoying (and embarrassing) time.  

4. Useful experience is less about exhaustive knowledge and more about navigating new situations and tech.

Expertise can sometimes be demonstrated by knowing who wrote what language, what the most vital book is about a subject, or the history of the specific design decisions and needs that went into a framework. But this is surface trivia, and what’s most important (to me, so far) is context and the experience that provides it. It’s still the thing I crave most often, when I find myself in those disorienting moments where I don’t know the answer and am not even entirely certain of the right question.

It can be extra frustrating because I don’t just want to know how something works. I also want to know the situations where considering that thing as a solution on a project is appropriate, what would inform that recommendation, and what you heard about its past releases and future plans that might make everything terrible in six months.

I felt this basically constantly at the beginning and now, fourteen months in, I still feel this way pretty often. I choose to view it as still finding this field incredibly interesting. I can’t imagine what being bored or plateauing would look like in this job because there is always, always more stuff. And, after a while, you’ll have experienced enough of it that you’ll know better how to navigate the next big thing. 

My suggestion to you: hang in there, mostly. And just be willing to try things, volunteer for new projects, and get all of the experience you can – within reason.

5. My sense of curiosity is a valuable job qualification.

I have, in the past, annoyed lesser bosses by asking why. When I asked, I wasn’t questioning their judgment – or not usually, anyway. What I needed was to understand what went into a given decision, so that I could make my own decisions to support it appropriately. (Yes, it does make sense that I have user research in my background too.)

This quality is really useful in this job – in fact, in a well-functioning environment, I’d call it essential. It’s particularly so when you do consulting for clients, as my company does. Sometimes we serve them better not by doing exactly as they request but by asking why enough (and politely enough) to find out what it is they really want. From there, good work actually gets done.

My suggestion to you: your beginner enthusiasm and curiosity are valuable tools. When you don’t take anything for granted, you can notice things more seasoned engineers don’t. If something isn’t clear, ask about it (even if only privately to your boss) until it becomes clear.

6. Sometimes the tool is broken. Not you.

Early last year, I was doing some experimenting with AWS on my own at work, going between the command line and the web UI to launch instances, tailor and tweak them, and get used to the interaction between different aspects of the tool. But for a few weeks at the very beginning, things just didn’t work right. I’d follow a tutorial, enter a command, and – what even the hell? Trying to spin up an instance would fail. Security groups wouldn’t work right. And, worse still, I was so new and the failures were inconsistent enough that I couldn’t deduce any logic from what was happening. I was failing and didn’t feel like I was learning from it, one of the worst feelings. I rarely have reason to wonder if maybe I’ve been secretly stupid all along, but in that handful of weeks, I’d stop sometimes and wonder if engineering was finding some sad new quality of mine that had been hiding throughout my career.

Then another senior engineer got hired and had a little time before being put on a client. He found that our AWS account was old enough that it worked differently than more recently created ones do. He made a new account. Suddenly, tutorials made sense, and my results were predictable – including my errors. I was so relieved I had to stop and stare into space for a few minutes to absorb it all. AWS and I are friends now, despite our rocky start, but I would never have figured this out on my own.

My suggestion to you: sometimes the problem is between keyboard and chair, sure. But sometimes it is not. Ask questions, pair with someone, and make sure that someone who knows more than you witnesses your sticky moments sometimes. It’s ego-deflating, but it’s better than spending days or weeks flailing in some swamp that isn’t of your own making.

7. Timing is everything.

If you have even a semi-active sense of curiosity, you can spend endless amounts of time reading docs, essays, StackOverflow speculation, comments, comics, reviews by the competition, helpful blog posts, amusingly bitchy blog posts, and so many other things that may be very useful, completely useless, or – worst of all – approximately 29 percent useful. It’s that last one that can eat your afternoon. If you aren’t aware of this particular hazard, you can lose an hour or four much more easily than you might have ever suspected.

My suggestion to you: timebox that shit. And if you have access to someone more experienced than you, work out a relationship where you can come to them pretty regularly for reality checks and course corrections before you sail yourself deep into the ocean of chatty, chatty internet people. It’s ok to ask a more senior person to rule out some obvious stuff before you dig into researching your problem.

8. Unless the docs are shit, trust the docs.

(And if the docs are shit, should you really be using the thing it’s documenting at all?)

I realized recently (thanks to talking with one of my bosses; see the previous section), that I’d developed a habit I’ve nicknamed narrative research. I’d come to believe that the most efficient way to work through problems was to try to match my problem to someone else’s phrasing, find their solution on this or that third-party site, try to get that solution working to fix my problem, and then work backward to find out why what I had done worked, to learn a larger lesson from there.

Perhaps you’re already seeing the problem here.

If the tool you’re using requires the backassward methodology of someone in a completely different context than you to get it to work, it may be time to examine if you’re using the right tool – or, perhaps more likely, if you’re doing it right at all. You can stir your coffee with a screwdriver if you really want to, but there are better ways to use it. If you have a problem to solve, research just enough to find what library or whatever it is you need to use – and then use its own documentation. Don’t work off-label unless you really need to. Probably check with someone more experienced, if you really think this is a good idea.

My suggestion to you: there be dragons in Stack Overflow sometimes. Stay with primary resources as much as you can.

9. If you’re a person who does the caffeine thing, get your coffee game down.

I most often need one between three and four pm, just to perk my brain up to get through the rest of my day. A single Americano is a great way for me to address this. Recently, I messed up and overcaffeinated myself via the rookie mistake of using a bigger glass than usual for my cold brew. I spent the afternoon sweaty, with racing thoughts. Not a good look.

This is general life advice too, but I’ve found it more critical in this job than any other. It may seem surface, and maybe it is surface, but having your biological needs in check will let you do better at this.

My suggestion to you: know thyself.

10. Don’t be a hero when you’re sick.

This is especially important for me and my consulting colleagues who have a vested interest in quality billable hours, but: if you’re sick, be sick. Don’t soldier through. (And not just because of the obvious part about not being a disease vector. Seriously, stay off my BART if you’re ailing and have sick time to use.) If you feel like shit, you’re not going to be able to brain, and this work requires a functional brain more than any other job I’ve had. The others could be difficult too (especially the UX consulting gig I had just before I went to engineering school), but it’s just… different. Pack a snack, sleep enough, and pay attention when you’re sick.

My suggestion to you: be an adult and be honest with yourself. Sleep enough, eat enough, and stay home with pho from Seamless if you’re under the weather. Treat yourself like you’re parenting a toddler – you know, honest assessments. Sometimes you just need a snack; sometimes you need to stay the hell in bed.

11. And, finally: decency counts.

This is an industry riddled with social fuckery, and even people who found it worthwhile to stick it out usually have at least a couple really vile stories of colleagues and managers acting like total assholes. I work in a magical unicorner of the industry that’s largely free of that, but – get this – I still get points just for being housebroken and friendly enough that it’s pleasant to share space with me. It still seems to be considered remarkable in this industry (though it’s a requirement to work at my company). Can you treat a troublesome team with human decency? Can you be polite and keep it together even when you’re having a bad feeling and not getting your way? Do you have a regular life, and can you make nice chit-chat about it without it being a big thing? Congratulations: you have an important skill.

Beyond that, social stuff in tech is just different than it is in other industries. I’ve always been lucky enough to have coworkers I wanted to be friends with too, but there’s a certain all-banding-together kind of feeling in tech that I haven’t seen anywhere else. In some companies, it’s a natural side effect of putting a bunch of 22-to-29-year-olds with a shared predilection for alcohol in the same space for 60-plus hours a week. But even then, it has a function – when stuff gets hard, that empathy and caring and shared knowledge comes together, and everything functions better.

My suggestion to you: be cool, honey bunny. And, even if you have limited social energy (I certainly do), try to conserve some of it to spend time with your coworkers once every week or two. A lot of people are lovely, and the stuff about being a good member of a team is easier if you’ve taken a real interest in the people around you.

There you go, new engineer. There you go, Breanne of a year or two ago. And here are a few more resources that I’ve found really useful in the last year. I didn’t even write all of them myself.

  • How to edit your PATH variable (and what PATH is): I had the hardest time getting an answer to this, which was tough when I was already learning a lot about how a computer works when you’re not just using it to dick around on the internet. So I pestered my coworkers for answers until it felt coherent and wrote it down. I hope it helps you too.
  • 7 Things I Wish I Knew Before Starting at a Developer Bootcamp: my friend and coworker Emily Chen wrote this, and I really wish I could teleport it back to myself in spring 2015. Why this isn’t a prereq for every immersive programming school, I do not understand.
  • The rad illustrations of Julia Evans: always thorough and yet always approaching subjects from a unique angle, her illustrations are such a nice companion for whatever you’re learning.
  • And, just, you know what? Wikipedia is the shit for computer science stuff. Surprise! There’s a lot of legit documentation out there (ahem, man pages, ahem), but Wikipedia is so often a great place to start, and seeing unfamiliar stuff laid out in a familiar format can be really helpful if you’re stumped.

Restoring Github Repo Access in the Terminal after Adding 2FA

Molding, windows, and walls of central room of Oakland's 16th Street Station

This is the second time I’ve had to look this up, and the instructions are spread across a couple of pages. So for the sake of my future self, I’m compiling the steps here. Maybe it’ll help someone else too. That’d be cool.

So you added 2FA to your Github account. (Very smart; nice job.) Problem: your terminal no longer accepts your (still valid) password when you’re trying to push to a repo. Here’s how to fix that.*

1. Create a new personal access token.

Go to Github and ensure you’re logged in. (An aside: you’ve saved your access codes in a password manager or some other secure place in case the device you designated for 2FA authentication codes catches on fire, right? Excellent, carry on.)

Go to the upper-right corner of the page, click your profile image, and click Settings. In the sidebar of the next page, scroll to the bottom and click Personal access tokens. Click Generate new token. Give it a name that will make your future self grateful for today’s clarity. Select what permissions you want this token to give. Then: Generate token. The token that appears will only be visible right now, so make sure you save it or do what you need to do with it. (Or save it to 1Password and then proceed to the next step.)

2. Save it to your OSX keychain credentials

Here’s the choose your own adventure part: do you have Github credentials saved for more than one Github account on this machine?

2a. Nah, just one. Let’s nuke it and replace it.

Cool, let’s get rid of the old credential first. Go into your terminal and type this:

git credential-osxkeychain erase

There’ll be a pause and then nothing. That means success. (I still find this convention strange, but I like a verbose interface, so that’s my own struggle to deal with.)

Now, do that push that you were probably trying to do when you realized you’d cut off access on a machine you perhaps don’t code from so often. It’ll ask you for your username and password. Give it your username as usual, but instead of your password, paste that Personal access token that we created in the last step into the terminal. Hit enter. Wait. Enjoy your success (which will be confirmed by a message this time).

2b. I’m super cool and access SO MANY accounts. Let’s just replace one, please.

Alright, fancy. For this one, we’ll start with the Keychain Access app instead of the terminal. (Don’t worry, we’ll finish in the terminal for this step too, if you prefer.) Open Keychain Access and then type in “Github” to narrow down the options listed. You’re looking for an entry for github.com. To confirm you’re looking at the right one, ^click and select “Get info” from the menu that appears. Confirm that the username you’re looking for is the one listed under account, then select Access Control and confirm that the option listed is git-credential-osxkeychain. If those two things look right, we’ve found the right credential.

Now you can edit it there, replacing the existing password with your new token, or you can delete it and provide the new credential via the terminal when prompted after you try to push again, as described in 2a.

There you go! Such security, what authentication.

Want to double-check my work? Here’s what I used to put together these steps (both times, because sometimes that’s what it takes):

And if you haven’t set up 2FA for Github yet, my god get on that ok wow with the instructions here.

*On a Mac. That’s the context here. Mac Mac Mac.

Header image is from my recent visit to Oakland’s 16th Street Station, which is usually closed up. In this case, my access was granted by paying to be part of a photo tour. Authentication comes in many forms.