About once a week at work, someone hits an error while trying to deploy code. It’s always the same one: permission denied early in the Capistrano deploy process. The solution is always extremely simple: run ssh-agent; ssh-add from the command line, and the error goes away for so long that everyone forgets what to do to solve it. Today I solved the problem for us, hopefully once and for all. I want to talk about why this can keep happening, and share my code for others to solve it as well.

Capistrano uses ssh to deploy code by running commands remotely on your server. Most people are at least familiar with the concept that ssh uses public key authentication to allow you to securely connect to your server without entering a password. Most people are also aware that Github uses public key authentication to allow you to pull from and push to your git repos without a password. Very likely, both your server and Github look for the same public keys.

There are a lot of issues one can run into setting up public key authentication, but usually everything works fine once initially set up. The error we’re dealing with today however, tends to come back. What’s the difference?

The key is that when using Capistrano, there’s more happening than just authenticating with either your server or Github. You’re authenticating with Github, from a command run by Capistrano ON your server. Thus it’s essentially your server authenticating with Github. Your private key, however, is only on your local computer.

Fortunately, that’s exactly the problem ssh-agent solves. It’s built specifically for one thing: securely storing your private credentials in a way such that they can be used to authenticate over an ssh connection. Thus your server can talk to Github as if it owned your private key, but your key is safe on your own computer. If you have an ssh agent running and configured with your private key, of course.

To quickly spoil the mystery of the two commands needed to run to solve this issue, ssh-agent starts a background ssh agent process, but it’s initially not configured to use any ssh keys. They must be added with ssh-add. By default ssh-add adds private keys in the most common locations, so you almost never have to supply any parameters. Thus ssh-agent; ssh-add is usually enough.

So why is this such a recurring problem then, if you just have to run ssh-agent; ssh-add? I think there’s two reasons.

The first is that it’s hard to distinguish from the Capistrano output that the problem is your server failing to authenticate with Github, not your local machine failing to authenticate with your server. Since the second possibility is what most people think of first, it’s the one they try to debug.

The second reason is a bit more insidious. Its easy to remember how to solve a problem you encounter again and again. This problem happens infrequently, but regularly. As it turns out, unless you take steps to prevent it, it will happen every time you restart your computer (since ssh-agent doesn’t start automatically, although it probably should). We should count ourselves lucky that we don’t generally have to restart our computers very often these days, but as it turns out there is a downside.

Okay, so this problem is easy to solve if you know how, and easy to prevent yourself from running into ever again (just ensure ssh-agent; ssh-add is run every time your computer starts, for example by putting that line in your ~/.bashrc). I wanted to do a bit more though, and ensure not just I but everyone on my team would be safe from this problem for all time.

That’s why I’m happy to announce the hastily and uncreatively named capistrano_auto_ssh_agent gem. Simply put it in your gemfile, add require 'capistrano_auto_ssh_agent' to your Capistrano deploy.rb file, and Capistrano will run a task to ensure ssh-agent is running and correctly configured before every deploy.

It’s a simple gem and I don’t expect to have to do much more work on it, but feedback is very welcome. Enjoy!

Recently, I had the need to set up a Linux machine so that I could put it behind a home router with no special configuration, and still access the machine from anywhere. Machines behind home routers are normally difficult to access from the outside world for two reasons: dynamic IP addresses and network address translation.

Dynamic IPs mean you don’t know, without being behind the same home router, what the IP address to connect to is at any given time. Network Address Translation makes it impossible to connect to a machine behind a home router unless the router is specifically configured to allow it.

However, nothing about this arrangement disallows the machine behind the home router from making outbound connections, and thanks to the flexibility of SSH tunnels, we can use this to create a simple, self configuring and healing connection allowing us to SSH into the machine behind the home router.

Here’s the different pieces of hardware that play into the story:

HostA: This is a Linux machine with a known and publicly accessible IP address. Its best if the IP address is static. In my case it’s a VPS on Linode, but any Linux machine accessible via SSH can work. No root access or special configurations are required.

Router: This is the home router. For our purposes it’s just a standard Linksys router with no special configuration, and we do not have access to its control panel.

HostB: This is the Linux machine you will put behind the home router. Again, full root access is not required, only a user account. It’s assumed that we will lose physical access to this machine once its placed behind the home router, but can configure it as we wish before then.

HostC: another Linux machine of any type and network location, where we have direct physical access.

Given these machines, the requirement is simple: allow us to SSH from HostC to HostB.

SSH Tunnel Primer

One of the most powerful abilities of SSH is to set up port forwarding (much like might be done on a home router), but ensure the forwarded traffic is routed through a secure connection created by SSH.

One of the classic uses for SSH tunnels is in fact exactly what we need to get around the restrictions put in place by the router. If we execute the following command on HostB

ssh -R 10022:localhost:22 HostA

then we can connect to HostB from HostA this way:

ssh -p 10022 localhost

How does this work? The first command tells SSH to set up a secure port forwarding from port 22, where SSH is listening, on HostB, to port 10022 on HostA. The router does not disallow HostB from initiating outbound connections, so this command does not fail. Users other than root can set up sockets on ports above 1024 without issue, so this also doesn’t require any special permissions on HostA.

However, currently, this command would have to be run manually on HostB, and it simply drops us into an SSH session to HostA. As soon as that session was closed, or as soon as there was a network issue causing the SSH session to terminate, our tunnel to HostB terminates as well. A little more work is required to create a foolproof setup.

Setting up a hands-free SSH tunnel

To facilitate setting up persistent port forwarding, SSH has two handy options: -N, and -f. Combining these two with our command above will send SSH into the background after initializing our port forwarding. With this, a simple cron job can ensure that we have a connection to HostB even if the network temporarily goes down or HostB is rebooted. However, it will create a new SSH connection every time, which is at best inefficient, and at worst can consume all the resources on one of our machines.

How then, can we check if there is already a connection from HostB to HostA, and only initialize a connection when needed? Let’s take a look at another SSH port forwarding command:

ssh -L 19922:HostA:22 HostA

If run from HostB, this will set up a port forwarding from port 19922 on HostB, to port 22 on HostA, essentially the reverse of the previous forwarding. We can already ssh to port 22 on HostA directly from HostB, but now, we can also SSH to HostA from HostB with the following command:

ssh localhost -p 19922

Since port 19922 is only active when our ssh tunnel is in place, this allows us to check for the tunnel’s existence: if our tunnel is down, SSH will fail with a connection error.

One final trick SSH gives us is the ability to set up multiple port forwardings in one command. With that, our complete solution can be written in a simple bash script:

1
2
3
4
5
6
7
8
9
10
11
12
13

createTunnel() {
    /usr/bin/ssh -f -N -R 10022:localhost:22 -L19922:HostA:22 HostA
    if [[ $? -eq 0 ]]; then
        echo Tunnel to HostA created successfully
    else
        echo An error occurred creating a tunnel to HostA. Return code: $?
    fi
}
/usr/bin/ssh -p 19922 localhost ls > /dev/null
if [[ $? -ne 0 ]]; then
    echo Creating new tunnel connection to HostA
    createTunnel
fi

The meat of the script is in the createTunnel function, which creates the tunnel as described above. The canary tunnel is checked on line 9, and if the connection fails, a new tunnel is created.

I’ve been running this code myself for several months and haven’t had any problems, but would love to hear from anyone else using something similar. Happy coding!

I am a horrible procrastinator. Starting up new tasks can be next to impossible. My mind’s ability to unconsciously switch from problem-solving to Reddit-browsing mode continues to astound me. For a long time, I had no way to counter my bad habits. I felt bad, and got a lot less done than I would have liked, even for projects where I was very motivated.

I haven’t completely defeated my procrastinator tendencies, but after years of battling against them, I have at least found a few effective weapons. Well, really, just one. Break big tasks down in to small ones. Really small ones. Like, sometimes can be done in a few seconds small. Here’s how it works.

Most people agree that getting started on a project is perhaps the hardest part of finishing. It’s obvious why: every possible outcome, every possible task stares back at you from your blank sheet of paper or empty text editor. You could be done in five minutes, or you could have to invent a whole new branch in your field to finish. There’s no way to know. That’s a scary thought, better just read a few online news articles.

The good news is you don’t have to consider every possible step to get started on something. You just have to consider the first step. So what is it? It’s definitely not to build one whole part of your project. The first step is to simply get to the right place where you can get work done. Close Hacker News, and just open your text editor. You can’t possibly make progress from your web browser. Now, you need to open whatever file you’re working in. Now find the right part of that file. What you have to do might be no more clear, but at least your in the place where you can actually work on it.

Now the trick is just to stay there. Every time you come to another unconquerably large task, you’re going to default to going back into distraction mode. It’s amazing how quickly and subconsciously that can be done. Maybe you are stuck immediately, maybe you’ve been working along just fine for a while and then hit a wall.

Either way, the trick is to concentrate on the problem at hand, and break it into little pieces. The easiest way to do this is to not worry about coming up to the solution to why your stuck, but just concentrate on the question of why you’re stuck.

I literally write down a complete sentence: what question I’m going to have to answer. After that I start writing down all the possible ways I could solve that problem. I list the tradeoffs, I list the solutions I know would probably be bad.

About half the time, this is enough to keep me going. I knew what the right way to proceed was after all, but was letting a little uncertainty derail my productivity.

Sometimes I’m not as lucky. Maybe the problem is really hard, or maybe I just don’t know how all the pieces fit together yet. Fortunately, I just wrote down my plan of attack. I can investigate each possible solution and decide which one to go with.

On my most productive days, I stay in this cycle for hours. I’m not always making progress, but I’m always making progress on making progress. Try it out and see how it works for you. You know you’ve been successful when at the end of the day you finally open Reddit and see that there’s whole pages of new stuff that you haven’t seen. Dive in, you’ve earned it.

A few weeks ago I finally added a cast iron skillet to my cooking toolbelt. Since then, I’ve nearly burned myself several times, struggled constantly with its significant weight, and obsessed over the delicate cleaning and maintenance it requires. I’ve also made some of the best food of my life.

One of the most common formulae for a successful entrepreneur is to build a product or service to take the pain and difficulty out of some task. Often though, what’s given up for ease of use is flexibility, functionality, or power that is essential when going beyond the most basic tasks.

Most of the time, this is fine. Warming up some food for a quick bite before you run out the door was not so easy when we didn’t have microwaves. But when you want to go beyond the basics, and really do something challenging, your tools built to be easy to use can hold you back.

The iPhone camera gets better every year, but serious photographers will still need their many lenses, their wide depth of field, and countless other powerful features you can’t (yet) get on an iPhone.

That’s not to say all that is difficult to use is better. It’s notoriously difficult to shoot and develop quality photos with traditional film, but most photographers now believe film is not the best tool for most cases.

Sometimes, the “simpler” tool comes with a new feature useful everywhere. Part of the reason shooting photos with the iPhone is easy isn’t just the small form factor or simple interface, but how easy it is to take that photo and send it wirelessly to your computer, or straight to the internet. Many professional level cameras now have WiFi to enable this same thing.

Difficult tools aren’t difficult for no reason. They’re difficult in order to be powerful, and flexible. They’re difficult so that once you do know how to use them, they won’t hold you back.

So when you just need to get something done, by all means, make things easy for yourself. But when you want to make something great, it might help to make things a little harder.

With the tradition going strong over at r/flying, I figured I better share the story of my checkride before I forget it all. To get any real suspense out of the way, I passed, but it wasn’t a breeze. Let’s start at the beginning.

The Setup

A few months into some much needed funemployment, I rather suddenly realized there was nothing stopping me from learning to fly, something I’ve always wanted to do. I searched online for flight schools, found Advantage Aviation at Palo Alto, and searched through their instructor list. A few piqued my interest, and the first instructor I called was Andrew Dilworth, who runs his own flight instruction business out of Reid Hillview, and we set up a flight for the very next day. Training was a blast and between my high level of excitement, my flexible schedule, and most of all Andrew’s incredible instruction, things went fast.

I was quickly doing things as a student that many new private pilots won’t consider, like flying to Tahoe, all around the busy SFO airspace, and even doing a night touch and go at San Jose International (all with my instructor of course!).

The airplane I was flying, N9848J, was a fantastic 1980 C172. It didn’t have any luxuries, but the 180HP engine - 20HP more than most 172s - ran great, and unlike many of the other training aircraft out there, it never had any mechanical issues. When it came time for my checkride, I was comfortable both with my knowledge and in the airplane. In the week leading up to my checkride alone, I logged 7.9 hours, and was excited to show off what I had learned.

The Start of Checkride Day

My checkride was scheduled for December 15th, 2012 at Reid Hillview Airport. It was starting at the reasonable hour of 10:30AM, but I was living in San Francisco at the time, meaning simply getting to my checkride was quite an operational challenge, and an early morning. I planned everything out, and got to bed early, but wasn’t able to sleep very well. I wasn’t especially nervous, but was eager to rid myself of the uncertainty of what would happen at the checkride, failure or success, and move on to the next step.

The trip down from San Francisco went as planned, but was time consuming, a little tiring, and probably too much to bother with on an important day like a checkride. Next time, I’ll find a place to stay near where I’m flying.

One piece of advice I received from multiple pilots was to do some solo flying immediately before your checkride. The idea is this allows you to “warm up” like you might for a race. Fortunately, the I was flying and my examiner were based at different airports, so I didn’t even have to make this choice. I have to say the routine and familiar short flight from KPAO to KRHV really helped.

An Unexpected Story

While it didn’t phase me, an interesting thing happened to me while I was flying to Reid Hillview. Normally on this flight, you are handed off rather quickly from controllers at Palo Alto, then Moffet, then San Jose, and finally Reid Hillview as you pass through their respective airspaces. Today, the controller at Moffet instructed me to switch directly to the frequency for Reid Hillview Tower.

I immediately thought this was strange, since the transition through San Jose’s airspace includes crossing over their runways at only 1500 feet, something their control tower would surely want you doing under their supervision. Figuring, correctly, that if the controller made an error it would quickly get sorted out, I switched to RHV tower like he said. The controller there was a bit confused given my location, and quickly handed me back over to San Jose tower, where I expected to end up all along.

Again, having been specifically trained that both I AND controllers will make mistakes, this didn’t rattle me at all.

Amusingly, my instructor had been listening in on his handheld radio, and asked me about it when I landed. He introduced me to my examiner (actually, we had met before, another great idea), and had me explain what had just happend. As it turns out the whole situation worked great as I was able to demonstrate some of my hands on knowledge and my ability to handle the unexpected.

The Oral Exam

Now we got down to business. Many people fear the oral exam, but I was not one of them: I loved every bit of ground school, and even the dry writing of my aviation textbook couldn’t stop me from seeing the value in even the most mundane bits of knowledge required of us pilots. It may be beyond belief but I even loved reading the FARs, AIM (these two combined constitute a 1000+ page book of nothing but endless regulations written in an extremely precise language native only to lawyers and lawmakers), and anything else published by the government.

I say all this simply to make it clear what happens next. It must be a skill honed over years of practice, because within seconds of starting, my examiner had drilled into the weak spots of my knowledge, and I was quickly forced to rely on my reserve fallback answer of “I don’t know but I know how to look it up” for what felt like every question. In reality, I’m sure I demonstrated my knowledge just fine, as the examiner was quite happy overall. That said, I probably learned as much in my oral exam, which only lasted about an hour, as I did during some entire weeks of my primary training.

I can no longer remember too much of what I was asked, with one exception: we focused specifically on fuel consumption calculations for my planned cross country from Reid Hillview to Santa Barbara (KSBA). The reason here is that the Los Angelese area is just in range of the bay area in a 172 on a single tank of gas. My examiner made me very aware of the fact that whether or not a given long range trip can be completed can actually depend quite a bit on the outside temperature.

Why? A quick check of the cruise performance tables for a 172 gives the answer: on a cold day, at a normal cruising altitude of 6000 feet and 2500 RPM, we’ll be burning fuel at 8.5 GPH, whereas on a warm day, the same cruise would require only 7.5 GPH. It’s easy to simply plug a single fuel flow value into Foreflight and use it for all your flights. That’s not a problem, if you put in 8.5 GPH, because you’ll just sometimes make an extra fuel stop. But if you put in 7.5 GPH, you’ll get there only MOST of the time.

There’s also quite a bit of paperwork involved with a checkride, and once we got that out of the way, it was time to go flying.

The Practical Exam

After finishing up in my examiner’s office, he told me to head out to the airplane and do my preflight checks, while he finished with our paperwork. I took my time, hoping that he would be out soon and could watch first hand as I masterfully ensured our bird was ready to fly, but alas, I was sitting patiently in the plane by the time he arrived.

He told me to start up the plane, depart as if we were flying our cross country to Santa Barbara, and to show him a short field takeoff on our way out of Reid Hillview. Feeling confident, I got us started up and waiting in line to takeoff. Departing Reid Hillview is never chaotic, but it’s a busy airport with two runways, and I’m less familiar with it than Palo Alto, or some of the other airports where I did lots of training, so there was a lot for me to keep track of here. Once I had my takeoff clearance, I taxied out to the runway, pushed the throttle in all the way, and was halfway through the takeoff roll when I realized something terrible: I had forgotten to do a short-field takeoff!

Unlike most takeoffs, where you roll onto the runway and continue to accelerate, a short-field takeoff is designed to allow you to make maximum use of a short runway, and requires you stop on the runway to bring the engine to full power befor you even start moving. Realizing my mistake, I quickly tried to satisfy my instructor by explaining what it is I WOULD have done, had I not forgotten. In all likelihood, my examiner didn’t care much that I was more focused on simply safely departing a busy airport than demonstrating something I could easily have done later, but the damage was done: for the rest of the checkride I was nervous and agitated.

To make matters worse, the weather that day was terrible, at least compared to what you’d like on a checkride. A 2000 foot ceiling, rain, and light turbulence meant everything was a bit harder than it needed to be. I found myself making a silly mistake over and over - adjusting throttle in level flight instead of simply tweaking the trim - when I had never done such a thing during my training.

Soon, things went from bad to worse. Unsurprisingly, a critical skill of flying is figuring out where you are. Every private pilot checkride includes a specific test of what’s known as lost procedures: you are required to verify your position through some reliable means (visually with a landmark, or more likely through the use of radio navigation beacons called VORs. Ung GPS, which is what we all do every day after our checkride, is discouraged), and then, while flying, plot a course to another airport. Simple enough to do on the ground, plotting an accurate course using a pencil on a paper chart while flying a plane through turbulence is a bit more of a challenge.

We happened to be directly over South County airport at the time, and having trained there extensively (it’s where I first soloed), I was quite familiar with the area. Still, eager to show off my lost procedure skills, I began dutifully tuning the frequency for the nearby San Jose VOR into the navigational radio to confirm our position, until my examiner yelled at me: “We’re at South County”, he said, “Now fly me to Modesto”.

Between us and Modesto was the Diablo Range, with many 3000+ foot peaks, including Mt. Hamilton at 4400 feet not far away.

Given the clouds were still somewhere between 2000 and 3000 feet, it was clear to me we wouldn’t be flying there today. My examiner, however, had other plans, and told me to examine my chart more carefully to find a pass through the mountains. These mountains were unfamiliar to me, even though I had driven through them many times, so this did not sit very well with me.

Today, I would have no problem laying out clearly what I was and wasn’t comfortable with, but I’m sure I didn’t do a great job with it then. Sometimes saying clearly what you aren’t willing to do is better than demonstrating what you can do. In any case, my examiner, likely totally comfortable flying through some hills in his backyard when there are a few clouds, continued to firmly suggest we continue.

As I finished roughly plotting our course on the paper map, my examiner suggested I find a few waypoints along the way to plug in to the GPS. Embarrassingly, this caused me a little bit of consternation. Aerial navigation systems are great from taking you many places, but through a small mountain range at under 2000 feet is not one of them, and I spent what spent like a very long time trying to resolve this trivial dilemma, without much success.

Worse still, the one GPS waypoint I DID find on the paper map happened to be a seaplane base, and likely wasn’t in our GPS database at all. I wasted yet more time, and became even more unsettled struggling to enter this waypoint before simply continuing in the direction I knew would take us where we wanted to go.

After all that buildup, our journey into the mountains was short. I knew to expect my examiner, like my instructor during my training, to be constantly looking for a particularly inconvenient moment to reach over and pull the throttle to idle. I would then be implicitly tasked with demonstrating that, had the engine actually failed, I would be able to bring the plane in for a safe landing … somewhere.

This involves a few different things, such as going through a checklist of common engine issues to see if the engine could be restarted, and making a mock mayday call to air traffic control asking for assistance, as well as picking a suitable landing spot and then, without the flexibility of any engine power to ajust your approach, maneuver to come in for a safe landing.

Surrounded by unfamiliar mountains and low clouds, with not many suitable places to land, my examiner knew it was time. I was able to balance the workload of the clouds, the simulated engine failure, and the mountains relatively well, but I do remember being quite concerned here.

During training, it was usually more convenient to practice engine failures over an airport so we could actually make a power off landing. Even though I had done some very low approaches to farm fields, and knew I could possibly expect the same on my checkride, I spent a lot of mental energy wondring how low my examiner would have us fly.

Looking back I was clearly quite concerned over something trivial. I was passing all parts of the checkride without issue, but was focusing on what had gone wrong and was feeling bad for it. We flew back to South County airport where I finished the engine out procedures with a perfectly good power-off approach and landing, this time simulating an engine fire where you’re required to dive to extinguish the fire.

One last hurdle remained in the checkride at this point: practicing stalls. During my training we practiced any number of stalls: power off, power on, turning. Today, we did exactly one, but it was a tough one: a full power turning stall. While I had practiced both full power stalls and turning stalls, I actually don’t think I had ever done both at once.

During a checkride, if your examiner ever has to take control of the airplane when you are demonstrating a maneuver, you can generally assume you have failed. A power on turning stall is a delicate procedure and if not done correctly, can cause you to start to go into a spin, at which point the examiner would most definitely be taking control and your flight is going to come to an early conclusion.

Clearly, the stakes were high, and I wasn’t entirely sure I could get it right on the first try. Explaining all this to my examiner and finishing with “I’m sure I can do it, but…” lead to a predictable response: “then do it!”. I wasn’t getting out of it that easy.

I flew a perfect full power turning stall, and we headed back to Reid Hillview and landed with 1.6 total on the Hobbs.

The aftermath

A few weeks prior, when I flew my first solo laps around the pattern, I was on top of the world. If just a stepping stone on the way to my license felt good, actually getting my license should feel great, but it didn’t. I couldn’t stop thinking about the things that had gone wrong, even after my examiner and instructor had congratulated me.

Fortunately that feeling only lasted for a short while. After resolving to simply improve what I didn’t like about my fling on my checkride, I was feeling optimistic, and a few days later as I embarked on the 350 mile journey to San Diego, my first adventure as a private pilot, I was feeling spectacular.

Lessons learned

I’m sure I heard all this before my checkride, but in case other student pilots are better at taking things to heart than I, here’s some advice:

Your examiner will try to rush you. Unlike your instructor who is paid by the hour, your examiner gets paid a flat fee. He wants to go home. He also wants to see that you push back on someone rushing you. Eventually, someone who isn’t a pilot but is even more intimidating than your examiner will sit in that right seat and try to tell you how and where to fly the plane, and you have to be ready to say no.

Your examiner will try to make you nervous. Part of being a pilot is staying calm in an intense situation. Stay calm, focus on the task at hand and don’t worry if things haven’t gone perfectly. If you’re still flying the checkride isn’t over yet, so there’s no sense overthinking what you’ve already done.

Finally, make sure you fill out all the boxes on the forms sent to the FAA for your license. After a month and a half of anxious waiting for my license in the mail, I got a call from my examiner saying the FAA had returned my form because it was incomplete. A single checkbox was checked and it was sent back, but it cost me precious weeks of having my hard earned license to carry around, and almost required me to renew my temporary certificate.