I despise remote desktop access. The way it breaks keybindings is something that messes with my muscle memory pretty badly. That’s why I ended up working on Bazel’s macOS port at Google: I wanted to expand the ability to develop on Macs natively instead of relying on remote Linux workstations. But I now have to use remote Windows VMs for Azure development because that’s the way it is, so I’ll have to cope. Or will I?

As surprising as it may sound, recent versions of Windows ship with OpenSSH and, combined with PowerShell, it should be possible to operate a remote machine without the need for a graphical interface. (Except for editing files. I don’t understand how there is no built-in text-based editor. Bring back EDIT.COM!)

Furthermore, it seems like I’m free to choose any development environment I want (unlike at Google, where there were only a few blessed ways to operate) and some of my new teammates use VSCode. And this is a good thing because VSCode has excellent remote development abilities—including debugging—by means of a trivial SSH connection. Thus I want to see how far I can get by with using VSCode and a terminal, both connected to the remote VM via SSH.

Unfortunately, setting up SSH with key-based authentication wasn’t trivial. I spent about an hour figuring things out, in part because the official documentation is broken, and in part because all the “fixes” I found online in arbitrary forums were as misleading as you can imagine.

You can consider these to be my lab notes to get my Windows 10 laptop (build 19041) to connect to my VM on a Windows Server 2019 machine using SSH with key-based authentication. I’m sure I’ll need these in the future again.

Setting up SSH on Windows Server 2019

Setting up the server was easy. The instructions worked without a problem and I could redirect you to them, but I’m reproducing them here for completeness:

Install the SSH server optional feature by querying the exact package name and then installing it. Make sure to use the correct version printed in your case:
```
Get-WindowsCapability -Online | ? Name -like 'OpenSSH.Server*'
Add-WindowsCapability -Online -Name 'OpenSSH.Server~~~~0.0.1.0'
```
Start the sshd service and make sure it’s always available by configuring it as automatic:
```
Start-Service sshd
Set-Service -Name sshd -StartupType 'Automatic'
```

Confirm that the firewall rule to allow inbound SSH access on port 22 is configured. If not, configure it:

Get-NetFirewallRule -Name *ssh*
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22

(Optional, but really, why put yourself through cmd.exe.) Set PowerShell as the default shell:

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

So far so good. These straightforward instructions should allow inbound SSH connections to the Windows Server to work. You can probably do most of the above from the UI too if that’s your game, but I’m trying to figure out CLI-based administration as I go.

Setting up SSH on Windows 10

Install the SSH client optional feature by querying the exact package name and then installing it. Make sure to use the correct version printed in your case:

Get-WindowsCapability -Online | ? Name -like 'OpenSSH.Client*'
Add-WindowsCapability -Online -Name 'OpenSSH.Client~~~~0.0.1.0'

You can do this from the UI as well via the Optional features control panel and selecting OpenSSH Client.

But that’s it. There is no step two.

Setting up the SSH agent

This is where things get tricky because the official documentation is wrong. You see, the official documents talk about installing the OpenSSHUtils package… but that package seems to be gone. And before you think that this is my fault for not knowing how Windows works yet (which, hey, it was my assumption too)… there are plenty of months-old issues in the bug tracker describing this problem and I can’t understand how it hasn’t been fixed yet.

So, how do we get the SSH agent installed? We don’t: it’s already there (as part of the OpenSSH Client package, I suppose).

The problem is that, even after knowing that the agent is installed, enabling it as the instructions say doesn’t work:

PS C:\> Start-Service ssh-agent
Start-Service : Service 'OpenSSH Authentication Agent (ssh-agent)' cannot be started due to the following error:
Cannot start service ssh-agent on computer '.'.
At line:1 char:1
+ Start-Service ssh-agent
+ ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service], ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

This is a very helpful (not) error message: OK, the service cannot be started. But… why? No information. Seeing this error and the request to install the OpenSSHUtils package made me assume that the service was missing. But it wasn’t as I already mentioned above. Then, how come the start request failed? Well, the service’s status defaults to Disabled and thus requests to start it fail. To fix:

Set-Service -Name ssh-agent -StartupType 'Automatic'
Start-Service ssh-agent

Alright. One fewer problem. With this, ssh-add started working as expected.

Setting up the authorized keys

But there was one more problem. No matter what I did, the server would not accept my key: it would always fall back to password-based authentication even after the client correctly offered the key during connection establishment. A lot of the online “tips” talk about setting the right ACLs on the %UserProfile%\.ssh\authorized_keys file to ensure they are restrictive enough, and I sank a ton of time down this path without success.

My next guess was that key authentication was disabled for some super-strange reason, but I had to confirm this. I spent quite a while trying to find where the sshd_config file lived: I couldn’t guess its location from the documentation, and recursively searching through the contents of C:\ didn’t yield any results. Strange. “Maybe sshd has been patched to assume certain defaults so that there isn’t a need for an sshd_config file”, I thought.

It wasn’t until later that I found a random blog post with a clue: if your (remote) account happens to have administrator privileges (and mine does)… then the public key has to be placed in the special location %ProgramData%\ssh\administrators_authorized_keys, not under your personal %UserProfile%\.ssh\authorized_keys file. Wonderful, because the documentation had no mention of this.

OK! Let’s try that then. I went to create this file, but was greeted by:

PS C:\> mkdir ProgramData
mkdir : An item with the specified name C:\ProgramData already exists.
At line:1 char:1
+ mkdir ProgramData
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceExists: (C:\ProgramData:String) [New-Item], IOException
    + FullyQualifiedErrorId : DirectoryExist,Microsoft.PowerShell.Commands.NewItemCommand

Wait, what? The call to create the directory failed because the directory did exist? I certainly had not seen it earlier when looking directly at C:\. Alright, so… what’s inside?

PS C:\> cd ProgramData
PS C:\ProgramData> ls


    Directory: C:\ProgramData


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
... 🥁🥁🥁 ...
d-----        2020-05-10     22:43                ssh
...

Ta-da! The promised ssh directory. And guess what was within it? The sshd_config file I had been looking for earlier (which, by the way, confirmed the need to put administrator credentials into the administrators_authorized_keys special file).

This is when things clicked and explains why I hadn’t reached this point much earlier. Hidden files. I had forgotten about those. The C:\ProgramData\ directory is hidden by default, and that’s why none of my earlier file searches had encountered the sshd_config file.

But wait! The administrators_authorized_keys file doesn’t exist by default, and if you create it without giving it a second thought, things won’t work: sshd for Windows expects obnoxiously specific permissions and, if they are wrong, silently ignores the file. The permissions must grant Full Access to the System and the Administrators scopes alone. This is how the ACLs should look like:

PS C:\ProgramData\ssh> icacls .\administrators_authorized_keys
.\administrators_authorized_keys NT AUTHORITY\SYSTEM:(F)
                                 BUILTIN\Administrators:(F)

Successfully processed 1 files; Failed processing 0 files

And, with that, I was able to finally log in without having to type my password every single time 🙌.