OpenConnect and NetworkManager

In the past year, many universities including mine have moved to remote options, requiring a VPN connection for certain endpoints. In my case, Cisco AnyConnect is used and of course, the only client supported by the university is the proprietary one developed by Cisco. For a while, this was not really an issue thanks to the OpenConnect project, which works as a replacement VPN client and integrates with NetworkManager. I started running into a problem after some changes to the VPN by the university. This post serves as a guide for using these free projects to debug their own connections.

The Problem

In the beginning, everything ran perfectly, including the two-factor authentication portion of the connection. However, the tunnel groups were later modified to include web authentication as the primary groups, with the previous connections suffixed with “alt”. While there are options that might allow connections to these new primary groups, there is not yet integration with my desktop environment (KDE), which depends on NetworkManager. Even this was not really an issue after I tried out each option and learned that “alt” groups worked while the others didn’t. Then, at some point, I found that the “alt” groups disappeared completely. I resigned myself to finding another way to access university computing resources until I found the “alt” groups again on a different computer. A few days later, they disappeared there too. What was going on?

Initial Workaround

After playing around with settings to find any way to get it working, I ran into a strange pattern. The first time I changed any setting, the “alt” groups would show up for one connection, and then go back to the same problem on the second connection. So for the time being, with my limited mental energy for focusing on this minor inconvenience, I made an “edit” to the connection in my GUI every time before connecting to the VPN. At some point, while chatting with a friend with a setup similar to mine, I was reminded of this issue and I learned that he had the same issue, so we decided to do a little more investigation.

A Sanity Check

One important thing I learned while exploring bugs is to try to isolate them to the minimal possible component. In this case, if this problem is the result of KDE using NetworkManager which uses OpenConnect to connect to the VPN, what about OpenConnect on its own? What about a level above that? So I read through openconnect --help, adding in relevant options, and came up with this test:

sudo openconnect --protocol=anyconnect vpn.example.com --verbose

It worked every time! With this, I could move up a step to NetworkManager.

Finding the Cause

Finding the configuration files for NetworkManager was simple enough: /etc/NetworkManager/system-connections, but they aren’t helpful on their own. Using the autocompletion for nmcli, the command-line interface for NetworkManager, I explored its options and ended up using the following sequence to connect to the VPN with NetworkManager alone:

# Get a list of my saved connections
nmcli connection show
# Connect to the VPN
nmcli connection up "My VPN"
# Follow menu for group selection and two-factor authentication
# Disconnect from the VPN
nmcli connection down "My VPN"

With it, I could perfectly reproduce my problem. After deleting the connection from NetworkManager config (and running nmcli connection reload), I could make a connection, connect to it, and run into issues the second time I tried connecting. After saving a copy of the nmconnection config file before and after the first time I used the connection, I diffed them against each other and found a few added settings, including one for xmlconfig, encoded in base64. Its contents contained a list of groups to be shown to the user and their internal names to be sent to AnyConnect. This was it! So it turns out the problem was not OpenConnect, not NetworkManager, but rather a proprietary protocol and a university assuming that every client supports it. Who would have guessed?

Looking for a Solution

I needed to prevent NetworkManager from changing the connection, specifically the xmlconfig key. Since it runs as root, permissions are useless in this regard; it simply overwrites the old protected version with a new file. Unfortunately, the xmlconfig key was poorly documented, and my search engine was not really helping. I knew that xmlconfig was mostly self-explanatory, mainly consisting of base64 decoded xml configuration. This configuration is added by the VPN server on first connect, and here I noticed that the list of available VPN groups was being overridden by the config provided by the server. I also noticed a few forum posts referencing xmlconfig-flags, but all mentions were limited to config dumps or logs, with no discussion of the key itself. Clearly, it’s used to somehow modify the behavior of xmlconfig, but without documentation, I can really only try out values until something works/breaks. Instead, I went back to the NetworkManager documentation and searched again, finding this tidbit that I skimmed past on my first read through the outline. xmlconfig was a secret property for the vpn, so xmlconfig-flags must be the associated flags, with the possible values listed. I deleted xmlconfig, added xmlconfig-flags=2 to my config, ran nmcli connection reload to reread the new configuration, and tried out my new connection. As promised by the documentation, it worked the first time, and every time after that. Finally!

[vpn]
xmlconfig-flags=2

[vpn-secrets]
#Delete xmlconfig from here

Conclusions

In retrospect, it seems obvious that NetworkManager must have been modifying the config in some way during that first connection, though from my limited GUI view in KDE, I couldn’t see any of that. It reminds me of the sometimes annoying fact that (almost) everything on a free system is fixable, if you look deep enough. Since I enjoy doing exactly that, it’s almost like every time I sit down to work on my machine, I’m pulled towards having fun digging through one of these issues and fixing it. Eventually, I’ll run out of things to fix, right?