Jon's Notes

More bits more places

Ubiquiti AirFiber 5X: The Perfect Medium-capacity Backhaul

Ubiquiti is going to announce their AirFiber 5X radio tomorrow at the Animal Farm User Group (AFMUG) meeting in Salt Lake City tomorrow. The quick start guide was posted early to their web site and gives us a sneak peak. From their teaser emails (that mention price and nothing else) it has a MSRP of $399 per radio (without dishes).

Essentially it’s a connectorized version of the AirFiber 5. It uses airFiber branded dishes or RocketDishes with a retrofit kit. Dishes are cross-pol instead of V/H. It supports channel sizes from 10Mhz to 50Mhz.

I’m pretty excited about this radio. It fits a niche between the Rocket & ePMP radios on the low end and licensed microwave on the high end. This is a market where Cambium’s PTP3/5/600 radios have reigned supreme but at a significantly higher price point. The ability to use large dishes means it has a fair chance to actually hit the marketing numbers of 500Mbps HDX or 200Km range.

Also, it doesn’t hurt that this is coming from their AirFiber group out of Chicago. They seem to have a better track record of delivering products with fewer teething problems relative to the AirMax M/Ti/AC line. I assume that’s because their people brought along some of that Motorola engineering discipline when they left the Canopy group. In any case, I expect this to be a radio that I can drop in to replace a couple wonky AirMax ac links and use instead of upgrading to PTP500 in some other cases and have it actually work without fuss.

Ubiquiti AirFiber24HD: Solid Improvement With Excessive Hype

Ubiquiti announced their AirFiber24HD today. It looks like a solid incremental improvement to the classic AirFiber24 with a roughly 250 Mbps (FDX) increase in throughput.

Key differences

Feature AF24 AF24HD
Maximum Modulation:   64QAM   256QAM
RX Gain: 38 dBi   40 dBi
Housing: Plastic Aluminum
Maximum Throughput (FDX):    774 Mbps   1Gbps
Link MSRP: $3,000 $6,000

Excessive hype

A unlicensed-frequency radio that can do bog standard 1Gbps FDX is actually a big deal. You can use these for short links between buildings and achieve just as much throughput as if you trenched fiber and used normal SFPs. No more caveats about actually only having 700Mbps of throughput.

Unfortunately with Ubiquiti a decent improvement isn’t good enough. Every product or service has to be a revolutionary disruptive breakthrough for the masses. This was on full display with the AF24HD launch where apparently line-rate 1G wasn’t good enough.

That’s right. The headline throughput number for a 1Gbps FDX (2Gbps HDX/aggregate) radio is shown as 20 Gbps because that’s the theoretical aggregate throughput of devices that may be connected behind the link. If that was a thing I’d be bragging about my network having over 3Tbps of throughput, except we don’t because that would be dishonest.

Cisco Site-to-site VPN With Outside VRF

TL;DR: Outside interface is a VRF, inside on global, must Specify VRF on crypto keychain.

I recently set up a IPSec VPN between two Cisco routers where the Internet-facing interface was in a VRF and the tunnel interface was in not in a VRF.

The VTI was configured in the usual way:

VTI Configuration
interface Tunnel0
 description VPN: site2 via provider A
 ip address
 ip mtu 1438
 keepalive 5 3
 tunnel source
 tunnel mode ipsec ipv4
 tunnel destination
 tunnel vrf Outside-A
 tunnel protection ipsec profile vpn-to-site2-via-provider-a

With the usual crypto configuration:

IPSec configuration
crypto isakmp policy 1
 encr aes
 authentication pre-share
 group 2
 lifetime 3600
crypto isakmp key SecretKey address
crypto ipsec transform-set vpn-peer esp-aes esp-sha-hmac
crypto ipsec profile vpn-to-site2-via-provider-a
 set transform-set vpn-peer

but the tunnel wouldn’t come up.

“debug crypto isakmp” showed a error to the effect that a preshared key wasn’t configured or couldn’t be found for the peer. This was puzzling because there was clearly a preshared key defined.

There weren’t any messages in the debug logs about incoming connections or policy mismatches so I assumed that the problem was occurring before the router tried to communicate with its peer or it wasn’t seeing traffic from the peer.

From the documentation I found it seemed like the “tunnel vrf Outside-A” statement in the VTI was all that was necessary for IOS to know that the outside was in a VRF (the tunnel endpoint is in a VRF so the protection profile must be on the VRF as well). This turned out to not be the case. The vrf statement on the VTI is just for the tunnel. You need to specify the VRF in the crypto keyring definition otherwise the router isn’t looking for VPN traffic in the VRF.

I changed the crypto configuration to use a keyring and specified the outside vrf on the keyring definition:

Working IPSec configuration (VRF keychain)
crypto keyring vpn-peer vrf Outside-A
  pre-shared-key address key SecretKey
crypto isakmp policy 1
 encr aes
 authentication pre-share
 group 2
 lifetime 3600
crypto isakmp profile vpn-to-site2-via-provider-a
   keyring vpn-peer
   match identity address
   keepalive 10 retry 5
crypto ipsec transform-set vpn-peer esp-aes esp-sha-hmac
crypto ipsec profile vpn-to-site2-via-provider-a
 set transform-set vpn-peer

and now it all works. Of course now that it’s working I find a Cisco doc on New Version Site-to-Site Configuration that shows this.

Power & IO Connector for Cisco 2955

If you bought a used Cisco 2955 switch and it didn’t come with the power/IO terminal block connector, it’s a Phoenix Contact 1827761 (Mouser part number 651-1827761).

IPv6 via 6rd on Charter With Cisco IOS

Charter does not yet offer native IPv6 but they do have IPv6 via 6rd tunnels for testing. It mostly works but I’ve seen some weirdness with various v6 destinations. Thumbnail images breaking on Facebook periodically and things like that.

Cisco IOS Configuration

I have this working on a Cisco 881W running IOS 15.3(2)T with the advsecurity license.

Enable IPv6:

ipv6 unicast-routing
ipv6 cef

Set up a DHCPv6 pool:

ipv6 dhcp pool home-pool
 dns-server 2607:F428:1::5353:1
 dns-server 2607:F428:2::5353:1

Tunnel interface:

interface Tunnel0
no ip address
no ip redirects
ipv6 enable
 tunnel source FastEthernet4
 tunnel mode ipv6ip 6rd
 tunnel 6rd prefix 2602:100::/32
 tunnel 6rd br

Routes & general-prefix:

ipv6 general-prefix DELEGATED_PREFIX 6rd Tunnel0
ipv6 route 2602:100::/32 Tunnel0
ipv6 route ::/0 Tunnel0 2602:100:4472:A501::1

LAN interface:

interface Vlan1 
    ipv6 address DELEGATED_PREFIX ::1/64
    ipv6 nd other-config-flag
    ipv6 dhcp server home-pool

Basic Configuration in Ciena SAOS

Ciena makes Ethernet & Optical switches for the service provider market. There’s essentially zero publicly available documentation available which is a shame because they make some interesting equipment and as carriers upgrade there’s a lot of used equipment available that smaller providers can put to good use. I’m not sure why it’s traditional for telco-oriented companies to be stingy with their documentation but it’s annoying.

I have some LE-311v, 3911, and a 3940 here for a project so here’s my notes. They are applicable to all devices as far as I know unless otherwise noted.

This also applies to devices running World Wide Packets LE-OS. Ciena acquired WWP and built their packet Ethernet offerings on tech from the acquisition.

Configuration style

CLI is subject-verb instead of verb-subject (think “interface show” not “show interface”). “configuration” enters config mode.

Configuration is a blend of Cisco & Juniper Style:

  • JunOS: set system hostname lab
  • Cisco: hostname lab
  • Ciena: system set host-name lab

You can view the current configuration with:

configuration show

The configuration is displayed as a sequence of commands that are entered to achieve the desired state when you are starting from the default state.

You save the configuration with:


You have to be in configuration mode to save the config but you don’t have to be configuration mode to change the configuration.

You can navigate the configuration in a sort of tree format as on a Juniper or Mikrotik device. For example you can enter “interface remote show” to view management interface status or you can enter “interface remote” and then “show” or “set” as appropriate.

Hardware reset to factory default

I found someone else’s notes on configuring Ciena devices that covered this and got me started.

On the LE-311v and 3940 there’s a pinhole reset button. There’s three options:

  • Less than 2 seconds: soft reboot
  • Between 2 and 10 seconds: factory default
  • More than 10 seconds: hard reboot.

Management access

Default users

There are two default users: su & user. su is the superuser with a default password of wwp. user doesn’t have a password.

Serial console

My devices have a DB9 serial port. You’ll need a null modem cable for this. The 3911 has DB9 port visible from the user compartment but if you remove the inner cover to the carrier compartment you’ll see that it’s just a jumper to a Cisco-style cat5 console port.

The serial ports use the typical console settings of 9600 baud, 8 data bits, 1 stop bit, no parity (9600 8N1).

Local Management Ethernet port

The LE-311v & 3940 have a ethernet management port for local access. The local access part is important. You can plug your laptop into it. You can’t put a default gateway or other route on it and hook it up to your out of band network.

Management IP:

You can see current settings with:

interface show local

For example:

lab> interface show local

+-----------------------------------  local -----------------------------------+
| Parameter            | Operational       | User            | DHCP            |
| IP Address           |    |  |         |
| Subnet Mask          |     |   |         |
| Index                | 1                 |                 |                 |
| Admin State          | Enabled           |                 |                 |
| Oper State           | Disabled          |                 |                 |
| Broadcast Address    |    |                 |                 |
| MAC Address          | 00:03:18:8b:cf:5e |                 |                 |
| VLAN                 | 0                 |                 |                 |
| Priority             | 0                 |                 |                 |
| MTU                  | 1500              |                 |                 |

Out of the box you should be able to plug in, set you a IP in the subnet on your laptop, and telnet in.

Managing users

You can see the current users with:

user show

For example, with default users:

CN 3911> user show
+--------- USER ACCOUNT TABLE -----+-------+
| Username         |   Privilege   | Flags |
| su               | super         |  DP   |
| user             | limited       |  D    |

Removing the read-only user:

user delete user user

CN 3911> user delete user user
CN 3911> user show
+--------- USER ACCOUNT TABLE -----+-------+
| Username         |   Privilege   | Flags |
| su               | super         |  DP   |

Adding a management user:

user create user labadmin access-level super password blah123

Changing the password for the built-in user “su”:

user set user su password blah123

Software version and licenses

Check OS version with:

software show

For example:

lab>  software show

| Software Information Slot #01                                                |
| Installed Package   : leos-04-08-00-0066                                     |
| Running Package     : leos-04-08-00-0066                                     |
| Running Kernel      : Build 4529 10:16:02 Sep  2 2009 C:\AR\BUILD_4529\      |
| Running Application : Build 6830 04:54:20 Mar 24 2011 C:\AR\BUILD_6830\      |
| Running MIB version : 02-08-00-0013                                          |
| Release Status      : CA                                                     |
| Last command file: unknown                                                   |
| Last configuration file: unknown                                             |

Check feature licenses with:

software license show

For example:

CN 3911> software license show
|                              |       | License |       |  Sequence  |  Days |
| Feature Name                 |Status | Domain  | Admin |   Number   | Remain|
| Base-Features                |enable |         | 00000 | 0000000000 | 00000 |
| Advanced-Security            |disable|         | 00000 | 0000000000 | 00000 |
| Advanced-Ethernet            |disable|         | 00000 | 0000000000 | 00000 |
| Advanced-OAM                 |disable|         | 00000 | 0000000000 | 00000 |
| Feature Name                 | License Key                                  |
| Base-Features                |                                              |
| Advanced-Security            |                                              |
| Advanced-Ethernet            |                                              |
| Advanced-OAM                 |                                              |


I’m testing with a WWP-branded LX SFP and a Hoyos Consulting-branded GLC-T SFP that identifies as Cisco SFP.

My LE-311v doesn’t show a difference between the SFP modules. The 3911 shows a difference (GLC-T in 9, LX in 10):

CN 3911> port show

| Port Table      |           Operational Status            |  Admin Config   |
| Port   | Port   |    |  Link State  |    |   |       |Auto|    |       |Auto|
| Name   | Type   |Link|   Duration   |XCVR|STP| Mode  |Neg |Link| Mode  |Neg |
| 1      |10/100/G|Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |1000/FD| On |
| 2      |10/100/G|Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |1000/FD| On |
| 3      |10/100/G|Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |1000/FD| On |
| 4      |10/100/G|Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |1000/FD| On |
| 5      |10/100/G|Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |1000/FD| On |
| 6      |10/100/G|Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |1000/FD| On |
| 7      |10/100/G|Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |1000/FD| On |
| 8      |10/100/G|Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |1000/FD| On |
| 9      |Uncertif|Down|   0d 0h 0m 0s|UCTF|Dis|       | On |Ena |Auto/FD| On |
| 10     | Gig    |Down|   0d 0h 0m 0s|    |Dis|       | On |Ena |Auto/FD| On |

You can see the type of transceiver with “port xcvr show”:

CN 3911> port xcvr show

|    |Admin| Oper|                                       |Ether Medium &  |Diag|
|Port|State|State|      Vendor Name & Part Number        |Connector Type  |Data|
|9   |Ena  |UCTF |OEM SFP-T Rev11.0                      |1000BASE-T/LC   |    |
|10  |Ena  |     |WORLDWIDEPACKETS XCVR-010Y31 Rev10     |1000BASE-LX/LC  |    |

The port will still come up and pass traffic with a uncertified transceiver.

A Zone Named Management on a Juniper SRX

TL;DR: Don’t name a zone “management” on a Juniper SRX (11.4R7.5).

One of my on again, off again projects involves moving a datacenter management network with devices on public IP space with ACLs for protection to private IP space with a zone-based firewall (Juniper SRX240).

When I last touched it I ran into a problem where one zone would not pass traffic even though it had identical rules to a different zone that worked. It happens that the zone that didn’t work was named “management”.

Earlier today I was browsing around and found a article that mentioned the management functional zone which got me wondering if there was something special with naming a zone “management”. I thought that didn’t make sense since the functionality comes from the functional-zone tag not the zone name. Cue more unsuccessful searching for any mention of reserved zone names. Eventually I decided to just rename the zone and see what happened. One quick rename statement and “management” became “dcn-mgmt” and everything started to work.


Then I came across a post to J-NSP that mentions management being a reserved keyword for zones. Oh. That explains it.

Cisco CSR 1000V on SmartOS

In pursuit of some Friday Evening Fun I decided to try to get a Cisco CSR 1000V VM running on SmartOS. I was successful. It runs. It seems to run quite well.

First, a overview of the systems involved since the CSR1000V is fairly new and SmartOS is relatively uncommon.

The Cisco Cloud Services Router 1000V (CSR1000V) is Cisco’s entry in the virtualized router field. It runs IOS-XE and brings many of the handy features that you can find on the ASR1k platform into the cloud so you can do fun and exciting things with MPLS, VPLS, LISP, etc. This is a pleasant change from Vyatta which lacks any sort of MPLS support and Mikrotik which blackholes traffic for no good reason.

SmartOS is a hypervisor based on Illumos which is in turn a fork of OpenSolaris which was next-gen Solaris before Oracle borged Sun. SmartOS has all the wonderful features of OpenSolaris (ZFS, Zones, network virtualization) and adds support for KVM. There are some interesting design choices in SmartOS that turn out to be pretty cool once you get used to them, in particular: Netboot only (or USB if you are desperate) with all local disk used for VMs and VM Management by JSON config files.

I started by downloading the CSR1000V ISO (csr1000v-universalk9.03.10.00.S.153-3.S-ext.iso) from CCO and then uploading it to a server in the datacenter. I followed the instructions for creating a KVM VM on SmartOS except I copied the CSR1000V ISO over instead of a Debian ISO. Installation was smooth but after boot and initial configuration I wasn’t able to reach the VM. It turns out that SmartOS has some decent network security policies by default so I had to add "allow_ip_spoofing": true to the VM config before traffic started to flow.

After the CSR was installed and configured for basic connectivity I set it up as a route reflector and sent it the partial BGP feed that I run on our old Cisco 6500 series routers (~100,000 routes). sho proc cpu hist showed up to 40% CPU usage while it was loading. After that I added a table-map containing a deny statement to the BGP config to prevent it from wasting time trying to update the RIB. This reduced loading time to two seconds with no response lag on the terminal.

My feeling in general on the CSR is that it simply has too large of a footprint for general cloud routing & VPN duties. Vyatta gets the job done with a lot less than 4G of RAM and four CPU cores. If your needs go beyond simple routing & VPN then the CSR becomes more appealing. So far it seems well suited to the route reflector role but I have more testing to do before I commit to it.

If the CSR holds up to testing I hope to run a pair of them in place of the aging 7200s that are currently acting as my route reflectors. Cisco’s recommended route reflector platform is the ASR1k but as a small ISP I just can’t justify the cost on something that doesn’t generate revenue. If this works I can stay with Cisco instead of cobbling something together on a different platform.

Configs used are available as Github Gists: VM config json and CSR config.

Home Directories and SMF Error 96

tl;dr: make sure the user’s home directory exists.

The other day I was setting up a Tomcat instance (for Atlassian Stash) to run under SMF on SmartOS with a alternate user and low ports (on 443 without root) and the service kept going into maintenance state with a reason of Start method exited with $SMF_EXIT_ERR_CONFIG and a service log message of

[ Jul 12 20:45:38 Enabled. ]
[ Jul 12 20:45:39 Executing start method ("/opt/stash/bin/"). ]
[ Jul 12 20:45:39 Method "start" exited with status 96. ]

Everything I found when searching the web referred to config files missing and other sorts of things that are fairly obvious from log messages, except there wasn’t anything relevant in the logs. I replaced the start method with a shell script that printed a message to a file in /tmp and the file wasn’t updated. That got my focus back on SMF. Why would SMF not run a start method and say it was a configuration error?

After a few rounds of checking the SMF manifest and head banging I remembered that I didn’t specify -m (create home directory) when I created the user that the service was going to run under. I changed the user’s home directory from null to the app’s working dir in /var. The service started up after that.

I’m not sure why neither SMF’s verbose documentation nor the extended documentation online that is referenced in the error mentioned that when you start a service as a user other than root that user has to have a home directory. Instead everything points toward an application configuration error.

For reference, the SMF manifest I’m using to start Tomcat (as bundled with Atlassian Stash) is available as a Github Gist: stash.xml.

Mikrotik DHCP & FreeRADIUS With a Hint of Unlang

Mikrotik has two particularly useful subscriber management features built into the DHCP server on RouterOS. First, If you set the rate-limit property on a DHCP lease it will dynamically manage a Simple Queue to enforce the rate limit. Second, it will authenticate to a RADIUS server using the DHCP Client’s MAC address as the username. The RADIUS server can reply with the IP address, pool, or traffic shaping paramaters.

We use this combination to rate limit customers on Ubiquiti AirMax equipment since Ubiquiti is somewhat unaccommodating towards OSS/BSS integrations. When we set up our system we envisioned three states of a customer so far as DHCP RADIUS is concerned:

  1. Users that aren’t known to the system don’t get a IP address. We do this by setting the default address-pool on the DHCP server instance to static-only.
  2. A user that is known to the system and has a active account is assigned a DHCP Pool of ubnt-cust (for Ubiquiti customer).
  3. A user with a delinquent account is assigned to the ubnt-deact (for Ubiquiti deactivated) pool. Users in this pool are assigned a RFC1918 IP address and redirected to a splash page.

You might think that we would want unknown users to get a splash page, and we do on our Cambium Canopy network which comes with strong AAA support. On the Ubiquiti network we assume that anyone we don’t know about that manages to connect to our network is hostile and want to give them as little data as possible. Every little obstacle helps. I may relax this after exploring the fairly new RADIUS authentication support in Ubiquiti AirOS.

Anyway, We ended up with a Mikrotik router at each AirMax tower running RADIUS-backed DHCP. Our B/OSS provisions our existing FreeRADIUS system with Ubiquiti device MAC addresses as usernames and puts them in a group with a Mikrotik-Rate-Limit attribute that matches the customer’s speed package. Everything is working great until…

We started to retrofit existing towers with AirMax. These towers were often connected in a star topology, as opposed to our previous expansion where we build out in a line and then expand to the side to make a box. Since we had a central tower it didn’t make sense to deploy a router at every spoke site1. Of course each tower (at a minimum) has its own VLAN to make management easier and contain bad behavior. This presented a problem since the Mikrotik DHCP server can’t dynamically select the correct subnet based on the ingress interface (unlike the Cisco IOS or ISC DHCPd servers). The solution is to create a separately named pool for each interface/tower and get the RADIUS server to provide the appropriate pool in the RADIUS reply. How do we do that? Well, the “easy” way would be to manually set a different pool for each tower in RADIUS. The problem is that is more work on a ongoing basis.

What to do? Well, the Mikrotik DHCP server sends the DHCP server name in the RADIUS request as the Called-Station-ID. On all our towers we had been setting this to ubnt-server. On routers serving multiple towers I set the server name to ubnt-server-TWRID2. Then I setup the pools for each tower as ubnt-cust-TWRID and ubnt-deact-TWRID. In the FreeRADIUS server config I used unlang to extract the TWRID portion of each server name in the request and append it to the pool in the reply.

in sites-enabled/default:

post-auth {
 # Rewrite Mikrotik IP Pool assignments for routers with multiple pools
 if (request:Called-Station-Id =~ /^ubnt-server-(.*$)/) {
  update reply {
   Framed-Pool := "%{reply:Framed-Pool}-%{1}"

Example Mikrotik DHCP server config:

/ip pool
add name=ubnt-cust-HRT2 ranges=199.X.X.2-199.X.X.14
add name=ubnt-deact-HRT2 ranges=
add name=ubnt-deact-HRT1 ranges=
add name=ubnt-deact-SLNG1 ranges=
add name=ubnt-cust-HRT1 ranges=199.X.X.18-199.X.X.30
add name=ubnt-cust-SLNG1 ranges=199.X.X.34-199.X.X.46

/ip dhcp-server
add add-arp=yes authoritative=yes disabled=no interface=v100-ether7 \
    lease-time=30m name= ubnt-server-HRT2 src-address=199.X.X.1 use-radius=yes
add add-arp=yes authoritative=yes disabled=no interface=v100-ether8 \
    lease-time=30m name=ubnt-server-HRT1 src-address=199.X.X.17 use-radius=yes
add add-arp=yes authoritative=yes disabled=no interface=v201-ether9 \
    lease-time=30m name=ubnt-server-SLNG1 src-address=199.X.X.33 use-radius=yes

[1] Since you aren’t doing subscriber management at the tower you need some kind of rate limiting in the CPE as a safety net to prevent subscriber-originated UDP floods saturating the tower backhaul on their way to the router.

[2] TWRID is the unique ID for each tower. We base it off of psudo-CLLI code. SLNG1, HRFD2, WHWR1, etc.