How to Write an Ansible Filter – Dynamically Configuring Interface Descriptions in a Multivendor Environment

[Special thanks to Jason Edelman for his kind help and support during all the stages of this article. I’d have not written this without his precious suggestions and hints!]

It’s been a while since my last post. I’ve been really busy lately with work, study and other projects and I have not found the time to write something here, even if I wanted to post this article at least one month ago. Anyway, welcome back to me ūüôā


 

Back to the start of December, Jason Edelman published another awesome blog post on Network Automation and Ansible. There he wrote an Ansible playbook to:

  1. Discover the device using snmp_device_version from Patrick Ogenstad.
  2. Discover the LLDP neighbors of the device using ntc-ansible.
  3. Configure the interface descriptions using the nxos_interface Ansible module.

This is pretty damn cool! Here you can see the whole magic playbook:

---

  - name: AUTO CONFIGURE PORT DESCRIPTIONS
    hosts: cisco
    gather_facts: no
    connection: local

    tasks:

      - name: GET SNMP DISCOVERY INFORMATION
        snmp_device_version: host={{ inventory_hostname }} community=networktocode version=2c
        tags:
          - snmp
          - neighbors

      - name: GET LLDP NEIGHBORS
        ntc_show_command:
          connection=ssh
          platform={{ ansible_device_vendor }}_{{ ansible_device_os }}
          template_dir='/home/ntc/library/ntc-ansible/ntc_templates'
          command='show lldp neighbors'
          host={{ inventory_hostname }}
          username={{ un }}
          password={{ pwd }}
        register: neighbors
        tags: neighbors

      - name: CONFIGURE PORT DESCRIPTIONS USING NEIGHBOR DATA
        nxos_interface:
          interface={{ item.local_interface  }}
          description="Connects to {{ item.neighbor_interface }} on {{ item.neighbor }}"
          host={{ inventory_hostname }}
          username={{ un }}
          password={{ pwd}}
        with_items: neighbors.response

If you run this, you’ll see Ansible dynamically configuring your Nexus interfaces. How cool is this? ūüôā

It’s that cool that I wanted to try it in a multivendor environment, adding a couple of Arista switches. So I did, and this is the playbook I used:


---
 - name: Auto-configure port descriptions
   hosts: all
   gather_facts: no
   connection: local

   tasks:
     - name: GET SNMP DISCOVERY INFORMATION
       snmp_device_version: host={{ inventory_hostname }} community=networktocode version=2c

     - name: GET NEIGHBOR INFORMATION
       ntc_show_command:
         connection=ssh
         platform={{ ansible_device_vendor }}_{{ ansible_device_os }}
         template_dir='./ntc_templates'
         command='show lldp neighbors'
         host={{ inventory_hostname }}
         username={{ un }}
         password={{ pwd }}
       register: neighbors

     - name: AUTO-CONFIGURE PORT DESCRIPTIONS FOR ARISTA
       eos_interface:
         name={{ item.local_interface }}
         description="Connects to {{ item.neighbor_interface }} on {{ item.neighbor }}"
         connection={{ inventory_hostname }}
       with_items: neighbors.response
       when: type == "arista"

     - name: AUTO-CONFIGURE PORT DESCRIPTIONS FOR CISCO
       nxos_interface:
         interface={{ item.local_interface }}
         description="Connects to {{ item.neighbor_interface }} on {{ item.neighbor }}"
         host={{ inventory_hostname }}
         username={{ un }}
         password={{ pwd}}
       with_items: neighbors.response
       when: type == "cisco"

The only difference, compared to the one from Jason, is the additional task to support Arista EOS and the¬†when¬†conditional to select the correct task based on vendor type. Anyway, when I run this, it didn’t work at all and below you can see the error it returns!

Let’s try to understand why focusing only on Arista now.

 

fail

Why did it fail? The reason is pretty explicit if we read the error message from Ansible:

  • msg: invalid interface Et1
  • msg: invalid interface Et2

The problem is that Arista interfaces need to be entered using their full name. Neither Et or Eth are accepted and so the playbook fails as soon as it tries to access interface to configure descriptions.

Where do Et1 and Et2 come from? Let’s investigate a little bit more.

If we run the current playbook with a¬†-v¬†check, we’ll be able to see all the params returned from run:

check

Here we can see how Ansible discovers what kind of device are these in the first task. But here we are more interested on the second one. In fact, we can notice that local_interface¬†param is set to¬†Et1¬†and¬†Et2. These values are returned by¬†eos_interface and that’s where I’m gonna try to work to solve the issue.

Ansible Filters

From Ansible: Up and Running:

Filters are a feature of the Jinja2 templating engine. Since Ansible uses Jinja2 for evaluating variables, as well as for templates, you can use filters inside of {{ braces }} in your playbooks, as well as inside of your template files. Using filters resembles using Unix pipes, where a variable is piped through a filter.

What I want to do, is to build a filter to change what Ansible returns.

The basic syntaxt of a filter is this:

{{ some_variable | some_filter }}

Here, some_variable is a variable passed or returned back from Ansible. Then, we apply¬†some_filter¬†to our variable and the result is given back to us.¬†Ansible already have lot of built-in filters and you can take a look at some of them here. But you can also develop your own custom filter! ūüėČ

normalize_interface

This is the content of my normalize_interface.py file, which will become my filter:


from ansible import errors

interfaces = {
        'arista':   {
                    'et': 'Ethernet',
                    'ma': 'Management',
                    'lo': 'Loopback'
                },
        'hp':       {
                    'gi': 'GigabitEthernet',
                    'te': 'Ten-GigabitEthernet',
                    'fo': 'FortyGigE',
                    'lo': 'LoopBack',
                    'br': 'Bridge-Aggregation',
                    'ro': 'Route-Aggregation',
                    'tu': 'Tunnel'
                }
        }

def _get_interface(interface):
    splitted_interface = interface.split(' ')
    if len(splitted_interface) == 2:
        int_number = splitted_interface[-1].strip()
        int_type = splitted_interface[0].strip().lower()
        if len(int_type) > 2:
            int_type = int_type[0:2]
    else:
        for i in range(0, len(interface)):
            if interface[i].isdigit():
                int_type = interface[0:i].lower()
                if len(int_type) > 2:
                    int_type = int_type[0:2]
                int_number = interface[i::]
                break
    return (int_type, int_number)

def normalize_interface(interface, vendor):
    try:
        interface_type, interface_number = _get_interface(interface)

        interfaces_dict = interfaces[vendor]
        fixed_interface_type = interfaces_dict[interface_type]

        fixed_interface = fixed_interface_type + interface_number
        return fixed_interface
    except Exception, e:
        raise errors.AnsibleFilterError(
                'normalize_interface plugin error: {0}, interface={1},'
                'vendor={2}'.format(str(e), str(interface), str(vendor)))

class FilterModule(object):
    ''' A filter to fix interface's name format '''
    def filters(self):
        return {
            'normalize_interface': normalize_interface
        }

The¬†normalized_interface¬†function defines the Jinja2 filter. It accepts 2 params,¬†interface¬†and¬†vendor¬†which are passed in different ways (we’ll be back on this soon).

More from Ansible: Up and Running:

The FilterModule class defines a filters method that returns a dictionary with the name of the filter function and the function itself. The FilterModule class is Ansible-specific code that makes the Jinja2 filter available to Ansible.


{{ item.local_interface | normalize_interface("arista") }}

The above line is an examble of how to call the filter against an interface. As I said, the¬†normalize_interface¬†accepts 2 args: the first, item.local_interface,¬†is passed automatically using the pipe “|“, while the second one, arista,¬†is passed explicitly.

After the filter is called as above,  the FilterModule class calls the proper function. Here, the first thing we do is to call another function, _get_interface, to split interface types and numbers.

As you can see, I’ve defined one dictionary with two nested dictionaries. Here I added support for Arista and HP too. The goal should be to write something general instead of Arista specific, so we can add even more vendors.

Once we splitted our interface with _get_interface, we extract vendor information and then we use the above dictionary to normalize our interface type. After this, the fixed interface is returned.

Now, let’s try our filter:

- name: CONFIGURE PORT DESCRIPTIONS USING NEIGHBOR DATA
        eos_interface:
          name={{ item.local_interface | normalize_interface("arista") }}
          description="Connects to {{ item.neighbor_interface }} on {{ item.neighbor }}"
          connection={{ inventory_hostname }}
        with_items: neighbors.response

Results:

success

As I said before, we may want to make it even more general. We could choose another approach, instead of keep adding vendor dictionaries.

In fact, we can notice how most of the interface’s name are quite similar even between different vendors: for example, an interface whose name starts with¬†et¬†is 99.9999% an ethernet interface, regardless the vendor. So, we could also have a big dictionary like this:

interfaces = {
        'as': 'Async',
        'br': 'Bri',
        'di': 'Dialer',
        'et': 'Ethernet',
        'fa': 'FastEthernet',
        'fd': 'Fddi',
        'fo': 'FortyGigE',
        'gi': 'GigabitEthernet',
        'hs': 'Hssi',
        'lo': 'Loopback',
        'ma': 'Management',
        'mg': 'Mgmt',
        'nu': 'Null',
        'po': 'Port-Channel',
        'ro': 'Route-Aggregation',
        'se': 'Serial',
        'te': 'TenGigabitEthernet',
        'tu': 'Tunnel',
        'vl': 'Vlan',
        'vx': 'Vxlan',
    }

Anyway, we’d need to pay attention to some limit cases: for example, a 10G port is a¬†TenGigabitEthernet¬†for Cisco but it’s a¬†Ten-GigabitEthernet¬†for HP. So, we may want to work a little bit more on this ūüėČ

So, what do you think about this? How would you make it better and more general? ūüôā

 

 

So you want to start with Network Automation…

As we all know, things in networking are changing rapidly and so is changing the needed skillset for those who manage networks.

I’m definitely not an expert (I’m far from it) but lately many people asked me how to start with Network Automation. Now I’ve just received a message from a LinkedIn’s friend asking for something like this and I suddenly realized this would be a nice topic to write about ūüôā

In this post I’ll briefly summarize what you need to start your journey (or, at least, what I used to start mine).

Python

Even if we may be still far to deploy Software Defined Networks everywhere, software managed networks are a real thing and Python is the core of them.

Python is a pretty well-know programming language which is loved for its¬†ease of learning. I’ve studied C and Java at university and hated both of them, while I simply love Python ūüôā

There are plenty of available resources for those who wish to study for free and the following is a little list of stuff I’ve personally used:

  • CodeAcademy: really nice course to start your journey. It let you approach the language in a very practical way. Anyway, it does not dig very deep into the language.
  • Coursera: the website is full of Python courses, from the basics to more advanced topics. I’ve attended a couple of them and I really appreciated them.
  • How to think like a computer scientist: this was the very first Python resource I’ve ever used. It is a very well written book covering all the foundation in a pretty deep and clear way.
  • Dive Into Python: this is a more advanced book for those of you who are¬†hungry of knowledge.

I’m sure the list of someone else would look completely different since there are so many resources out there. So just pick one of them and start ūüôā

APIs

Networking vendors have developed specialized APIs to help engineers interact with their devices. I’ll introduce some of them within this section.

Juniper PyEZ

Juniper¬†is working hard on automation and has developed the PyEZ library, supported by almost every JunOS device. Once you installed all the requirements, it’s really easy to start talking to your remote device:


>>> from jnpr.junos import Device
>>> from jnpr.junos.utils.config import Config
>>> from pprint import pprint
>>> my_device = Device(host='172.16.1.1', user='gabriele', password='gabriele')
>>> my_device.open()
Device(172.16.1.1)
>>> pprint(my_device.facts)
{'2RE': False,
'HOME': '/var/home/gabriele',
'domain': None,
'fqdn': 'Router1',
'hostname': 'Router1',
'ifd_style': 'CLASSIC',
'model': 'olive',
'personality': 'UNKNOWN',
'serialnumber': '',
'switch_style': 'NONE',
'vc_capable': False,
'version': '12.1R1.9',
'version_info': junos.version_info(major=(12, 1), type=R, minor=1, build=9)}

Here there are some other practical reference about it:

Cisco

Cisco is working toward enabling automation in today’s network as well (of course).

Cisco NX-API

If you want to talk to Cisco NX-OS devices, you can use their NX-API.

Jason Edelman did an awesome work on both introducing NX-API here and developing another API called pycsco that simplifies working with Cisco NX-OS switches that support NX-API.

Here you can also find the latest reference from Cisco itself: NX-API book.

Cisco IOS-XR

Elisa Jasinska¬†developed an API¬†to help interact with Cisco devices running IOS-XR. It’s called¬†pyIOSXR.

Arista EOS

If you want to use Arista EOS, you can pick eAPI. You can also find some references here and on Packet Pushers.

Netmiko

Another super useful tool is Netmiko. It’s not a¬†specialized¬†API but instead it’s used to send commands to network devices and retrieve their output. That’s a great resource for those who want to start with network automation and I’ve extensively used it in pretty much every¬†project I’ve done.

In addition, the list of supported devices is huge:

Cisco IOS
Cisco IOS-XE
Cisco ASA
Cisco NX-OS
Cisco IOS-XR
Cisco WLC (limited testing)
Arista vEOS
HP ProCurve
HP Comware (limited testing)
Juniper Junos
Brocade VDX (limited testing)
F5 LTM (experimental)
Huawei (limited testing)
A10 (limited testing)
Avaya ERS (limited testing)
Avaya VSP (limited testing)
OVS (experimental)
Enterasys (experimental)
Extreme (experiemental)
Fortinet (experimental)
Alcatel-Lucent SR-OS (experimental)

Netmiko’s been developed by¬†Kirk Byers¬†and he also wrote an amazing post on how to use it. Thank you Kirk ūüôā

NAPALM

This name shouldn’t sound new to you! ūüėČ

In fact, I’ve extensively talked about NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) in my previous¬†post.

If you didn’t read it, repent, go read it and come back here ūüôā

Automation tools: Ansible

Like NAPALM, this shouldn’t sound new! I’ve talked about Ansible in two of my previous posts (here and here).

Anyway, those posts could be difficult to understand if you’re completely new. In this case, don’t worry, you definitely can¬†be guided by Kirk and Jason (these two guys are awesome!):

  • Kirk wrote a very nice guide introducing Ansible playbooks and templates, splitting it into 3 parts (Part1, Part2, Part3). This is what I used to write my first blog post on Ansible (see just above).
  • Jason extensively wrote about Ansible basics¬†(this post is precious for those who just started to use the tool) and other more advanced applications as well (here, here and here).

Have I already said that these two are awesome? ūüôā

Fast-paced Courses

Last but not least, if you really want to boost your automation skills and have enough resources (or you’re lucky enough to receive support from your company) you¬†may want to attend live classes on Network Automation.

Jason Edelman si¬†delivering an awesome¬†Network Programming and Automation¬†course all around the world. It covers everything you need to move from novice, writing your first “Hello World!” in Python, to NetOps Ninja developing a working network automation Flask app.

During the course you’ll not just sit there listening to Jason, but you’ll go through 10+ hours of labs too. I’ve reviewed the whole lab section and it took me almost the full 10 hours to complete it (and I was not new to most of the topics!). So I think you can expect to spend at least 2 extra hours on this.

Summarizing: 4 days digging deep on Python and Network Automation including some cool tools like Ansible + 12 hours of practical labs + guidance from Jason Edelman, one of the most expert guy on the field =¬†How cool is this? ¬†ūüėÄ

tweet

Here it is the course schedule for the first part of the next year. Don’t¬†miss it! ūüėČ

 Conclusion

These are just some of the available¬†resource I used to start my journey with NetOps. There are tons of more resources out there¬†if you want to start practicing Network Automation or simply improve your coding skills so you have no excuses! Choose what¬†you want and just start! ūüôā

start

 

Adding Cisco IOS support to NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support)

If ¬†you’re a networking passionate¬†I’m pretty sure you’ve already heard about NAPALM (no, I’m not talking about the¬†flammable liquid used in warfare ūüôā ). Anyway, if you’ve not yet, you’re going to discover a very nice project for network automation.


 

NAPALM

What is it? Let’s quote its documentation page:

NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) is a Python library that implements a set of functions to interact with different network device Operating Systems using a unified API.

It’s a project developed by David Barroso¬†and¬†Elisa Jasinska¬†(thank you guys ūüėÄ ), owned by Spotify and, as the quote says itself, it is used to interact with different hardware networking vendors. Basically, it works like an API on top of other APIs, adding another level of abstraction.

Lately, many vendors have developed APIs to making it easier to interact with their equipments. For example, most of the JunOS devices support¬†Juniper PyEZ, and so do Cisco’s Nexus with its¬†NX-API.

This way, if I want to interact with a Juniper device I can¬†use PyEZ, whereas I’d use NX-API if I wish to talk with a Nexus switch, and the example continues¬†with other specialized¬†APIs.

What NAPALM does is hiding this layer unifying the way we access a networking device, regardless who built it.

napalm

Back to few days ago, NAPALM supported the following network OS:

  • Arista¬†EOS
  • Juniper JunOS
  • Cisco IOS-XR
  • Cisco NX-OS
  • Fortinet FortiOS
  • IBM OS

This is possible thanks to the introduction of the NetworkDriver concept. Every time we want to interact with a device, we can only specify what OS we are going to talk to and NAPALM will select the correct NetworkDriver (basically, a library with all the functions related to that OS).

>>> from napalm import get_network_driver
>>> get_network_driver('eos')
>>>
>>> get_network_driver('iosxr')
>>>
>>> get_network_driver('junos')
>>>
>>> get_network_driver('fortios')

NAPALM will still use third party APIs but this will be trasparent to the user.

Cisco IOS support

Unlike NX-OS, Cisco IOS have no API support. Therefore, it’s not that straightforward to obtain structured data from it and at first NAPALM didn’t support it.¬†So¬†I thought this could be a nice spot to play ūüôā

I forked the main repository and started to code.

IOSDriver

Since no native API exists, I had to use something more general: netmiko. This is a pretty sweet Python library making it super easy to connect and interact with networking devices. Once a command is sent, netmiko can give me back the output and then I can start to filter and parse it.

The module is composed by 12 methods:

  • open():¬†opens the connection with the remote device. It is the first method to be used.
  • close():¬†closes the connection with the remote device.
  • load_merge_candidate(filename, config):¬†loads a candidate configuration from a textfile or a configuration string. If both are passed, filename is picked. At this point no configuration is pushed to the device yet.
  • compare_config():¬†simply shows the list of commands proposed by the load_merge_candidate method and¬†ready to be executed if commited.
  • discard_config():¬†what if we notice some errors after the¬†compare_config? We can discard¬†the proposed changes using this method.
  • commit_commit():¬†pushes the configuration from¬†load_merge_candidate¬†and saves the configuration.
  • rollback():¬†we can rollback the commited¬†changes using this method. This simply adds the¬†no¬†keyword to commands (anyway it’s smart enough to recognize parent/child commands)
  • get_lldp_neighbors():¬†extracts lldp neighbors information from the device.
  • get_facts():¬†extracts information like¬†uptime, vendor, os_version, serial_number, model, hostname, fqdn,¬†and¬†interface_list.
  • get_interfaces():¬†extracts information about interfaces including¬†status and¬†speed.
  • get_bgp_neighbors(): extracts information about BGP neighbors.
  • get_interfaces_counters():¬†extracts information about counters.

Demo

Now let’s see an example of how to use NAPALM.

The first thing we’re gonna do is to connect to our remote device specifying the OS type, username, password and IP address.


from netmiko import ConnectHandler
from napalm import get_network_driver
get_network_driver('ios')
driver = get_network_driver('ios')
device = driver('172.16.1.1', 'gabriele', 'gabriele')
device.open()

Once we’re done, if everything went fine we’ll see the Python interactive shell confirming the SSH session has been established.

python1

Now we can start to interact with our device. Let’s ask for some facts, for example:

python2

As we can see, it’s a Cisco¬†3640¬†device whose IOS version is 12.4(16). It has 3 interfaces and its¬†uptime value is set to¬†9 minutes.

Cool, right? ūüėÄ

Let’s try some other methods:

python3

Earlier we discovered 3 interfaces exist in our device. Now we’ve just obtained some specific information about them using the get_interfaces()¬†as well as BGP neighbors information thanks to¬†get_bgp_neighbors().

Let’s see how NAPALM can help us with configuration management. Imagine we want to implement OSPF on our network. Just to keep it simple, we want to push the following configuration:


router ospf 1
!
network 0.0.0.0 0.0.0.0 area 1
!

Using¬†load_merge_candidate(filename=new_good.conf), we’ll load our configuration (assuming new_good.conf¬†is the textfile containing the above config). Then, we can see what changes would be implemented using¬†compare_config(). At that point, we can decide to either commit or discard these changes.

Here we’ve sudden realized our OSPF area should be¬†0¬†and not¬†1. So we decide to discard the candidate configuration with¬†discard_config(). We can use¬†compare_config()¬†to confirm that every possible change’s been discarded.

python4

Since our last¬†compare_config()¬†doesn’t show anything, it means everything went fine.

Anyway, we still want to implement OSPF, so we fix the configuration and give it another try. This time we want to use a configuration string instead of a .conf¬†file. We do this with¬†load_merge_candidate(config=’router ospf 1\nnetwork 0.0.0.0 0.0.0.0 area 0′). Then, if we are happy with it, we can commit.

python5

..and this is how the router’s config look like:

python6

At this point, if we want to rollback the change we can simply use rollback().

Conclusion

It’s been lot of fun to work on this patch and I’m happy to announce that now NAPALM supports IOS too since my PullRequest has been merged to the main repo ūüôā

NAPALM is a really cool project and it’s popular among NetOps community and it’s also been presented at a NANOG conference. Here you can find the video from the awesome guys who actually designed and implemented it. Enjoy ūüôā


 

P.S.

If you’re interested about NAPALM or Network Automation in general you should definitely join the SLACK channel at network.toCode(). Here you’ll find lots of cool guys discussing fancy stuff on networking ūüôā