Pre and post change checks with pyATS and Forward Networks

A dry-run diff tells a reviewer what config will change. It says nothing about whether the network will still work afterwards. That is the job of pre and post checks: capture operational state before the change, capture it again after, and fail loudly if anything that matters moved.

This post builds the two layers NetStacks uses in a change pipeline. pyATS for device state, because it already knows how to parse every platform's show output. Forward Networks for intent and paths, because it can answer "can customer A still reach customer B" across the whole network without logging into anything. Both run from the same pipeline stage and both return an exit code the pipeline can act on.

Layer one: device state with pyATS

pyATS and its Genie library ship with learn models for most protocols. A learn model logs in, runs the relevant show commands, parses them, and writes a normalized structure to disk. Two runs and a diff give a change report with no custom parsing.

The testbed names the devices in scope for the change:

# testbed.yml
devices:
  pe2-den:
    os: iosxr
    type: router
    connections:
      cli:
        protocol: ssh
        ip: 10.0.1.12
    credentials:
      default:
        username: "%ENV{NET_USER}"
        password: "%ENV{NET_PASS}"

Before the change window:

$ pyats learn bgp isis interface --testbed-file testbed.yml --devices pe2-den --output pre

The output directory holds one file per feature per device, for example bgp_iosxr_pe2-den_ops.txt. After the change, the same command with --output post, then:

$ pyats diff pre post --output diffs

When nothing operational changed, the diff directory is empty and the command exits zero. When something did:

--- pre/bgp_iosxr_pe2-den_ops.txt
+++ post/bgp_iosxr_pe2-den_ops.txt
 info:
  instance:
   default:
    vrf:
     default:
      neighbor:
       10.0.0.12:
-       session_state: established
+       session_state: idle

A BGP neighbor went from established to idle during a VLAN change. That is either a coincidence worth investigating or the change did something the diff didn't show. Either way the pipeline should stop here, and it does, because pyats diff exits non-zero when it finds differences.

Deciding what counts

Not every difference is a failure. Interface counters change. Uptime changes. The learn models already exclude the noisiest fields, and --exclude handles the rest:

$ pyats diff pre post --exclude "[Uu]ptime" "last_reset" "in_pkts" "out_pkts"

The useful discipline is to agree the exclusion list once per change type and keep it in the repository next to the pipeline. A list that grows every time a check fails is a list that is being used to hide problems.

Layer two: intent and paths with Forward Networks

pyATS says what each device thinks. Forward says what the network as a whole will do with a packet, computed from a snapshot of every device's config and state. The checks worth gating on are the ones the team already writes down in change tickets and then verifies by hand: this customer stays isolated from that one, this prefix stays reachable from the core, no device loses its NTP source.

A pipeline stage takes a snapshot, waits for it, and asks three questions.

#!/usr/bin/env python3
"""Post-change verification against Forward Networks. Exit 1 on any failure."""
import os
import sys
import time
import requests

BASE = "https://fwd.app/api"
AUTH = (os.environ["FWD_KEY"], os.environ["FWD_SECRET"])
NETWORK = os.environ["FWD_NETWORK_ID"]


def collect():
    r = requests.post(f"{BASE}/networks/{NETWORK}/startcollection", auth=AUTH)
    r.raise_for_status()
    while True:
        latest = requests.get(f"{BASE}/networks/{NETWORK}/snapshots/latestProcessed", auth=AUTH).json()
        if latest.get("state") == "PROCESSED":
            return latest["id"]
        time.sleep(15)


def failing_checks(snapshot):
    r = requests.get(f"{BASE}/snapshots/{snapshot}/checks", params={"status": "FAIL"}, auth=AUTH)
    r.raise_for_status()
    return [c["name"] for c in r.json()]


def path_exists(snapshot, src, dst):
    r = requests.get(f"{BASE}/snapshots/{snapshot}/paths", params={"srcIp": src, "dstIp": dst}, auth=AUTH)
    r.raise_for_status()
    return bool(r.json()["info"]["paths"])


def main():
    snap = collect()
    failures = []

    for name in failing_checks(snap):
        failures.append(f"intent check failed: {name}")

    # Customer isolation must hold after the change.
    if path_exists(snap, "10.42.0.10", "10.88.0.10"):
        failures.append("path exists from cust-1042 to cust-0988")

    # Core reachability must hold too.
    if not path_exists(snap, "10.0.0.1", "10.42.0.10"):
        failures.append("no path from core1-den to cust-1042")

    for f in failures:
        print(f"FAIL  {f}")
    if failures:
        sys.exit(1)
    print(f"PASS  snapshot {snap}: 0 failing checks, isolation holds, core reachable")


if __name__ == "__main__":
    main()

The intent checks come from Forward itself, defined once in the UI or as NQE queries and run against every snapshot. Pulling the list of failures is the cheap way to inherit all of them. The two path queries are the change-specific part: for a VLAN change on an access port, the question is whether the customer on that port can suddenly reach a neighbor they shouldn't.

For compliance rules the team has but Forward doesn't ship, NQE is a query language over the snapshot. A rule that every XR device points at the approved NTP servers:

foreach d in network.devices
where d.platform.vendor == "CISCO" && d.platform.os == "IOS_XR"
let servers = (foreach s in d.ntp.servers select s)
where !(servers == ["10.0.0.5", "10.0.0.6"])
select { device: d.name, ntp: servers }

Any rows returned are devices out of policy, and the pipeline treats a non-empty result the same as a failed check.

Wiring both into the pipeline

The change pipeline from the approach page has a pre-check stage before the dry run and a post-check stage after apply. Both call the same two scripts:

pre-check:
  stage: pre-check
  script:
    - pyats learn bgp isis interface --testbed-file testbed.yml --devices $DEVICES --output pre
    - python3 verify_forward.py
  artifacts:
    paths: [pre/]

post-check:
  stage: post-check
  needs: [apply]
  script:
    - pyats learn bgp isis interface --testbed-file testbed.yml --devices $DEVICES --output post
    - pyats diff pre post --exclude "[Uu]ptime" "last_reset" --output diffs
    - python3 verify_forward.py
  artifacts:
    paths: [post/, diffs/]
    when: always

when: always on the post-check artifacts matters. The diff is most valuable when the stage fails, and GitLab's default is to drop artifacts on failure.

A failed pre-check blocks the change before anything is pushed, which usually means the network was already in a state nobody knew about. A failed post-check stops the pipeline after apply with the evidence attached, and the runbook for that change says what to do next: roll back automatically if the rollback criteria say so, or page the owner with the diff in hand.

What this gives a team

Every change now produces four artifacts: pre state, post state, the diff between them, and a Forward verdict with a snapshot ID. They live on the pipeline run, linked from the ticket. Six months later, when someone asks what changed on pe2-den in October, the answer is a link, not an archaeology project.

That is the difference between automation that pushes config and automation that operators trust. The config push is the easy half.