A CI pipeline for NSO service packages, with netsim and dry-run diffs

Most NSO installations have no pipeline. Packages are edited on the server, reloaded, and tested in production by the next change that uses them. This post builds the smallest pipeline that stops that: every merge request compiles the package, loads it into a throwaway NSO with netsim devices, runs a test through RESTCONF, and posts the native dry-run diff for a reviewer to read.

The example uses GitLab CI and the Cisco NSO development container. The same five stages work in GitHub Actions or Jenkins with the YAML swapped.

The repository

.
├── .gitlab-ci.yml
├── packages/
│   ├── cisco-iosxr-cli-7.52/        # NED, vendored or fetched in CI
│   └── vlan/
│       ├── package-meta-data.xml
│       ├── python/vlan/main.py
│       ├── src/Makefile
│       ├── src/yang/vlan.yang
│       └── templates/vlan-template.xml
└── tests/
    ├── devices.xml
    ├── dry-run.cli
    └── test_vlan.py

The package is the VLAN service from the NSO stall patterns post: a name, a VLAN ID, and a list of devices. The mapping is a template applied once per device:

# packages/vlan/python/vlan/main.py
import ncs
from ncs.application import Service


class VlanService(Service):
    @Service.create
    def cb_create(self, tctx, root, service, proplist):
        for device in service.device:
            vars = ncs.template.Variables()
            vars.add('DEVICE', device)
            vars.add('VLAN_ID', service.vlan_id)
            ncs.template.Template(service).apply('vlan-template', vars)


class Main(ncs.application.Application):
    def setup(self):
        self.register_service('vlan-servicepoint', VlanService)

<!-- packages/vlan/templates/vlan-template.xml -->
<config-template xmlns="http://tail-f.com/ns/config/1.0">
  <devices xmlns="http://tail-f.com/ns/ncs">
    <device>
      <name>{$DEVICE}</name>
      <config>
        <interface xmlns="http://tail-f.com/ned/cisco-ios-xr">
          <GigabitEthernet>
            <id>0/0/1.{$VLAN_ID}</id>
            <description>vlan-{$VLAN_ID}</description>
            <encapsulation>
              <dot1q>
                <vlan-id>{$VLAN_ID}</vlan-id>
              </dot1q>
            </encapsulation>
          </GigabitEthernet>
        </interface>
      </config>
    </device>
  </devices>
</config-template>

The container

Cisco ships NSO as two container images, cisco-nso-prod and cisco-nso-dev, as tarballs on the software download site. The dev image carries the full SDK, including ncs-netsim and ncs-make-package. Load it once and push it to the registry the runners can reach:

$ docker load -i nso-6.3.container-image-dev.linux.x86_64.tar.gz
$ docker tag cisco-nso-dev:6.3 registry.internal/cisco-nso-dev:6.3
$ docker push registry.internal/cisco-nso-dev:6.3

The NED is the other dependency. Either vendor it into the repository under packages/ or fetch it from an artifact store in the build stage. Vendoring is simpler and the NED tarball is small enough.

The pipeline

# .gitlab-ci.yml
stages: [lint, build, netsim, dry-run, review]

default:
  image: registry.internal/cisco-nso-dev:6.3
  before_script:
    - source /opt/ncs/current/ncsrc

lint:
  stage: lint
  script:
    - yanger -p packages/vlan/src/yang packages/vlan/src/yang/vlan.yang
    - python -m pyflakes packages/vlan/python

build:
  stage: build
  script:
    - make -C packages/vlan/src clean all
  artifacts:
    paths: [packages/vlan/load-dir]
    expire_in: 1 day

netsim:
  stage: netsim
  script:
    - ncs-netsim create-network packages/cisco-iosxr-cli-7.52 3 pe
    - ncs-netsim start
    - ncs-setup --netsim-dir ./netsim --dest ./ncs-run
    - cp -r packages/vlan ncs-run/packages/
    - cd ncs-run && ncs --with-package-reload && cd ..
    - echo "devices sync-from" | ncs_cli -C -u admin
    - pip install -q requests pytest
    - pytest tests/ -q --junitxml=report.xml
  artifacts:
    reports: { junit: report.xml }

dry-run:
  stage: dry-run
  script:
    - ncs-netsim create-network packages/cisco-iosxr-cli-7.52 3 pe
    - ncs-netsim start
    - ncs-setup --netsim-dir ./netsim --dest ./ncs-run
    - cp -r packages/vlan ncs-run/packages/
    - cd ncs-run && ncs --with-package-reload && cd ..
    - echo "devices sync-from" | ncs_cli -C -u admin
    - ncs_cli -C -u admin < tests/dry-run.cli | tee dry-run.txt
  artifacts:
    paths: [dry-run.txt]

review:
  stage: review
  script:
    - |
      body=$(printf '### Dry run for %s\n```\n%s\n```' "$CI_COMMIT_SHORT_SHA" "$(cat dry-run.txt)")
      curl -s --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
        --data-urlencode "body=$body" \
        "$CI_API_V4_URL/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  rules:
    - if: $CI_MERGE_REQUEST_IID

ncs-netsim create-network makes three devices named pe0, pe1, and pe2. ncs-setup writes a run directory with ncs.conf and a CDB init file that registers them, so after ncs --with-package-reload and a sync-from, the throwaway NSO has three in-sync XR devices and the package loaded. The whole thing takes about ninety seconds on a modest runner.

The test

The test creates a service with dry-run=native and checks the diff before committing for real. It asserts two things that matter: the service touches only the device it was given, and the generated config contains the VLAN.

# tests/test_vlan.py
import requests

NSO = "http://localhost:8080"
AUTH = ("admin", "admin")
HEADERS = {
    "Content-Type": "application/yang-data+json",
    "Accept": "application/yang-data+json",
}


def create(body, dry_run=False):
    url = f"{NSO}/restconf/data" + ("?dry-run=native" if dry_run else "")
    return requests.post(url, json=body, headers=HEADERS, auth=AUTH)


def test_dry_run_touches_only_the_target_device():
    body = {"vlan:vlan": [{"name": "t-1042", "vlan-id": 1042, "device": ["pe0"]}]}
    r = create(body, dry_run=True)
    assert r.ok, r.text
    devices = r.json()["dry-run-result"]["native"]["device"]
    assert [d["name"] for d in devices] == ["pe0"]
    assert "encapsulation dot1q 1042" in devices[0]["data"]


def test_commit_leaves_device_in_sync():
    body = {"vlan:vlan": [{"name": "t-1042", "vlan-id": 1042, "device": ["pe0"]}]}
    assert create(body).status_code == 201
    r = requests.post(
        f"{NSO}/restconf/data/tailf-ncs:devices/device=pe0/check-sync",
        headers=HEADERS, auth=AUTH,
    )
    assert r.json()["tailf-ncs:output"]["result"] == "in-sync"


def test_vlan_id_out_of_range_is_rejected():
    body = {"vlan:vlan": [{"name": "bad", "vlan-id": 4095, "device": ["pe0"]}]}
    assert create(body).status_code == 400

The third test is the one teams skip and the one that catches the most. The YANG range "1..4094" is the only thing standing between a typo and a broken trunk, and a test proves the range is really there.

The dry run on the merge request

tests/dry-run.cli is a plain CLI script that creates a representative instance and asks for the native diff:

config
vlan review-1042 vlan-id 1042 device [ pe0 pe1 ]
commit dry-run outformat native
abort

The review stage posts the output as a comment on the merge request:

native {
    device {
        name pe0
        data interface GigabitEthernet0/0/1.1042
              description vlan-1042
              encapsulation dot1q 1042
             !
    }
    device {
        name pe1
        data interface GigabitEthernet0/0/1.1042
              description vlan-1042
              encapsulation dot1q 1042
             !
    }
}

That comment is the whole point. A reviewer who has never opened the Python can see what the change will do to a router. When somebody edits the template and the diff grows an unexpected line, it shows up here, not in production.

What this does not cover

Netsim devices accept any config the NED's YANG allows. They do not run a routing stack, so a service that produces valid but wrong BGP config passes this pipeline. For that, the next stage is a lab with real or virtual devices and pre and post checks that look at operational state, not just config. The pipeline above is the floor, not the ceiling, and it is the stage most stalled projects never reach.