NetStacks Blog

Why network automation projects stall, and the 7 fixes

Mon, 12 Oct 2026 00:00:00 GMT

Network automation projects rarely fail. They stall, quietly, somewhere around month six. Nobody cancels anything. The platform stays up, the licenses get renewed, and there is still a line for it in the roadmap. But new workflows stop shipping, the people who were excited about it drift to other work, and changes go back to being typed into devices by hand. We have been asked to look at projects like this on Cisco NSO, Cisco Crosswork, NetBrain, Forward Networks, and plain Python. The platform changes. The patterns don't. Each one below comes with the evidence that reveals it, because a stall is a measurable condition, not a feeling. ### 1. The first workflow was too big The first thing the team tried to automate was the hardest thing they do: a full customer turn-up, every platform, every variant, every option anyone has ever ordered. A year later it still isn't done, and there is nothing in production to show for it. The evidence is the ratio of model to instances. On NSO, count the leaves in the service YANG and count the service instances in production: ```text admin@ncs# show running-config services l3vpn | count Count: 0 lines admin@ncs# show packages package l3vpn package-version packages package l3vpn package-version 0.0.14 ``` Fourteen versions of a service model with zero instances is a project that has been designing instead of shipping. The fix is to ship one narrow workflow within 30 days. Pick something frequent, low-risk, and boring: an interface description, a VLAN on an access port, a pre-change check that runs before every maintenance window. A model with four leaves and two hundred instances teaches a team more than one with forty leaves and none. ### 2. A year on the source of truth, and nothing on the network The project started with the inventory, because every vendor and every consultancy said it should. Eighteen months in, NetBox or Nautobot is beautifully modeled, half populated, and no change has ever flowed from it to a router. The evidence is in the webhook and job logs. A source of truth that drives automation emits events that something consumes: ```bash $ curl -s -H "Authorization: Token $NB_TOKEN" \ "https://netbox.example.net/api/extras/webhooks/" | jq '.count' 0 ``` Zero webhooks after a year of modeling means the data has no consumer. It is a database, not a source of truth. A source of truth is necessary. It is not a deliverable. The fix is to build it as the workflows demand it, not before. Pick the first workflow, model only the data that workflow needs, wire the webhook or the pipeline that reads it, and let the inventory grow one workflow at a time. ### 3. The network didn't match the data Half the devices show drift between what the platform thinks is there and what is actually configured. Nobody wants to reconcile, because nobody is sure what the sync will pull in or overwrite. So the gap widens every week, and every workflow has to work around it. The evidence is a single command, repeated weekly and plotted: ```text admin@ncs# devices check-sync | include out-of-sync | count Count: 61 lines admin@ncs# show devices list | count Count: 118 lines ``` More than half the estate out of sync is not a data problem. It is a process problem: out-of-band changes are normal and nothing notices them. The fix is to make drift visible and boring. Run the comparison on a schedule, publish the report, and work through the devices one at a time with `compare-config` before any `sync-from`. Then agree on a policy for out-of-band changes, so the network doesn't drift again the week after the reconciliation finishes. There is a [full procedure for NSO](/blog/reconciling-out-of-sync-devices-in-nso) in a separate post. ### 4. No path from a laptop to production The workflow works on the developer's machine. Getting it into production means copying files onto a server, editing them in place, and hoping. There is no history, no review, and no way to know whether a change breaks something until it does. The evidence is on the server itself: ```bash $ ls -la /var/opt/ncs/packages/ drwxr-xr-x vlan drwxr-xr-x vlan.bak drwxr-xr-x vlan.bak2 drwxr-xr-x vlan-working $ cd /var/opt/ncs/packages/vlan && git log --oneline | head -1 fatal: not a git repository ``` Four copies of a package and no repository is a project that cannot roll back. The fix is a pipeline, even a small one. Git as the source of truth for the code, netsim or a lab to run against, and a dry-run diff that somebody reads before anything is committed: ```text admin@ncs(config)# vlan access-1042 vlan-id 1042 device pe2-den admin@ncs(config-vlan-access-1042)# commit dry-run outformat native native { device { name pe2-den data interface GigabitEthernet0/0/1.1042 description vlan-1042 encapsulation dot1q 1042 ! } } ``` If the diff surprises the reviewer, it would have surprised everyone in production too. The [pipeline post](/blog/ci-pipeline-for-nso-packages-with-netsim) walks through building this on GitLab. ### 5. The process was never written down The automation encodes a process. Nobody wrote that process down, so the developer guessed. The workflow asks for inputs operators don't have, skips the approval the change board requires, and doesn't know about the one thing the senior engineer always checks. Operators work around it, and then they stop using it. The evidence is in the ticketing system. Pull the last ninety days of network changes and count how many reference the platform at all: ```bash $ curl -s -u "$SN_USER:$SN_PASS" \ "https://example.service-now.com/api/now/table/change_request?sysparm_query=category=network^opened_at>javascript:gs.daysAgo(90)&sysparm_fields=number,short_description" \ | jq '[.result[] | select(.short_description | test("nso|crosswork|automation"; "i"))] | length' 3 ``` Three of several hundred changes touched the platform. The other several hundred went around it, and each one is a process the automation didn't model. The fix is to map the process before building anything. Sit with the people who request, approve, and make the change, and write down what actually happens, including the spreadsheet and the chat message. Twenty minutes walking an operator through the inputs saves weeks of rework later. ### 6. The one person who built it left One engineer built most of it, and that engineer moved on. What is left works, mostly, but nobody understands it well enough to change it. Every new request is a risk, so the answer is always no. The evidence is the commit history, when there is one: ```bash $ git shortlog -sn --since="2 years ago" packages/ 214 jsmith 3 kpatel ``` The fix is to document what exists, refactor the critical paths first, and pair with the team's own staff while doing it. The work isn't finished until there is a handoff runbook the team has actually used without the author in the room. ### 7. Nothing consumes it The platform works. Requests still arrive as tickets, an engineer still reads each one and types the change, and the platform is one more tool they could use if they had time. The evidence is the audit log. A platform that is consumed shows API calls from something other than a person at a CLI: ```bash $ grep -c 'restconf' /var/log/ncs/audit.log 0 $ grep -c 'ncs_cli' /var/log/ncs/audit.log 1872 ``` The fix is to connect the platform to the request path: a form, an API call from the ticketing system, a button on the ticket itself. On NSO that is one RESTCONF call per change, which the ticketing system can make directly: ```bash $ curl -s -u "$NSO_USER:$NSO_PASS" -X POST \ -H "Content-Type: application/yang-data+json" \ -d '{"vlan:vlan":[{"name":"CHG0048211","vlan-id":1042,"device":["pe2-den"]}]}' \ "https://nso.example.net/restconf/data" ``` Make it the normal way changes happen, and track how many changes go through it so adoption can be measured instead of assumed. ### A quick self-check If three of these sound familiar, the project is stalled, not slow. None of them are technical dead ends. Each one has a fix that fits into a few weeks of focused work, and none of the fixes require changing platforms. If a project got stuck in a way that isn't on this list, or got moving again, tell us about it in the comments.

A CI pipeline for NSO service packages, with netsim and dry-run diffs

{$DEVICE} — Mon, 12 Oct 2026 00:00:00 GMT

{$DEVICE} 0/0/1.{$VLAN_ID} vlan-{$VLAN_ID}

Reconciling out-of-sync devices in NSO without breaking anything

Mon, 12 Oct 2026 00:00:00 GMT

Every NSO estate older than a year has an out-of-sync backlog. Somebody fixed something at 2 a.m. from the console, a vendor TAC session changed a timer, a script nobody remembers pushed an ACL. NSO notices and marks the device out of sync. Then nothing happens, because the two commands that fix it both sound dangerous, and nobody is sure which one to run. This is the procedure we use on a backlog of any size. It is slow on purpose. Speed is what caused the backlog. ### What out-of-sync means NSO keeps a copy of each device's configuration in CDB and a transaction ID, or a hash, that the device reported when it was last synchronized. `check-sync` asks the device for its current ID and compares: ```text admin@ncs# devices device pe2-den check-sync result out-of-sync info got: 7c1e4f9a expected: 2b90d1c3 ``` It says the device changed. It does not say how. Anyone who runs `sync-from` or `sync-to` at this point is guessing. By default NSO refuses to commit a change to an out-of-sync device. That is the right default, and the reason a backlog freezes automation: every service touching that device fails until someone reconciles it. ```text admin@ncs# show running-config devices global-settings out-of-sync-commit-behaviour devices global-settings out-of-sync-commit-behaviour reject ``` Leave it on `reject`. Teams that flip it to `accept` to get unstuck are choosing not to know what the network looks like. ### Step 1. Measure the backlog Before touching anything, get the list and keep it: ```text admin@ncs# devices check-sync | include out-of-sync | count Count: 61 lines admin@ncs# show devices list | count Count: 118 lines ``` Then capture every diff. `compare-config` shows what the device has that NSO doesn't, and what NSO has that the device doesn't, without changing either side: ```bash $ for d in $(echo "show devices list | nomore" | ncs_cli -C -u admin | awk 'NR>1 {print $1}'); do echo "devices device $d compare-config" | ncs_cli -C -u admin > "diffs/$d.txt" done $ wc -l diffs/*.txt | sort -n | tail -5 4 diffs/pe1-den.txt 12 diffs/pe3-slc.txt 38 diffs/pe2-den.txt 412 diffs/agg7-slc.txt ``` Sort by size. The small diffs are easy. The 412-line one is where the actual problem lives, and it is probably a device somebody rebuilt from a template without telling NSO. ### Step 2. Read the diffs and sort them into three piles In `compare-config` output, a minus sign is config present on the device and absent in NSO. A plus sign is the reverse. ```text admin@ncs# devices device pe2-den compare-config diff devices { device pe2-den { config { interface GigabitEthernet0/0/3 { - ip access-group cust-1042-in in } ntp { - server 10.0.0.9 + server 10.0.0.5 } } } } ``` Two different things happened here. Somebody removed an access-group by hand, and somebody changed the NTP server. Each line goes in one of three piles: 1. **Keep it.** The out-of-band change is correct and NSO should learn it. The NTP change is probably this: the old server was decommissioned and the device was fixed by hand. 2. **Reassert it.** The out-of-band change is wrong, or it undid something a service owns. The missing access-group is this, if a service put it there. 3. **Investigate.** Nobody knows. The 412-line diff is all of this pile. Write the pile down per device, in the ticket. This is the only part of the procedure that needs judgment, and it is the part teams try to skip. ### Step 3. For "keep it": sync-from, after a dry run `sync-from` pulls the device's config into CDB and overwrites NSO's copy. It never touches the device. It is safe for the network and dangerous for NSO's understanding of it, because anything NSO had that the device doesn't will disappear from CDB. Preview first: ```text admin@ncs# devices device pe2-den sync-from dry-run ``` The output is the same diff as `compare-config`, shown as what CDB will become. If it matches the "keep it" pile and nothing else, run it: ```text admin@ncs# devices device pe2-den sync-from result true ``` For a large device where only one subtree changed, `partial-sync-from` pulls just that path and leaves the rest of CDB alone: ```text admin@ncs# devices partial-sync-from path [ /devices/device[name='pe2-den']/config/cisco-ios-xr:ntp ] ``` ### Step 4. For "reassert it": find the service, then re-deploy If the removed access-group was placed by a service, NSO knows which one: ```text admin@ncs# show running-config devices device pe2-den config interface GigabitEthernet 0/0/3 | display service-meta-data ``` The output annotates each line with the service that owns it and a reference count. With the owner known, a re-deploy puts the config back. Dry run first, always: ```text admin@ncs# services l3vpn cust-1042 re-deploy dry-run cli { local-node { data devices { device pe2-den { config { interface GigabitEthernet0/0/3 { + ip access-group cust-1042-in in } } } } } } ``` Note that `re-deploy` on an out-of-sync device will itself be rejected. The order matters: `sync-from` to accept the device's current state, then `re-deploy` to push the service's intent back on top of it. After both, `check-sync` reports in-sync and the service is whole. ### Step 5. For config that is already right: no-networking Sometimes the device already has exactly what a service would produce, but NSO doesn't know the service exists. This is the brownfield case: the network predates the automation. Creating the service normally would try to push config that is already there, and the NED may or may not treat that as a no-op. `commit no-networking` writes the service to CDB and marks the device config as service-owned without sending anything to the device: ```text admin@ncs(config)# vlan access-1042 vlan-id 1042 device pe2-den admin@ncs(config-vlan-access-1042)# commit no-networking ``` Then verify the device agrees: ```text admin@ncs# devices device pe2-den check-sync result in-sync ``` If it reports out-of-sync after a `no-networking` commit, the device did not actually have what the service produces, and the right move is to `commit dry-run outformat native` and look at the gap. For a service that should take ownership of config that is already on the device and also already in CDB, `re-deploy reconcile` sets the reference counts so the service owns it and a future delete of the service removes it: ```text admin@ncs# services vlan access-1042 re-deploy reconcile { keep-non-service-config } ``` `keep-non-service-config` is the safe option. It means config under the service's path that the service did not produce stays on the device when the service is deleted. ### Step 6. Never blanket sync-to `sync-to` pushes NSO's copy to the device and overwrites whatever is there. On an out-of-sync device, that means deleting every out-of-band change at once, including the one the NOC made at 2 a.m. to keep a customer up. There is a place for `sync-to`: a device that was factory reset and needs its config restored. There is no place for it in a reconciliation loop. ### Step 7. Make it boring The backlog came from out-of-band changes that nothing noticed. The fix after reconciliation is to notice them within a day, not a year. A nightly job that records the count and the per-device diffs is enough: ```bash #!/usr/bin/env bash # /etc/cron.daily/nso-drift set -euo pipefail out=/var/lib/nso-drift/$(date +%F) mkdir -p "$out" echo "devices check-sync" | ncs_cli -C -u admin > "$out/check-sync.txt" grep -B1 out-of-sync "$out/check-sync.txt" | awk '/device/ {print $2}' | while read -r d; do echo "devices device $d compare-config" | ncs_cli -C -u admin > "$out/$d.diff" done count=$(grep -c out-of-sync "$out/check-sync.txt" || true) echo "$(date +%F) $count" >> /var/lib/nso-drift/history.txt ``` Publish the number where the team sees it. A count that goes from 61 to 0 and stays there is the proof that reconciliation worked. A count that creeps back to 10 in a month is a process problem with a name on it, and the diffs say whose. The last piece is policy, not tooling: out-of-band changes are allowed in an emergency, the ticket must say so, and the device gets reconciled the next business day. Teams that write that down and hold to it stop having a backlog. Teams that don't will be reading this post again next year.

Pre and post change checks with pyATS and Forward Networks

Mon, 12 Oct 2026 00:00:00 GMT

A dry-run diff tells a reviewer what config will change. It says nothing about whether the network will still work afterwards. That is the job of pre and post checks: capture operational state before the change, capture it again after, and fail loudly if anything that matters moved. This post builds the two layers NetStacks uses in a change pipeline. pyATS for device state, because it already knows how to parse every platform's `show` output. Forward Networks for intent and paths, because it can answer "can customer A still reach customer B" across the whole network without logging into anything. Both run from the same pipeline stage and both return an exit code the pipeline can act on. ### Layer one: device state with pyATS pyATS and its Genie library ship with `learn` models for most protocols. A learn model logs in, runs the relevant `show` commands, parses them, and writes a normalized structure to disk. Two runs and a diff give a change report with no custom parsing. The testbed names the devices in scope for the change: ```yaml # testbed.yml devices: pe2-den: os: iosxr type: router connections: cli: protocol: ssh ip: 10.0.1.12 credentials: default: username: "%ENV{NET_USER}" password: "%ENV{NET_PASS}" ``` Before the change window: ```bash $ pyats learn bgp isis interface --testbed-file testbed.yml --devices pe2-den --output pre ``` The output directory holds one file per feature per device, for example `bgp_iosxr_pe2-den_ops.txt`. After the change, the same command with `--output post`, then: ```bash $ pyats diff pre post --output diffs ``` When nothing operational changed, the diff directory is empty and the command exits zero. When something did: ```diff --- pre/bgp_iosxr_pe2-den_ops.txt +++ post/bgp_iosxr_pe2-den_ops.txt info: instance: default: vrf: default: neighbor: 10.0.0.12: - session_state: established + session_state: idle ``` A BGP neighbor went from established to idle during a VLAN change. That is either a coincidence worth investigating or the change did something the diff didn't show. Either way the pipeline should stop here, and it does, because `pyats diff` exits non-zero when it finds differences. ### Deciding what counts Not every difference is a failure. Interface counters change. Uptime changes. The learn models already exclude the noisiest fields, and `--exclude` handles the rest: ```bash $ pyats diff pre post --exclude "[Uu]ptime" "last_reset" "in_pkts" "out_pkts" ``` The useful discipline is to agree the exclusion list once per change type and keep it in the repository next to the pipeline. A list that grows every time a check fails is a list that is being used to hide problems. ### Layer two: intent and paths with Forward Networks pyATS says what each device thinks. Forward says what the network as a whole will do with a packet, computed from a snapshot of every device's config and state. The checks worth gating on are the ones the team already writes down in change tickets and then verifies by hand: this customer stays isolated from that one, this prefix stays reachable from the core, no device loses its NTP source. A pipeline stage takes a snapshot, waits for it, and asks three questions. ```python #!/usr/bin/env python3 """Post-change verification against Forward Networks. Exit 1 on any failure.""" import os import sys import time import requests BASE = "https://fwd.app/api" AUTH = (os.environ["FWD_KEY"], os.environ["FWD_SECRET"]) NETWORK = os.environ["FWD_NETWORK_ID"] def collect(): r = requests.post(f"{BASE}/networks/{NETWORK}/startcollection", auth=AUTH) r.raise_for_status() while True: latest = requests.get(f"{BASE}/networks/{NETWORK}/snapshots/latestProcessed", auth=AUTH).json() if latest.get("state") == "PROCESSED": return latest["id"] time.sleep(15) def failing_checks(snapshot): r = requests.get(f"{BASE}/snapshots/{snapshot}/checks", params={"status": "FAIL"}, auth=AUTH) r.raise_for_status() return [c["name"] for c in r.json()] def path_exists(snapshot, src, dst): r = requests.get(f"{BASE}/snapshots/{snapshot}/paths", params={"srcIp": src, "dstIp": dst}, auth=AUTH) r.raise_for_status() return bool(r.json()["info"]["paths"]) def main(): snap = collect() failures = [] for name in failing_checks(snap): failures.append(f"intent check failed: {name}") # Customer isolation must hold after the change. if path_exists(snap, "10.42.0.10", "10.88.0.10"): failures.append("path exists from cust-1042 to cust-0988") # Core reachability must hold too. if not path_exists(snap, "10.0.0.1", "10.42.0.10"): failures.append("no path from core1-den to cust-1042") for f in failures: print(f"FAIL {f}") if failures: sys.exit(1) print(f"PASS snapshot {snap}: 0 failing checks, isolation holds, core reachable") if __name__ == "__main__": main() ``` The intent checks come from Forward itself, defined once in the UI or as NQE queries and run against every snapshot. Pulling the list of failures is the cheap way to inherit all of them. The two path queries are the change-specific part: for a VLAN change on an access port, the question is whether the customer on that port can suddenly reach a neighbor they shouldn't. For compliance rules the team has but Forward doesn't ship, NQE is a query language over the snapshot. A rule that every XR device points at the approved NTP servers: ```text foreach d in network.devices where d.platform.vendor == "CISCO" && d.platform.os == "IOS_XR" let servers = (foreach s in d.ntp.servers select s) where !(servers == ["10.0.0.5", "10.0.0.6"]) select { device: d.name, ntp: servers } ``` Any rows returned are devices out of policy, and the pipeline treats a non-empty result the same as a failed check. ### Wiring both into the pipeline The change pipeline from [the approach page](/services#how) has a pre-check stage before the dry run and a post-check stage after apply. Both call the same two scripts: ```yaml pre-check: stage: pre-check script: - pyats learn bgp isis interface --testbed-file testbed.yml --devices $DEVICES --output pre - python3 verify_forward.py artifacts: paths: [pre/] post-check: stage: post-check needs: [apply] script: - pyats learn bgp isis interface --testbed-file testbed.yml --devices $DEVICES --output post - pyats diff pre post --exclude "[Uu]ptime" "last_reset" --output diffs - python3 verify_forward.py artifacts: paths: [post/, diffs/] when: always ``` `when: always` on the post-check artifacts matters. The diff is most valuable when the stage fails, and GitLab's default is to drop artifacts on failure. A failed pre-check blocks the change before anything is pushed, which usually means the network was already in a state nobody knew about. A failed post-check stops the pipeline after apply with the evidence attached, and the runbook for that change says what to do next: roll back automatically if the rollback criteria say so, or page the owner with the diff in hand. ### What this gives a team Every change now produces four artifacts: pre state, post state, the diff between them, and a Forward verdict with a snapshot ID. They live on the pipeline run, linked from the ticket. Six months later, when someone asks what changed on pe2-den in October, the answer is a link, not an archaeology project. That is the difference between automation that pushes config and automation that operators trust. The config push is the easy half.

Why NSO projects stall, and the 7 fixes

Sun, 11 Oct 2026 00:00:00 GMT

Most NSO projects don't fail. They stall, quietly, around month six. Nobody cancels anything. The licenses get renewed, the server stays up, and there's still a line for it in the roadmap. But new services stop shipping, the people who were excited about it move on to other work, and changes go back to being typed into devices by hand. When we are asked to look at one of these projects, the cause is almost always some mix of the same seven patterns. Each one leaves a trace in NSO itself. ### 1. The first service was too big The first service tried to cover every variant: every platform, every edge case, every option a customer has ever ordered. Months later it still isn't done, and nothing is in production to show for the work. ```text admin@ncs# show packages package l3vpn package-version packages package l3vpn package-version 0.0.14 admin@ncs# show running-config services l3vpn | count Count: 0 lines ``` Fourteen package versions and no instances. The model has been redesigned more times than it has been used. The fix is to ship one narrow service within 30 days. Pick something frequent, low-risk, and boring, like a VLAN or an interface description. The whole model fits on a screen: ```text module vlan { namespace "http://example.net/vlan"; prefix vlan; import tailf-ncs { prefix ncs; } list vlan { key name; uses ncs:service-data; ncs:servicepoint vlan-servicepoint; leaf name { type string; } leaf vlan-id { type uint16 { range "1..4094"; } } leaf-list device { type leafref { path "/ncs:devices/ncs:device/ncs:name"; } } } } ``` A small service that operators use every day teaches a team more than a large one that's always almost finished. ### 2. Brownfield config never got reconciled Half the devices show out-of-sync, and nobody wants to run `sync-from` because nobody is sure what it will pull in or overwrite. So NSO's view of the network drifts further from the real network every week. ```text admin@ncs# devices check-sync sync-result { device pe1-den result in-sync } sync-result { device pe2-den result out-of-sync } ``` The fix is to make drift visible and boring. Look before pulling: ```text admin@ncs# devices device pe2-den compare-config diff devices { device pe2-den { config { interface GigabitEthernet0/0/3 { - ip access-group cust-1042-in in } } } } ``` That minus sign means the device has something NSO doesn't. Somebody removed an access-group by hand. Decide whether that change is wanted, then `sync-from` to accept it or push the service again to reassert it. Run `check-sync` on a schedule, publish the count, and work through a reconciliation plan device by device. Then agree on a policy for out-of-band changes, so the network doesn't drift again the week after the reconciliation finishes. The full procedure is in [Reconciling out-of-sync devices in NSO](/blog/reconciling-out-of-sync-devices-in-nso). ### 3. NSO and NED versions drifted Upgrades keep getting postponed because nobody knows what will break. Each skipped release makes the next upgrade bigger and scarier, until the platform is several versions behind and stuck there. ```text admin@ncs# show ncs-state version ncs-state version 5.7.6 admin@ncs# show packages package package-version | include cisco-iosxr packages package cisco-iosxr-cli-7.33 ``` NSO 5.7 reached end of support, and a NED from the same era with it. Every bug the team hits has been fixed in a version they can't reach without a jump. The fix is a version matrix that lists NSO, every NED, and every package, plus an upgrade path tested in netsim before it touches production. NED upgrades are a per-device action, which makes them safe to roll out in batches: ```text admin@ncs# devices device pe2-den migrate new-ned-id cisco-iosxr-cli-7.52 dry-run modified-path { path /ncs:devices/device{pe2-den}/config/cisco-ios-xr:router/bgp info "Schema changed" } ``` After that, upgrade on a routine cadence so each one stays small. ### 4. No CI or testing Packages get edited directly on the server. There's no history, no review, and no way to know whether a change breaks a service until it does, in production. ```bash $ ls /var/opt/ncs/packages/ vlan vlan.bak vlan.bak2 vlan-working ``` The fix is to make git the source of truth and run every change through a pipeline built on netsim devices. Before anything is committed, someone reviews the dry-run diff: ```text admin@ncs(config)# commit dry-run outformat native native { device { name pe2-den data interface GigabitEthernet0/0/1.1042 description vlan-1042 encapsulation dot1q 1042 ! } } ``` If the diff surprises the reviewer, it would have surprised everyone in production too. [Building that pipeline on GitLab](/blog/ci-pipeline-for-nso-packages-with-netsim) takes a day. ### 5. Nobody owns the service model The YANG models were written by developers without much input from the operators who will use them. The models expose the wrong inputs, miss the options people actually order, and nobody has the authority to fix them. The trace is in the data. Count the leaves operators always set to the same value, or never set at all: ```text admin@ncs# show running-config services l3vpn | include qos-policy | count Count: 0 lines ``` A leaf that is never set is a leaf that should not be in the model. The fix is a named owner for each service model and a model review with operators before anything gets built. Twenty minutes walking an engineer through the inputs saves weeks of rework later. ### 6. The one developer left One person built most of the packages, and that person moved on. What's left works, mostly, but nobody understands it well enough to change it. ```bash $ git shortlog -sn packages/ 214 jsmith 3 kpatel ``` The fix is to document what exists, refactor the critical paths first, and pair with the team's own staff while doing it. Start with `show running-config services | display service-meta-data`, which shows which service owns every line of device config, because that is the map nobody drew. The work isn't finished until there's a handoff runbook the team has actually used. ### 7. Nothing consumes it NSO works, but requests still arrive as tickets, and an engineer still reads each ticket and types the change. NSO is just one more tool they could use if they had time. ```bash $ grep -c restconf /var/log/ncs/audit.log 0 ``` No RESTCONF calls means no system is driving NSO. Only people are. The fix is to connect NSO to the request path, through a portal, an API, or the ticketing system itself. One call per change: ```bash $ curl -s -u "$NSO_USER:$NSO_PASS" -X POST \ -H "Content-Type: application/yang-data+json" \ -d '{"vlan:vlan":[{"name":"CHG0048211","vlan-id":1042,"device":["pe2-den"]}]}' \ "https://nso.example.net/restconf/data?dry-run=native" ``` Run it with `dry-run=native` from the ticket first, attach the result to the change record, and run it for real once the change is approved. Make it the normal way changes happen in the change process, and track adoption so the team can see whether it's working. ### A quick self-check If three of these sound familiar, the project is stalled, not slow. The good news is that none of them are technical dead ends. Each one has a fix that fits into a few weeks of focused work. If a project got stuck in a way that isn't on this list, or got moving again, tell us about it in the comments.

The SQLite index that only failed on macOS

Sat, 05 Sep 2026 00:00:00 GMT

A tester opened NetStacks on a Mac that had been running the app since spring and got this in the sidecar log: `Failed to initialize database: Migration("error returned from database: (code: 1) no such column: execution_id")` The same build started cleanly on a Linux box, on a fresh macOS install, and in every test run. Only a database that had been created by an older version failed. That is the shape of a migration ordering bug, and it is worth walking through because the fix was not "add a migration". **How the agent builds its database** The NetStacks agent is a Rust process that owns a SQLite file. On startup it runs `schema.sql`, which is a plain list of `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` statements, and then runs a list of migrations that alter existing tables: add a column here, rename one there. New installs get the current schema in one shot. Old installs get the old tables and then the migrations bring them forward. A recent change added an `execution_id` column to the `snapshots` table so that a config snapshot can be tied to the MOP execution that produced it. The schema file got the column in the table definition and an index on it, and a migration added the column to existing databases. **Why it only failed on the tester's machine** On a fresh install, `CREATE TABLE` creates the table with the column, then `CREATE INDEX` on that column succeeds, then the migration sees the column already exists and skips. Fine. On an old install, `CREATE TABLE IF NOT EXISTS` does nothing because the table exists without the new column. Then `CREATE INDEX ... (execution_id)` runs, and there is no such column yet, because the migration that adds it has not run. SQLite refuses. The agent aborts. The app shows a spinner forever. The two mechanisms were correct on their own and wrong in that order. Index statements in the schema file assume the table is already at the current shape, which is only true when the file created the table. **The first fix, and why we did not keep it** The obvious patch is to move the `CREATE INDEX` into the migration that adds the column. That works, and it is what shipped first. But it means every future column-plus-index change has to remember the rule, and the schema file stops being the full description of the database. Someone would forget, and we would be back here. **The fix that stayed** Startup now splits the schema file into two passes. Everything that is not a `CREATE INDEX` runs first, then the migrations run, then every `CREATE INDEX` runs. Indexes stay declared next to their tables in the one file, and they always execute against a table that is at the current shape. The splitter is deliberately dumb: it looks for lines that start with `CREATE INDEX` or `CREATE UNIQUE INDEX`, and a test asserts that every index statement in the file was held back. If someone writes an index in a way the splitter does not recognise, the test fails before the app does. **What we would still change** The migration that adds the column now checks whether the column exists before trying to add it. It should have done that from the start. We also want the test suite to run against a snapshot of a genuinely old database file, not just a fresh one with the new schema; that would have caught this before a tester did. If you keep a schema file and a migrations list in the same project, check which one your indexes believe in.

Why Ctrl+V opens a dialog now (and how to turn it off)

Thu, 03 Sep 2026 00:00:00 GMT

Pasting into a router is the most dangerous thing most of us do every day. A stray blank line commits a half-typed command. A Windows line ending turns `no shutdown` into `no shutdown^M` on some platforms. A comment line pasted into IOS becomes a syntax error that scrolls the real error off the screen. SecureCRT solved this years ago with a confirmation dialog for multi-line pastes. Several people asked for the same thing, so it is in NetStacks now. This post covers the decisions, because two of them are the opposite of what we shipped first. **What happens on Ctrl+V** If the clipboard holds a single line, it is pasted. Nothing changes for the case that is almost always safe. If it holds two or more lines, a dialog opens. The left pane is the clipboard, and you can edit it. The right pane is what the device will receive after the flavour's preset has been applied. Ctrl+Enter sends it. Escape cancels. The presets are the interesting part. Each CLI flavour has a default chain of transforms. The IOS preset normalises line endings, strips trailing whitespace, collapses blank lines and removes `!` comment lines. The Junos preset normalises line endings and trailing whitespace and cleans up `display set` output so it can be pasted as `set` commands. You can edit the chains, add your own presets, and choose which flavours each applies to. **Why we swapped the chords** The first build put the dialog on Ctrl+Shift+V and left Ctrl+V as a plain paste, on the theory that a new behaviour should live on a new key. The feedback was immediate and unanimous: the dialog is the safe path, so it belongs on the key people already press without thinking. Ctrl+V now opens the dialog; Ctrl+Shift+V pastes exactly what the clipboard holds, no transforms, no dialog. Shift+Insert (paste) and Ctrl+Insert (copy) keep working as fixed aliases, and the main clipboard chords are rebindable under Settings → Keyboard. **Turning it off** Settings → Clipboard has a master switch, "Enable advanced paste". Off means every paste chord is a plain paste, no preset and no dialog. Below it, the line threshold controls when the dialog appears; the default of 2 means every multi-line paste, and 1 confirms single lines too. **What the history keeps** Everything copied inside NetStacks is recorded with where it came from: the session, the device, the CLI flavour, or the document and MOP if that is where you copied from. Open it with Ctrl+Shift+H, search it, pin the clips you reuse, and paste straight into the active terminal. Two things it does not do. It does not read your OS clipboard in the background; only copies made inside the app are recorded, plus anything you import on purpose. And it runs every clip through the same credential sanitiser the AI features use before storing it. `enable secret 5 $1$...` is stored as `enable secret 5 [REDACTED]`. The OS clipboard still gets the real text; only the history is scrubbed. Unpinned clips expire after 24 hours by default, and the store is capped at 500. Both are settings. **Things we learned building it** Dragging a dialog by its title bar used to close it. The browser dispatches the click at the end of a drag to the common ancestor of where the press started and ended, which was the backdrop, and our dismiss hook treated that as a backdrop click. Twenty-four dialogs shared the hook. The fix was to remember where the press began. If your app has movable dialogs and a click-outside-to-close hook, try dragging one onto the backdrop.

What the AI actually sees when you ask it about a router

Thu, 20 Aug 2026 00:00:00 GMT

The most common question we get about the AI features is not "which model is best" but "what does it see". Fair question. If you are going to paste a router's neighbour table into a language model, you should know exactly what goes with it. **The pieces** When you ask NetBot a question with a terminal tab active, the request is assembled from these parts, behind a fixed set of safety rules: 1. The system prompt for the current agent mode, Auto Pilot or Overlord. These are visible under Settings → Prompts and you can edit them. 2. The AI engineer profile: the name you gave the assistant, the working style and autonomy level you chose, and whichever knowledge packs fit inside the token budget you set. 3. Session context: which session you are focused on and the live state of the terminal, such as the CLI mode you are in, whether there are uncommitted changes and the commands you just ran. No credentials. 4. Device memory: the role, standing instructions and notes recorded for that device, plus the team's incident entries for it, most recent first. 5. Recent terminal output, up to a configurable number of lines, taken from the scrollback of the active tab. 6. Your question, plus the selected text if you used the context menu. Every one of those parts passes through the sanitiser before it leaves the machine. **What the sanitiser removes** The sanitiser is a set of patterns for things that should never reach a model: `enable secret` and type 7 passwords, SNMP communities and SNMPv3 keys, TACACS and RADIUS keys, Juniper `$9$` secrets, private-key and certificate blocks, and the general shapes of API tokens, passwords and shared secrets. Matches are replaced with `[REDACTED]` or a labelled variant such as `[PRIVATE-KEY-REDACTED]`. Those patterns are always on. You can add your own, switch on optional redaction of IP addresses, MAC addresses and hostnames, and test a sample against the whole set under Settings → AI → AI Data Security. Redaction is applied to the system prompt, the transcript, device memory, and the tool results that come back when the assistant runs a command. It is applied to your own question too, so a secret you paste into the chat by accident does not leave the machine. **What is fixed on purpose** Most prompts are yours to change, including the agent operating guide that is appended to every autonomous agent. A few pieces are not. The safety rules the agent puts in front of every request are fixed: never guess device state, no destructive commands without human approval, never fabricate CLI output, always identify as an AI. So are the rules that stop the assistant from exiting or committing a configuration session you left half-finished. And in the desktop app, commands the assistant runs on a device pass a read-only filter until you turn on AI config mode. We would rather be told these are too conservative than have them silently edited on a machine that has write access to a core. **Where it goes** Straight to the provider you configured. NetStacks does not run a proxy or a relay; the agent on your machine makes the HTTPS request with your key, to Anthropic, OpenAI, OpenRouter, your Ollama, LiteLLM or the OpenAI-compatible URL you gave it. For private gateways, per-provider base URL and TLS settings live under Settings → AI. If you are connected to a NetStacks Controller, the request goes to your Controller instead, which applies your organisation's sanitiser and calls the provider your administrator configured. **How to check for yourself** Open Settings → Prompts and open any prompt to read it. Then run a request with a local model through Ollama or a stub server and read the request body. We do exactly this in our own tests, with a stub that logs every request it receives, and it is how we caught an onboarding prompt leaking into ordinary helper calls earlier this year.

Device memory: putting incident history where someone will read it

Wed, 12 Aug 2026 00:00:00 GMT

Every team has a wiki full of incident write-ups and an on-call engineer who does not know they exist. The write-up is filed under the ticket number. The engineer is looking at a hostname at three in the morning. Those two never meet. Device memory is our attempt to fix the meeting point. Each session in NetStacks can carry a list of entries: the issue in one line, the root cause, the resolution, the commands that mattered and a ticket reference. Entries are written by whoever fixed it, usually in a minute, from the terminal they fixed it in. **What it looks like in practice** You connect to `core-rtr-01`. Before you type anything, a card appears in the corner: "c.davis noted: TenGigabitEthernet0/1/2 flapping; OSPF adjacency to dist-sw-02 reset 14 times in 20 minutes. INC-4821." If you are here because Te0/1/2 is flapping again, you have just saved twenty minutes. If you are not, you dismiss it and it stays out of the way. The same entry is fed to the assistant when you ask about this device. Ask why the interface is down and the answer starts from "this happened before, it was a dirty fibre on patch panel port 2/17, check the optic first" instead of a generic list of causes. **Rules we settled on** Entries belong to the device, not to a person. The author is recorded, but the entry does not live in their notebook. The card shows the latest entry only, with a count if there are more. Showing everything on connect trains people to dismiss it without reading. Ticket references are plain text. We do not integrate with a ticketing system to store them; we just make them easy to copy. Entries pass through the credential sanitiser like everything else that reaches a model, so a pasted `show run` fragment with a community string in it does not end up in a prompt. **Local and shared** In the free desktop app, memory lives in the local database with your sessions. With the NetStacks Controller it is shared across the organisation, so the entry the assistant reads about a device might have been written by someone on another continent last week. That is the point. **What it is not** It is not a CMDB, an incident manager or a replacement for the post-mortem. It is the two sentences from the post-mortem that the next engineer needs, in the one place we know they will look.

Writing a MOP that can fail safely

Thu, 30 Jul 2026 00:00:00 GMT

Here is a change we ran in the lab recently: prefer one transit provider over another during the other one's maintenance window. It is small, it is common, and it is exactly the kind of change that gets pasted straight into a router because "it is just a route-map". The plan has nine steps. Three pre-checks, two changes, three post-checks, one rollback. Each step is a command, a one-line description, and the output we expect to see. **Pre-checks that can say no** `show ip bgp summary` with expected output `Established`. If a peer is already down, the change is not the right thing to be doing right now, and the plan halts before anything is modified. `show ip route 0.0.0.0` expecting `203.0.113.1`. We are about to prefer this next hop; confirm it is there. `show run | section route-map RM-TRANSIT-IN` with no expected output. This one exists to capture the current policy into the execution record, so the rollback has something to be compared against. **Changes** Enter the route-map, set `local-preference 150`, end. Then `clear ip bgp 203.0.113.1 soft in`. Soft, not hard. The description says why, so whoever runs this at two in the morning does not "fix" it to a hard clear. **Post-checks that mean something** `show ip bgp 8.8.8.0/24` expecting `203.0.113.1` as the best path. `show ip bgp summary` expecting `Established` again, because the soft clear should not have dropped anything. `ping 8.8.8.8 source Loopback0` expecting `Success rate is 100 percent`. Reachability is the check that matters; the others explain it if it fails. **Rollback written first** Remove the `set local-preference`, soft clear again. It is in the plan before the plan is approved, and it is one click during execution. **What the tooling adds** Typed steps mean the executor knows which phase it is in. Pre-checks run, and if any expected output does not match, execution pauses with the mismatch highlighted. You choose whether to continue, retry, skip or roll back. Control mode can be manual (you run each step), auto-run (it proceeds while checks pass) or AI pilot (the assistant reads each output and, depending on the trust level you pick, comments on it, proposes the next step for you to approve, or runs the phases itself and stops on any failure). Strategy can be sequential or parallel across devices. Steps do not have to be CLI commands. A pre-check can be a saved script or a REST quick action against your monitoring system: "is anything alarming on this device right now?" is a fine pre-check and it is not a show command. Plans export as `.mop.json`, so they can live in a repository and be reviewed like code. Import brings them back with their variables. **The habit that matters most** Write the expected output. A step with a command and no expectation is a script. A step with an expectation is a check. The whole value of the format is that the executor can tell the difference and stop.

Bring your own model: running NetStacks against Ollama on a laptop

Thu, 16 Jul 2026 00:00:00 GMT

Some networks cannot send a neighbour table to a cloud API, full stop. NetStacks was built so that the AI features still work in that situation: the agent on your machine talks directly to whatever provider you configure, and one of the built-in providers is Ollama. **Setup** Install Ollama and pull a model. For CLI output analysis we have had the best results from the larger instruction-tuned models; the smaller ones are fine for explaining a log line and poor at reasoning across a whole `show ip bgp summary`. In NetStacks, open Settings → AI, enable Ollama, and set the base URL if it is not the default `http://localhost:11434`. Pick a model from the list the server returns. Save and test. If the first-run wizard is still open, the same steps are on its AI page. If your Ollama runs on another machine, the per-provider overrides let you set a different URL and disable TLS verification for a self-signed certificate. Keep that on a management network. **What works well** Explaining output. Select three syslog lines, right-click, Explain. Local models are good at this and it is the most frequent use. Command suggestions with the CLI flavour attached. Because NetStacks tells the model the device is Cisco IOS-XE, or Junos, or EOS, you get the right syntax far more often than a bare chat window gives you. Turning a troubleshooting transcript into a write-up. Long context, low stakes, and you are going to edit it anyway. **What does not** Letting a small local model run commands on its own. An agent that executes commands without asking needs a model that reliably produces well-formed tool calls and knows when to stop. Test it on a lab device with a model you trust before you let it near production, set the permission to Ask so each command waits for your approval, and leave AI config mode off so the assistant stays read-only on devices. Those guardrails exist for a reason. Latency. A laptop GPU is slower than a datacentre. Command autocomplete uses the API after a short debounce and can feel sluggish locally; there is a switch for it under Settings → AI. **Check what it sends** Because the request goes straight from the agent to Ollama, you can watch it. Run Ollama with debug logging, or put a tiny logging proxy in front of it, and read the request body. You will see the system prompt with the mode prompt and the profile, the session context, the device memory, the last lines of output with credentials replaced by `[REDACTED]`, and your question. No passwords. **Mixing providers** You can enable more than one provider and switch from the footer of the AI panel per conversation. A common pattern is a local model as the default for everything that touches device output, and a cloud model enabled for document drafting where the input is your own prose.

Reading show ip bgp summary faster than the assistant does

Thu, 25 Jun 2026 00:00:00 GMT

`show ip bgp summary` is the first command most of us type when BGP is in the incident channel. It is also one of the most misread. Here is how to read it in the ten seconds before you decide what to run next. **The header** `BGP router identifier` and `local AS number`: confirm you are on the box you think you are on. It sounds obvious; it has saved careers. `BGP table version` and `main routing table version`: if these are far apart and moving, the box is still converging. Wait before you judge anything below. The memory lines are noise until they are not. A router that has `941712 network entries` where it had 20000 last week has started receiving full tables from someone. **The neighbour table** `Neighbor`: the peer address. Match it against your documentation, not your memory. `V`: BGP version. Always 4. If you ever see anything else, stop reading this and start reading the vendor's bug database. `AS`: the remote AS. Same number as your own AS means iBGP; the rules for next-hop and route reflection apply. Different means eBGP. `MsgRcvd` / `MsgSent`: they should both be climbing. A peer with a large received count and a sent count that stopped moving is a peer that stopped listening to you; check the TCP session and your outbound policy. `TblVer`: the peer's table version. Should match the header's table version on a converged box. A peer stuck behind is a peer you are slow to update, usually because of a heavy outbound policy or a small hold queue. `InQ` / `OutQ`: non-zero and growing is a stuck peer or an overloaded box. Non-zero and steady is normal on a busy edge. `Up/Down`: the time since the last state change. `3w2d` is boring, which is what you want. `00:02:17` on a peer that should be boring is the peer to look at. `State/PfxRcd`: the column that matters. A number is a prefix count and means Established. A word is a state, and every word is a problem: - `Idle`: not trying. Usually an admin shutdown, a missing route to the peer, or the neighbour was just configured. - `Connect`: trying TCP and failing. Reachability, an ACL, or the wrong update-source. - `Active`: TCP failed and it is retrying. Same causes as Connect; the one you will see most. - `OpenSent` / `OpenConfirm`: TCP is up, BGP is negotiating. If a peer is stuck here for more than a few seconds, look for an AS mismatch, an MD5 password mismatch, or a capability the other side refuses. **The patterns** One peer `Active`, everything else Established with stable counts: it is that peer or their side. Check `show ip bgp neighbors | include Last reset|Notification` before you touch anything. Everything `Active` at once: it is you. Interface, routing to the peers, or someone just changed the router-id. Established with a prefix count of 0 from a transit provider: you are connected and receiving nothing. Inbound policy, a max-prefix that tripped and was reset, or they are sending you a session and no routes on purpose. A prefix count that jumped by an order of magnitude: a leak, yours or theirs. Look at `show ip bgp summary` on the peer's other neighbours next. **Why we wrote this on a product blog** Because the assistant in NetStacks is given this table when you ask about a BGP problem, and its answer is only as good as the reading above. It will tell you `198.51.100.9` is `Active` and that the other peers are healthy. You should already know that, and be asking it the next question.

Telnet in 2026: console servers, lab gear, and why we kept it

Fri, 12 Jun 2026 00:00:00 GMT

Every so often someone asks why a terminal built in 2026 supports Telnet at all. The short answer is that the console server in the rack next to you does, and the fifteen-year-old switch in the branch office does, and the lab full of virtual routers definitely does. **Where it shows up** Console servers. A terminal server maps each serial line to a TCP port, and the way you reach it is Telnet to that port, sometimes SSH to the server first. NetStacks models this directly: a session has an optional console host, port and protocol alongside its normal host, so the same device record can be reached in-band over SSH or out-of-band over the console when in-band is what broke. Lab gear. Virtual routers in a lab expose their consoles over Telnet ports. If you are testing a change before a window, that is where you are typing. Old hardware. Devices that predate SSH, or that shipped with SSH implementations modern clients refuse to talk to. For the second group, NetStacks offers the older key exchange and cipher algorithms alongside the modern ones, and negotiation picks the strongest set both sides support. **What the Telnet client does** Option negotiation for echo, terminal type and window size, so the device knows how wide your terminal is and resizes with you. Automatic login when the device prompts for a username or password, using the credential profile, so the password is not sitting in a script somewhere. Everything after that is the same terminal as SSH: same clipboard handling, same AI context, same recording and logging. **What it does not do** Pretend to be secure. Telnet is clear text end to end. The credential sanitiser still redacts secrets from anything sent to a model, but nothing can redact them from the wire. **Why keep it in the same app** Because the alternative is two terminals. The person who has to reach a device over its console during an outage is the same person who needs the device's incident history, the runbook and the assistant right then. Splitting the protocols across tools splits the context too. If you never use Telnet, you will never see it; SSH is the default for every new session and the protocol field is one dropdown.

AI Isn't Going to Fix Your BGP Convergence Time — But Here's What Will

Fri, 22 May 2026 00:00:00 GMT

When was the last time an AI fixed your BGP convergence problem without you asking? Never. That's the gap between what vendors sell and what network engineers actually need. The AI networking conversation is broken. Vendors show demos where a chatbot analyzes an outage, generates a fix, and pushes it — all while the audience nods politely and nobody asks what happens when it's wrong. Real networks don't work that way. Nobody pushes an AI-suggested BGP config change at 2 AM without understanding exactly what it does and why. **The Convergence Problem No One Talks About** When a BGP session drops across a multi-path fabric, you don't need a chatbot to tell you "BGP session went down." You need: every affected peer identified, the exact point of failure isolated across AS boundaries, and the downstream impact quantified across your VRFs. And you need it in the 90 seconds before your monitoring alert becomes an outage ticket. What happens in reality? You're juggling three SSH sessions, a show ip bgp summary that's still rendering, and a Slack thread where someone is asking "is it the fiber or the configuration?" Your legacy terminal gives you a blinking cursor. Your NMS shows you a red dot. Nobody is helping. **Where NetStacks Actually Helps** NetStacks doesn't try to be your autonomous NOC replacement. It does something more practical: when you're troubleshooting, it reads the CLI output you've already gathered, understands protocol state, and suggests the next diagnostic step. It's the difference between staring at truncated show bgp neighbors output and having someone immediately point out that the Hold Timer expired on one side because of asymmetric MTU on the transit link. Multi-device commands are where this gets powerful. Select the affected sessions and broadcast a show bgp summary to all of them at once; every device's result comes back in one dialog. Hand that output to the AI and it flags which neighbors are in Active state and which have zero prefixes received, while the topology map shows you where those devices sit. Five minutes of gathering output becomes a minute. **The Automation Gap Nobody Admits** Here's the uncomfortable truth: most network teams that claim to be doing "AI-driven automation" are running scheduled scripts that collect show commands, parse the output with regex, and dump it into a dashboard nobody looks at until Something Is Red appears. That's monitoring, not automation. And the AI piece is usually a bolt-on chatbot that you query separately — another tab, another context switch, another workflow. The gap isn't in the AI models. GPT-4 already understands BGP better than most junior engineers. The gap is in the workflow integration. The moment between seeing a problem and knowing what command to run next is where engineers lose time. That's what needs to be closed — not with autonomous AI, but with AI that operates in the same terminal session where you're already working. **What to Actually Do About Convergence Time** 1. Pre-stage diagnostic templates. When BGP drops, you should already have a Jinja2 template that queries every relevant show command across every affected device and collects the results into a structured report. No typing during the outage. 2. Use multi-device commands to run them. In NetStacks you can keep the commands as snippets so nobody types them from memory, send them with Broadcast Command to every session you select and collect the results in one place, or turn on Multi-Send in a split view of up to four tabs and watch the output side by side as you type. 3. Let AI highlight the anomalies. Once you have the output, select the confusing bits. The AI assistant compares the output against what it knows about BGP state machines and tells you exactly which neighbor is stuck and why. Not after you finish troubleshooting — during. 4. Map it visually. The topology view shows you the outage boundary. Double-click a device to SSH into it. No context switching between your NMS dashboard and your terminal. This isn't autonomous networking. It's augmented networking. And it's what you can implement today with tools you already have access to. The teams that will actually move the needle aren't the ones waiting for fully autonomous AI. They're the ones who figure out how to close the gap between gathering data and understanding it. That gap is measured in minutes during an outage — and minutes compound into hours per engineer per week that you're never getting back.

Running Methods of Procedure in Production Without Praying: The MOP Pattern That Actually Works

Wed, 20 May 2026 00:00:00 GMT

Every post-mortem includes the phrase "no MOP was followed" or "the MOP was incomplete." MOPs — Methods of Procedure — are supposed to be the structured, pre-approved, step-by-step change plans that prevent production disasters. And somehow, they fail every single time. The problem isn't that network engineers don't understand the need for structured change procedures. The problem is that most MOP tooling turns them into 40-line Word documents that nobody reads, no one can execute from directly, and that provide zero safety net when step 23 produces output the author didn't anticipate. **Why MOPs Fail in the Real World** A proper MOP has three sections that most tools get wrong: Pre-checks verify the current state matches your assumptions before you touch anything. Is the BGP session up? Are the routes present? Is the target device reachable? If any pre-check fails, the MOP should stop immediately — but most teams run these manually and just "assume things are fine." Change steps are the actual configuration commands. They need to be sequenced, validated, and — critically — each step needs to define what success looks like. "Add the route-map" isn't a complete step. "Add the route-map and verify it appears in show run | section route-map" is. Post-checks confirm the change achieved what it was supposed to. Not "the device is reachable" — "the new next-hop is installed in the FIB and traffic is flowing through it." The gap between what a MOP should be and what most teams actually use is enormous. Most MOPs live in Confluence, get copy-pasted into SSH sessions, and have zero programmatic validation of results. **The Approval Workflow Nobody Builds** Here's a gap most automation platforms ignore: MOPs need approval workflows. Not just "someone reviewed the Word doc." Actual, programmatic, pre-change approval gates where: - The MOP author defines who needs to approve - Approvers review it against the change window - Each step requires sign-off before execution - Rollback is pre-approved and pre-defined Without this, you're just running scripts. The MOP isn't a script — it's a governance boundary that protects production. **How NetStacks Handles This Differently** NetStacks treats MOPs as first-class automation objects, not scripts or documents. A MOP in NetStacks has structured pre-checks that run automatically and halt execution if conditions don't match. Each change step has its own success criteria that gets evaluated against the device output in real-time. Post-checks aren't an afterthought — they're baked into the execution flow. With the NetStacks Controller, MOPs go through review: the author submits the plan, a reviewer approves or rejects it with a comment, and the Terminal shows the plan as awaiting approval until it has been signed off. Everyone can see whether the plan they are about to run was reviewed. And when something goes wrong — because something always goes wrong — the rollback is already written into the plan. You don't figure out how to undo the change while the NOC is watching. You hit Roll back and NetStacks runs the rollback steps you defined, on one device or on all of them. **The MOP Pattern Teams Should Copy** Even if you're not using NetStacks, here's the pattern every team should adopt: 1. Every MOP gets pre-checks, change steps with success criteria, post-checks, and rollback — all structured, not free-text. 2. Pre-checks run automatically. If BGP isn't established on the peer you're modifying, the MOP stops. No human judgment call needed. 3. Each change step defines its own validation. After "ip route 10.0.0.0/8 192.168.1.1," the MOP immediately runs "show ip route 10.0.0.0/8" and verifies the next-hop. 4. Post-checks are independent verification, not just re-running the change command. They answer: "Is traffic actually flowing the way we intended?" 5. Rollback is defined before execution, not after failure. Every successful MOP should end with "post-check passed, no rollback needed." The teams that take MOPs seriously don't just write better documents. They build enforcement into the tooling so the process can't be skipped. That's the difference between "we have MOPs" and "our MOPs actually prevent outages."

Your NetBox Isn't a Source of Truth (Yet) — Here's the Missing Piece Every Team Skips

Mon, 18 May 2026 00:00:00 GMT

Almost every network team has NetBox installed. Far fewer actually use it as their source of truth. The gap between "we have NetBox" and "NetBox is our source of truth" is where most network automation projects die. The difference isn't the data model. NetBox already has interfaces, IPs, prefixes, VLANs, VRFs, circuits, and device roles. The difference is in the validation loop between what NetBox says and what your network actually shows. **The Drift Nobody Measures** When was the last time you verified that NetBox accurately reflects your network? Not "NetBox is up to date because we follow process." I mean, actually SSH'd into devices and compared show run against the NetBox data. Most teams can't answer that question because nobody has built the verification step. You enter data into NetBox, and it stays there. Network changes happen through SSH sessions, change tickets, and Slack messages. NetBox gets updated "when there's time" — which is never during an outage window or a busy sprint. This creates silent drift. NetBox says an interface is in VLAN 100. It's actually in VLAN 200. The interface shows as "up" in NetBox. The SFP failed three weeks ago. You're rendering stack configs from NetBox data that hasn't matched reality since the last person who bothered to update it. **The Integration Most Teams Build** The standard NetBox integration pattern looks like this: build a script that pulls device data from NetBox, generates configs, and pushes them to devices. This works — until NetBox is wrong. And because there's no continuous verification, NetBox is wrong more often than anyone admits. The missing piece is the reverse direction. Not just NetBox → device, but device → NetBox. Every time you connect to a device, you should be able to: - Snapshot the current config - Compare it against the expected state from NetBox - Flag any deltas immediately - Propose NetBox updates when the delta is intentional This isn't NetBox polling or SNMP scraping. This is interactive, engineer-driven verification at the moment of connection — when you already have the session open and the data is right there. **How NetStacks Closes This Loop** NetStacks brings NetBox to the session level, not just the config-generation level. With a NetBox source configured: - Your NetBox devices become terminal sessions, with credential profiles and CLI flavours mapped from site, role and platform - Hovering an IP address or interface name in the terminal looks it up in NetBox, right next to the output you are reading - The topology view can be built from NetBox data, and the live SNMP overlay shows you what those links are actually doing - The assistant can query NetBox while it helps you troubleshoot, so the recorded data and the device output are in the same conversation Discovery takes this further. It walks LLDP and CDP neighbours from the devices you point it at and builds a topology from what the network itself reports. Put that next to the topology you imported from NetBox and the drift shows up as a visible gap between what you discovered and what NetBox says. Fixing the record is still your call, but the gap is hard to ignore. **The Pattern Every Team Should Adopt** 1. Connect NetBox to your terminal workflow. If your SoT data isn't visible at the point of connection, it's just a database — not truth. 2. Run device discovery independently of NetBox. Compare the discovered topology against NetBox. The delta is your drift measurement. 3. Snapshot configs during every session. Not scheduled — during the session when you're already there. Compare against NetBox-derived expected state. 4. Make drift visible and actionable. If NetBox says VLAN 100 and the device says VLAN 200, the delta should be surfaced immediately, not discovered during an outage. 5. Treat NetBox as a living system. The moment you start treating it as "something we update quarterly," it becomes a historical record, not a source of truth. The teams who successfully use NetBox as an SoT don't have better discipline. They have better tooling that makes verification effortless. The gap isn't in the data model — it's in the feedback loop between what's recorded and what's real.

Why Your Terminal Is Your Biggest Automation Bottleneck (And Nobody Realizes It)

Fri, 15 May 2026 00:00:00 GMT

Network teams have invested millions in automation infrastructure. Ansible Tower, Python libraries, NetBox instances, Windmill platforms, CI/CD pipelines for configs. And yet, the moment something breaks, everyone opens a terminal and starts typing show commands. The terminal is where 80% of daily network engineering happens. It's where outages are investigated, changes are verified, and knowledge is captured. And for most teams, the terminal experience hasn't meaningfully changed since PuTTY in 2005. **The Terminal Gap Nobody Measures** Think about your current terminal workflow. How many SSH sessions are open right now? Where does the output go when you need to reference it later? How do you organize sessions across projects, sites, and device types? When you need to run the same command across 20 devices, what's the process? Most engineers use a combination of PuTTY, SecureCRT, or Terminal.app with saved session files scattered across folders. Output is captured in local log files (if capture is even enabled), which are never searched, never correlated, and almost never shared with the team. When someone asks "what did you see on that device?", you're scrolling through a local log file and hoping you captured enough. This isn't a minor inconvenience. It's a structural bottleneck that affects every aspect of network operations: - Knowledge capture happens in ad-hoc text files that nobody else can access - Multi-device operations require manual session switching and copy-paste - Outage investigation involves juggling 5-10 terminal windows with no correlation - Change verification is manual — you run the command and visually check the output - No structured link between what you see in the terminal and what's in your automation platform **The Credential Problem Hidden in Plain Sight** Here's a security issue most teams tolerate: engineers store device passwords locally. In PuTTY saved sessions, in SecureCRT connection files, in shell profiles, in sticky notes. Every laptop is a credential leak waiting to happen. When an engineer leaves, you're hoping they deleted their saved passwords. The terminal should never hold credentials. The platform should. The terminal should authenticate through the platform, and the credentials should never leave the server-side vault. This isn't theoretical — it's how every other cloud service works. SSH sessions are the exception, and only because the tooling has never provided a better alternative. **How NetStacks Addresses This** NetStacks approaches the terminal as an engineering platform, not a connection utility. The differences are practical, not cosmetic: Sessions are organized, not scattered. Folders, groups, and search across your entire device library. Turn on session logging and every log lands in a library you can filter by name and date. When you need to find what happened on dc1-spine-01 last Tuesday, it's one filter away — not a grep through three months of log files. Multi-send eliminates the session juggling. Enable it on a set of tabs and every keystroke goes to all of them; put up to four in a split view and the output comes back side by side. For a one-shot command across a selection of saved sessions, Broadcast Command collects every device's result in one place. No copy-paste, no session switching, no "let me check each one." The workspace is integrated. In the desktop app, file explorer, code editor with Python/YANG/XML support, integrated Git, and terminal sessions — all in one window. When you're debugging a Jinja2 template for config generation, you're not switching between a text editor, a file browser, and three SSH sessions. They're all there. With the NetStacks Controller, credentials never touch your laptop. The Controller holds them in its vault and proxies every SSH connection. No saved passwords, no credential files, no "I'll just store it in my config for now." This is the security model every platform should have. (On its own, the desktop app keeps credentials in an encrypted local vault.) AI assistance is built into the terminal, not bolted on as a separate tool. When CLI output is confusing, you select it and get an explanation in context. Not a chatbot in another tab — the assistant reads the output you've already gathered and tells you what matters about it. **The Real Impact** This isn't about making the terminal prettier. It's about closing the gap between the tools engineers use for investigation (terminals) and the tools teams use for automation (platforms). When these are disconnected, you get the pattern every team knows: build automation in the platform, debug in the terminal, update the platform based on what you found, and hope someone documents it. NetStacks puts the investigation and automation tools in the same workflow. Terminal sessions, file management, config editing, AI assistance, topology visualization — all in one application. The gap between "I found the problem" and "I fixed the automation" shrinks from hours to minutes. The question isn't whether your team needs a better terminal. It's how much time your team loses every week because the terminal experience hasn't evolved.

The SSH CA Pattern Every Network Team Needs But Nobody Implements

Tue, 12 May 2026 00:00:00 GMT

If your network team manages device access by distributing SSH keys, you're already in technical debt. Not "legacy system that works fine" debt. "Every departing engineer could have taken a copy of every device key they ever received" debt. SSH key management at scale breaks down into predictable stages that every team goes through: Stage 1: "Just add my public key to authorized_keys on the jumpbox." This works for three engineers across ten devices. Then you get to eleven. Stage 2: "We need a shared key management system." Someone sets up a script that pushes keys from a central list. It works until someone leaves and you need to rotate all their keys across every device. That's a weekend project. Stage 3: "We need certificates." Someone reads about SSH CAs and it makes perfect sense. Sign once, trust everywhere. Expire automatically. Principals and extensions for fine-grained control. It's the right solution. Nobody implements it because the infrastructure is complex and the migration from key-based access is painful. Stage 4: "We have SSH CAs now." This is what happens when the platform handles certificate issuance, signing, and distribution transparently. Engineers connect through the platform, get short-lived certificates, and never manage keys directly. This is where most teams want to be and very few actually are. **The SSH CA Model That Actually Works** The certificate-based approach should look like this from the engineer's perspective: they open their terminal, select a device, and connect. Behind the scenes, the platform requests a short-lived SSH certificate from an internal CA, the CA signs a certificate scoped to the engineer's identity and the target device, and the SSH connection uses that certificate. No key management, no authorized_keys files, no rotation projects. The certificate should be short-lived. Hours, not months. If an engineer's access is revoked, their existing certificates expire naturally — no forced key rotation across every device. The CA should enforce principal restrictions. An engineer authorized only for the spine layer shouldn't be able to certificate-scoped access to core routers. **What NetStacks Does Here** NetStacks implements the SSH CA pattern at the platform level. The Controller acts as the certificate authority: when a credential profile uses certificate auth, it mints a short-lived certificate at connect time, an hour by default, with the device login as the principal and the engineer's identity recorded in the certificate's key ID. No key material is stored. Engineers authenticate to the Controller, which issues the certificate and proxies the connection. This means: - No SSH keys on engineer laptops. Zero. The Controller handles authentication and certificate issuance. Every connection goes through the platform, under roles that control who can reach the device inventory. Every certificate the Controller mints is written to the audit log along with the device it was issued for, and with session recording turned on, terminal sessions can be saved to the Controller for replay. When an engineer leaves, you revoke their platform access. Their certificates expire. There are no device-side keys to rotate. **The Multi-Vendor Complication** Here's where the SSH CA gets complicated: every vendor implements SSH differently. IOS-XR has different SSH server behaviors than Juniper, which differs from Arista's EOS, which differs from Linux-based NOSes. Certificate validation, key exchange algorithms, and cipher preferences all vary. The SSH CA needs to understand these differences. A certificate that works on IOS-XR might fail on Juniper because of different key algorithm expectations. The CA needs to issue certificates that each platform's SSH server will accept. This is invisible to the engineer but critical for the platform. **The Team That Should Adopt This** Any team with more than 10 engineers and more than 50 devices should be on certificate-based SSH access. If you're still using key-based authentication, you're one departing engineer away from a key rotation project that nobody wants to do. The migration path: keep the SSH CA alongside existing key-based access during the transition. Engineers authenticate through the platform, the CA issues certificates, and the old keys stay on the devices so engineers can still connect the old way if a certificate is rejected. Once everyone is using the platform, disable key-based access entirely. Audit the transition. Confirm no device still has engineer keys in authorized_keys. This is the security model that scales. Not because it's complex, but because it eliminates the thing that makes SSH access unmanageable at scale: the exponential growth of key-device relationships that no one can track or audit.