Skip to main content

Troubleshooting

Common failure modes and their fixes. If your problem is not here, check the Infrahub documentation or open an issue.

Bootstrap timeout​

Symptom:

[ERROR] ConnectionRefusedError: [Errno 111] Connection refused

or the bootstrap script exits immediately with "Infrahub not reachable".

Cause: The bootstrap script runs too soon after docker compose up. Infrahub takes 20–40 seconds to initialise its database and API server.

Fix:

# Wait for Infrahub to be ready, then bootstrap manually:
uv run invoke start
sleep 40
uv run invoke bootstrap

Alternatively, run invoke init which includes a built-in wait, but if your machine is slow, increase the sleep in tasks.py:

# tasks.py — find the sleep call in init_demo and increase it
time.sleep(40)  # change to 60 or more on slow machines

Generator fails: "No free physical interface"​

Symptom:

RuntimeError: No free physical interface on pe-01

Cause: The L3VpnGenerator first looks for an InterfacePhysical on the target PE already described L3VPN <vpn name> — that is how a pre-wired PE-CE port is bound. When there is no such port it falls back to the lowest-numbered interface with status = free, and this error means there is none left: every interface on that PE is already active.

On the financial dataset each PE ships with two free customer-facing spares; the ports wired to the pre-provisioned CEs (Ethernet3/Ethernet4 on pe-01 and pe-08) are deliberately active so they can never be handed to another service.

Fix: Create a free interface on the PE via the Infrahub UI or GraphQL:

curl -s -X POST http://localhost:8000/graphql \
  -H "X-INFRAHUB-KEY: 06438eb2-8019-4776-878c-0941b1f1d1ec" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "mutation {
      InterfacePhysicalCreate(data: {
        name: {value: \"Ethernet10\"},
        status: {value: \"free\"},
        mtu: {value: 9000},
        device: {hfid: [\"pe-01\"]}
      }) { ok object { id } }
    }"
  }'

Repeat for whichever PE is out of free interfaces.

Configuration push fails: "not supported on this hardware platform"​

Symptom:

eAPI error: CLI command 11 of 57 'ip routing' failed: invalid command
last result: {'errors': ["Unavailable command (not supported on this hardware
platform) (at token 1: 'routing')"]}

Cause: The node answered eAPI before its agents finished registering their CLI commands, so the parser genuinely did not know the routing keyword yet. It is a timing problem, not a configuration problem — the same file pushes cleanly to the nodes that had longer to boot, which is why the failure tends to appear part-way through a run.

Fix: None needed. scripts/push_arista.py polls a read-only command served by the same agent that owns ip routing, and only pushes once that answers. If it times out instead, the node is genuinely stuck — check docker logs clab-mpls-backbone-1-<node>. Booting twelve cEOS nodes is memory-hungry; give Docker more RAM if nodes are being killed.

Pushed configuration does not match Infrahub​

Symptom: The pushed command counts or interface names do not match what Infrahub shows, or a device rejects interfaces the topology does not have.

Cause: lab/devices/*.cfg are fetched once by invoke lab.deploy and then reused by invoke lab.push-arista. After the dataset or a service changes, the files on disk are stale.

Fix: Re-run invoke lab.deploy. It re-fetches the topology artifact and every per-device configuration, then redeploys the lab so the running nodes and the configuration files agree.

LDP does not start, so inter-site VPN traffic fails​

Symptom: Every prefix is exchanged correctly — IS-IS is up, the iBGP mesh is up, PE-CE eBGP is up, and each PE holds the customer prefixes in the right VRF — but a host on one customer site cannot reach the same customer's other site. On any PE:

show mpls ldp neighbor
! Agent 'LdpAgent' is not running
! LDP is not enabled

Cause: Not a configuration defect. The rendered PE configuration contains the full mpls ldp block, including no shutdown and the per-interface statements, and it pushes without error — but cEOS never starts the LDP agent. It appears to be a limitation of the cEOS-lab image rather than something the configuration can influence.

Consequence: An L3VPN needs an MPLS label path between PEs to carry VPN traffic. Without LDP there is no transport, so VPNv4 routes cannot be used for forwarding even though BGP has exchanged them. The control plane is complete and demonstrable; the data plane between sites is not.

What still works: everything up to the label path — IS-IS adjacencies, the full iBGP mesh, PE-CE eBGP, VRF route import/export, and customer LAN reachability within a site (host to its own CE gateway over the tagged VLAN).

Fix: Unverified. This was diagnosed against cEOS 4.29.0.2F (which reports itself as 4.28.9M), whose LDP agent never starts. The lab now runs 4.36.0.1F, so the gap may already be closed — if you can confirm either way on a real deploy (which needs an x86_64 host), update this page. To test another build, point CEOS_IMAGE at it (CEOS_IMAGE=<image> uv run invoke lab.deploy). Segment routing instead of LDP would be the other option, and a larger change.

Containerlab deploy hangs at "Running postdeploy actions"​

Symptom: invoke lab.deploy creates every node and every link within a second or two, then stops making progress. The last lines repeat once per cEOS node and nothing follows them, for tens of minutes:

INFO Created link: pe-07:eth4 ▪-▪ pe-08:eth2
INFO Running postdeploy actions for Arista cEOS 'pe-07' node
INFO Running postdeploy actions for Arista cEOS 'pe-08' node

There is no error, no traceback and no exit — containerlab inspect shows the containers up, and docker logs on a node is empty or stops early.

Cause: the nodes booted as containers but EOS never came up inside them. containerlab's cEOS postdeploy step opens the EOS CLI (docker exec … Cli) and waits for a prompt before pushing the management and interface configuration. That wait is an unbounded retry loop — for { …; time.Sleep(2s); continue } in containerlab's utils/networkcli.go, with no deadline and no attempt cap — so an EOS that never finishes booting shows up as an indefinite stall rather than a failure.

invoke lab.deploy bounds it at 30 minutes and prints the per-node container state on timeout, which is what separates a node that died from one still booting. Adjust with LAB_DEPLOY_TIMEOUT_SECONDS.

Three things cause it:

  1. An ARM host. The most common one, and the one that looks least like itself. cEOS-lab is 32-bit x86 in every published build and Rosetta 2 does not translate 32-bit x86, so on Apple Silicon EOS reaches agentsToStart=[...], loops on ProcMgr.service, and exits 255 roughly 17 minutes later — while the cust-* netshoot hosts stay up, so docker ps still looks like a deployed lab. Confirm with:

    uname -m                                        # aarch64 / arm64 = unsupported
    docker ps -a --filter name=clab- --format '{{.Names}}  {{.Status}}'

    Exited (255) on the ceos nodes with the cust-* hosts up is the signature. invoke lab.deploy now refuses ARM hosts before deploying, so this only appears on an older checkout or with LAB_ALLOW_UNSUPPORTED_ARCH=true. There is no image that fixes it — see Architecture.

  2. cEOS-lab older than 4.32.0F on a cgroups v2 host. Earlier builds require cgroups v1 and silently fail to boot otherwise. cgroups v2 is the default on Ubuntu 21.04+, OrbStack, and most current distributions. Check the host:

    stat -fc %T /sys/fs/cgroup/     # cgroup2fs = v2, tmpfs = v1

    From 4.32.0F onward cEOS-lab detects the host's cgroup version itself. The images invoke lab.deploy selects are all newer than that, so this only bites a pinned older build — check CEOS_IMAGE.

  3. Not enough memory. cEOS wants ~2 GB per node, and the default financial dataset runs 12 of them. On an undersized host the nodes thrash or get OOM-killed mid-boot, which looks identical from the outside:

    free -h
    dmesg -T | grep -iE 'oom|killed process'
    docker stats --no-stream

    See Containerlab prerequisites for sizing. Use the isp dataset (2 lab nodes) if the host cannot take the financial one.

Confirming which: ask a node directly.

docker exec -it clab-mpls-backbone-1-pe-01 systemctl is-system-running

starting long after deploy means EOS is stuck mid-boot (any of the three); running means EOS is up and the problem is elsewhere. systemctl list-jobs on the node shows which unit is blocking.

Fix: deploy on a host that meets the prerequisites, with a cEOS image of 4.32.0F or newer matching the host architecture. Tear the partial lab down first — uv run invoke lab.destroy — so the retry starts clean.

Port conflict: Infrahub already running​

Symptom:

Error response from daemon: driver failed programming external connectivity ...
Bind for 0.0.0.0:8000 failed: port is already allocated

Cause: Another process (or a leftover container) is using port 8000, 4200 (Prefect), or 8501 (Streamlit).

Fix:

# Find what is using the port:
lsof -i :8000

# Stop leftover containers from this project:
docker compose -p sp-demo down

# If another unrelated project is running on the same port,
# override the port in .env:
echo "INFRAHUB_PORT=8001" >> .env
# Then update INFRAHUB_ADDRESS accordingly:
echo 'INFRAHUB_ADDRESS="http://localhost:8001"' >> .env
uv run invoke start

Containerlab: image pull failure​

SR Linux (ghcr.io/nokia/srlinux)​

Error response from daemon: Head "https://ghcr.io/...": unauthorized

SR Linux is on GitHub Container Registry and requires a GitHub token:

docker login ghcr.io -u <github-username> -p <personal-access-token>

Create a PAT at GitHub settings with read:packages scope.

Arista cEOS​

The OpsMill mirror (registry.opsmill.io/external/ceos-image) is public — no Arista account and no manual import.

Do not read the registry's architecture metadata as fact. docker import stamps the importing host's architecture into the image configuration, so a tag imported on an ARM machine claims arm64 while carrying the same 32-bit x86 EOS as every other tag. Docker then treats it as native, arranges no emulation, warns about nothing, and the node fails to boot. cEOS-lab has no ARM build; invoke lab.deploy refuses ARM hosts for that reason.

uname -m                                    # aarch64/arm64 cannot run cEOS-lab
echo "${CEOS_IMAGE:-<unset>}"               # what you asked for, if anything

To run your own Arista-supplied build, import it and point CEOS_IMAGE at it:

docker import cEOS-lab-4.36.0F.tar.xz ceos:local
CEOS_IMAGE=ceos:local uv run invoke lab.deploy

Streamlit catalog: "Infrahub not reachable"​

Symptom: The Streamlit app loads but shows "Cannot connect to Infrahub".

Cause: INFRAHUB_ADDRESS is not set or points to the wrong host/port.

Fix:

# Verify the env var is set:
grep INFRAHUB_ADDRESS .env

# If running Streamlit outside Docker Compose, ensure the address
# points to the correct host:
echo 'INFRAHUB_ADDRESS="http://localhost:8000"' >> .env
uv run streamlit run service_catalog/app.py

If you are running the Streamlit app inside Docker Compose, use the internal service name:

INFRAHUB_ADDRESS=http://infrahub-server:8000

Schema load order error​

Symptom:

SchemaNotFound: Node 'RoutingProtocol' not found

Cause: The SP schemas (schemas/sp/) reference base and extension nodes that were not loaded first.

Fix: Always load schemas in order:

infrahubctl schema load schemas/base/
infrahubctl schema load schemas/extensions/
infrahubctl schema load schemas/sp/

invoke bootstrap does this automatically. If you loaded schemas manually in the wrong order, run invoke destroy && invoke init to start clean.

Proposed Change: check fails after generator runs​

Symptom: The pe_interface_alloc check fails even though the generator ran and set the interface.

Cause: The check reads the current branch state. If the generator ran on a previous branch and the change was already merged, the interface status may already be cust on main, making it appear unavailable for a new site.

Fix: Use a different (free) interface for the new site, or set an existing cust-status interface back to free if it was decommissioned.

Proposed Change: schema integrity failure blocks the merge​

Symptom: Every check the demo owns passes, every artifact validator passes, and the diff is correct — but the proposed change carries a failed Schema Integrity validator and merging is refused:

Unable to merge proposed change containing failing checks

Its checks report conflicts on schema paths nothing in the change touched, most often:

schema/IpamIPAddress/uniqueness_constraints
schema/RoutingBGPSession/local_ip/cardinality
schema/RoutingBGPSession/remote_ip/min_count

Cause: Not a data or schema problem in this repository. Compare the schema on the two branches and they are identical:

curl -s -H "X-INFRAHUB-KEY: $INFRAHUB_API_TOKEN" \
  "$INFRAHUB_ADDRESS/api/schema?branch=main" > /tmp/main.json
curl -s -H "X-INFRAHUB-KEY: $INFRAHUB_API_TOKEN" \
  "$INFRAHUB_ADDRESS/api/schema?branch=<your-branch>" > /tmp/branch.json
diff <(jq -S '.nodes[] | select(.name=="BGPSession")' /tmp/main.json) \
     <(jq -S '.nodes[] | select(.name=="BGPSession")' /tmp/branch.json)

The conflicts also carry "branch": "placeholder" and "id": "unknown" rather than real references. Seen on Infrahub 1.10.6.

Behaviour: intermittent, and sticky to the branch once it happens. Closing the proposed change and opening a new one against the same branch reproduces it exactly; a service created on a fresh branch is unaffected. Most branches never see it — an empty branch does not raise the validator at all.

Fix: Recreate the service on a new branch. The catalog does that for you: submit the wizard again and delete the stuck branch afterwards. There is no way to clear the failed validator on the branch that has it, and no configuration in this repository influences whether it appears.

yamllint: line too long​

Symptom:

[error] line too long (105 > 100 characters)

Cause: Bootstrap YAML files sometimes have long inline object specs.

Fix: Break the long line or raise the limit in .yamllint.yml:

rules:
  line-length:
    max: 120
    level: warning   # demote from error to warning if needed

uv sync fails​

Symptom: uv sync fails with a resolver error or requires-python mismatch.

Fix:

# Verify your Python version:
python --version   # must be 3.10, 3.11, or 3.12

# If using pyenv:
pyenv install 3.12
pyenv local 3.12
uv sync