OS Collection

Parascope's OS collector bridges the gap between what your infrastructure platforms know (VMs, containers, bare metal) and what's actually running inside those machines. It connects via SSH, runs a self-contained collection script, and reports back operating system configuration, software inventory, security posture, and runtime metrics, all without installing agents on your targets.

Why OS Collection Matters

Infrastructure platforms like Proxmox, OpenStack, and Kubernetes track resources from the outside (CPU allocation, memory limits, network attachments). But they can't tell you:

• What Linux distribution and kernel version is actually running
• Which packages are installed and whether security updates are pending
• What services are listening on the network
• Whether SSH is configured securely
• Which TLS certificates are deployed and when they expire
• What containers are running inside the VM
• What physical network neighbors are connected via LLDP

OS collection answers these questions across your entire fleet, creating a rich layer of CI data that connects to infrastructure CIs you already track.

Architecture

Key design principles:

• Agentless: No software installed on targets. A self-contained bash script is streamed via SSH and executed in memory
• Minimal footprint: Only standard Linux tools required (bash, awk, grep, sed, df, uname, ss, ip). Optional tools like lldpctl and openssl enable additional sections
• Safe execution: Script writes nothing to disk on the target (except /tmp), and missing tools or permissions result in empty fields rather than failures
• Credential security: SSH keys and passwords stored securely in Parascope, never in config files or logs

CI Types Created

The OS collector creates four distinct CI types, each serving a specific purpose in your infrastructure model.

os.linux: Operating System Instance

The primary CI representing a Linux installation on a host. Contains comprehensive OS-level data organized into change-tracked configuration and point-in-time metrics.

Configuration (tracked in change history):

Metrics (point-in-time, not tracked):

Relationships: Each os.linux CI has a runs_on relationship to its parent infrastructure CI (e.g., proxmox.vm, openstack.instance).

os.software: Promoted Software

Not every installed package becomes a CI. The promotion engine identifies operationally significant software (services with listening ports, active daemons, and known infrastructure components) and promotes them to dedicated CIs.

Software CIs are deduplicated across the fleet: one CI per unique (software name, version, package type) combination, with per-host instance records tracking where it runs.

CI-level fields (shared identity):

Per-host instance fields:

This two-tier model means you can answer both "What version of nginx exists in our fleet?" (CI level) and "Which specific hosts run nginx, and is the service healthy?" (instance level).

os.certificate: X.509 Certificates

TLS certificates discovered in standard paths (/etc/ssl, /etc/pki, etc.) are parsed and deduplicated by SHA-256 fingerprint. One CI per unique certificate, with per-host instance records tracking file locations.

CI-level fields:

Per-host instance fields:

os.container: Container Instances

Docker and Podman containers discovered on hosts are tracked as individual CIs with a runs_on relationship to their host os.linux CI.

Key fields:

Ceph container stable naming: Cephadm containers include the cluster FSID in their name (e.g., ceph-28ea88f2-1234-5678-abcd-ef0123456789-osd-2). The collector strips this prefix to produce a stable name (osd-2@hostname) that persists across cluster rebuilds.

What Gets Collected

The collection script runs 13 independent sections, each gathering a specific category of data. Sections can be individually enabled or disabled per ruleset.

How Collection Works

1. Target discovery: The collector queries the Parascope API for CIs matching the ruleset filters (e.g., all running Proxmox VMs tagged "linux"). It extracts IP addresses from each CI's network data
2. Credential resolution: SSH credentials are resolved from Parascope's credential store, with per-target overrides taking priority over ruleset-level defaults
3. SSH connection: The collector connects to each target via direct SSH or through a jump host, with configurable timeouts and host key verification (see below)
4. Script execution: The collection script is streamed to the target via stdin (bash -s) and executed. No files are written to the target
5. Parsing: The script's JSON output is parsed into Parascope's structured format, separating config (change-tracked) from metrics (point-in-time)
6. Software promotion: The promotion engine evaluates packages against behavioral signals and known patterns
7. Publishing: Results are published for processing as separate messages per CI type (os.linux, os.software, os.certificate, os.container)
8. Processing: CIs are created or updated, relationships are established, and enrichment data is merged into parent CIs

Collection runs with bounded concurrency (default: 10 parallel SSH connections) and per-target error isolation. A failure on one host doesn't affect others.

SSH Host Key Verification

The OS collector pins SSH host keys so credentials are only ever sent to a host whose key matches the one Parascope already holds. Verification happens during the SSH handshake, before authentication, so a mismatched key aborts the connection with no credential sent.

How It Works

1. Pinning: Every os.linux CI stores the host's public keys. On each connection, the stored public key is passed to the SSH client as the expected key. A mismatch aborts the handshake
2. First contact: A host with no stored key has nothing to pin against. What happens then is the whole point of the mode setting below: the default reject refuses the connection, and tofu accepts the presented key once, stores it, and pins it from the next cycle onward
3. Cross-validation: The collection script also reads host keys from /etc/ssh/ssh_host_*_key.pub on the target. The collector compares the handshake key against the filesystem key. A mismatch is a strong indicator of a MITM attack (the handshake key differs from what the host itself reports)

Verification Modes

The verification mode is set per source via the host_key_verification key in the source's configuration:

reject is the strictest setting, and it cannot bootstrap on its own: pre-seed the keys first (see below). tofu is the practical default for a new fleet. It is exposed to a man-in-the-middle for exactly one connection per host, on the cycle where the host is first seen, and is pinned as strictly as reject from then on. Each such acceptance is logged and counted (os_ssh_host_key_tofu_first_contact_total).

warn and disabled send credentials to an unverified host on every cycle, not just the first. Use them for diagnosis, not as a steady state.

Where These Settings Live

Set them either on the OS Collection ruleset (Settings, OS Collection, the SSH Host Keys section of the ruleset form) or directly on the source's config through the sources API. A ruleset carries them onto the source it drives, so the ruleset form is the place to set them for a ruleset-driven collection.

The gateway fields appear on the form only when the ruleset's reachability strategy is Jump Host, and the gateway keys are entered as JSON keyed by SSH algorithm, matching the stored shape below. A ruleset that leaves the mode unset runs in reject, so it needs its target keys pre-seeded before first collection.

Pre-Seeding Keys

Point known_hosts_path at an OpenSSH known_hosts file to pin hosts before their first collection. The collector reads that file; it never writes to it. Populate it yourself:

ssh-keyscan -H 10.0.1.50 10.0.1.51 >> /etc/parascope/known_hosts

Mount the file into the collector and set known_hosts_path on the source. It applies in both reject and tofu mode. Note that once a path is set, it is the pin list for every host on that source: a host missing from the file is refused, in tofu mode too. If the configured file does not exist, connections fail rather than falling back to unverified.

For sources that reach targets through a jump host, the bastion's own key must be pre-seeded via gateway_host_keys or gateway_known_hosts_path. tofu does not cover the bastion hop: nothing stores a gateway key, so accepting one would mean accepting it unverified on every cycle rather than once. The bastion is a single long-lived host, so one ssh-keyscan settles it; until then a jump-host source is refused at the gateway, before any session opens.

This applies to a ruleset whose reachability strategy is Jump Host: the bastion hop is taken (see Reachability Strategies), and the mode set on the ruleset governs it with no separate policy for the gateway. A ruleset in reject or tofu with no pinned gateway key is refused at the bastion until the key is seeded; warn and disabled connect to it unverified, as they do for targets.

Stored Data

Host key fingerprints are stored on each os.linux CI in the ssh_host_key_fingerprints field:

{
  "ssh-ed25519": {
    "fingerprint": "SHA256:rNo3mjXJZLLC6R0SNbhvBjTMWCuJhK3cFlZps8HI2rI",
    "public_key": "ssh-ed25519 AAAA..."
  },
  "ssh-rsa": {
    "fingerprint": "SHA256:WRRx5bPX71T3pa7r0JIaJmgLl90JoVMOj9w9d75yseI",
    "public_key": "ssh-rsa AAAA..."
  },
  "_connection_ip": "10.0.1.50"
}

Keys from both the SSH handshake and the target's filesystem are merged. The handshake key takes precedence for its algorithm. The _connection_ip records which IP address was used for the SSH connection.

If Parascope cannot read the stored keys for a cycle (an API error mid-collection, for example), a tofu source falls back to reject for that cycle rather than treating every target as a new host. Collection resumes on the next cycle.

Key Rotation

When a host's SSH keys are legitimately rotated (after a reinstall, for example), the stored key no longer matches and the connection is rejected. Re-pin the host:

The stored key wins in both modes: while the os.linux CI still holds the old fingerprints they are what gets pinned, and tofu does not treat a host with stored keys as first contact. So clear the stored fingerprints first, then choose how the new key gets trusted:

1. Clear the ssh_host_key_fingerprints field on the host's os.linux CI (or delete and recreate the CI). On its own this is not enough, and in reject mode it leaves the host uncollectable rather than re-pinned
2. Then either seed the new key into known_hosts_path with ssh-keyscan, or set the source to tofu so the next cycle captures and pins it, or write the new key onto the CI directly

Confirm the new fingerprint out of band before you trust it. A rejected key is what a man-in-the-middle looks like, and it is also what a reinstall looks like.

Software Promotion Engine

With hundreds of packages installed on a typical Linux host, promoting all of them to CIs would create noise. The promotion engine identifies the packages that actually matter: the ones running services, listening on the network, or matching known infrastructure patterns.

Promotion Signals

The engine uses fuzzy name matching to connect packages to their processes. For example, postgresql-15 is matched to listener process postgres by stripping version suffixes and checking known aliases.

Known Infrastructure Patterns

The promotion engine recognizes over 100 infrastructure software patterns across these categories:

Promotion Overrides

Each ruleset can configure:

• Force promote: Package names to always promote, even without behavioral signals. Use for custom or internal software
• Suppress: Package names to never promote, even if they match signals. Use to filter noisy or uninteresting software

Real-World Use Cases

Security Incident Response: "Which hosts have vulnerable package X?"

When a CVE is announced for a critical package, you need to know your exposure immediately.

Scenario: A critical vulnerability is disclosed in OpenSSL 3.0.x (CVE-2024-XXXX). Your security team needs to identify all affected hosts within minutes.

With OS collection:

1. Search for os.software CIs with software_name = openssl to see every version deployed across your fleet
2. Click into each software CI to see the per-host instance list, which names the specific machines running the vulnerable version
3. Check available_version on each instance to see if patches are already available in your package repositories
4. Use the runs_on relationship from os.linux to trace back to the hosting VM and the physical infrastructure beneath it

Without OS collection: You'd need to SSH into each machine manually, or maintain a separate inventory tool, or hope your vulnerability scanner has recent data.

Patch Compliance: "How many hosts have pending security updates?"

Scenario: Your compliance policy requires security patches within 30 days. You need a dashboard-ready view of patch status.

With OS collection:

• Every os.linux CI tracks security_update_count, the number of pending security updates
• The security_updates field lists each pending update with the available version
• Filter the CI list for os.linux CIs where security_update_count > 0 to see non-compliant hosts
• Track changes over time. Parascope's change history records when security updates appear and when they get applied

Certificate Expiration Monitoring: "Which certificates expire soon?"

Scenario: An expired TLS certificate causes a production outage. You need visibility into certificate lifetimes across your fleet.

With OS collection:

• Every TLS certificate found on disk is tracked as an os.certificate CI with not_after (expiration date)
• Certificates are deduplicated: a wildcard cert used on 10 hosts appears as one CI, with an instance record for every file path on every host
• Sort certificates by expiration date to see what's expiring next
• The instance records show exactly which hosts and file paths would be affected

Fleet Software Inventory: "What's running across our infrastructure?"

Scenario: During an architecture review, you need to understand the software landscape across 50+ Linux hosts.

With OS collection:

• Browse os.software CIs to see every promoted software component across the fleet
• Each software CI shows how many hosts run it (via instance count)
• Filter by promotion reason to focus on network services (listening_port), active daemons (active_daemon), or known infrastructure (known_pattern)
• Identify version sprawl: the same software at different versions across hosts
• Discover unexpected services. The unpackaged_listener signal catches processes listening on ports without corresponding packages

Infrastructure Drift Detection: "Is this VM configured as expected?"

Scenario: A VM was rebuilt from a template, but something isn't working. You need to compare its current state against known-good configuration.

With OS collection:

• Compare the current os.linux CI's configuration with its change history to see what changed
• Check kernel version, installed packages, network configuration, and service states
• The IP mismatch detection in OS enrichment flags cases where the OS-reported IP addresses differ from what the infrastructure platform expects
• Compare two hosts by viewing their os.linux CIs side by side. Same distribution? Same kernel? Same key packages?

Container Visibility: "What containers run on this host?"

Scenario: A host managed by cephadm is having performance issues. You need to see what containers are running on it.

With OS collection:

• Each os.container CI shows the container's image, state, port bindings, and resource configuration
• The runs_on relationship connects containers to their host os.linux CI
• Ceph containers use stable naming (osd-2@hostname instead of ceph-28ea88f2-...-osd-2), so CI identity persists across cluster rebuilds
• Distinguish Docker vs Podman containers via the runtime field

Network Service Mapping: "What's listening on the network?"

Scenario: You're auditing network exposure across your fleet and need to know every listening port and what process owns it.

With OS collection:

• The listeners section captures every TCP and UDP listening socket with the owning process name
• Promoted software CIs include their listening ports in the instance data
• unpackaged_listener promotions flag processes with network sockets but no corresponding package (potentially unauthorized services)
• Combine with network data (interfaces, routes, DNS) to build a complete picture of each host's network posture

Physical Connectivity: "What switch port is this host connected to?"

Scenario: A host is experiencing network issues and you need to identify the physical switch and port it's connected to for troubleshooting.

With OS collection:

• The lldp section collects LLDP neighbor data from hosts running lldpd, showing the directly connected network switch and port
• Each LLDP neighbor record includes the remote system name, chassis ID, port ID, and management IP
• Cross-reference with SNMP LLDP data collected from switches to verify both sides of the physical link agree
• LLDP neighbor changes are tracked in change history, and a neighbor change means a physical cabling change
• Field names are aligned with the SNMP LLDP collector format, enabling future cross-source correlation

Note: LLDP data requires lldpd to be installed on the target host. Hosts without lldpd will simply report an empty neighbors list.

Configuration

Collection Rulesets

Rulesets define what to collect from, how to reach it, and what to collect. You can create multiple rulesets for different parts of your infrastructure (e.g., production vs staging, different network zones).

Key ruleset settings:

Target Discovery

Targets are discovered dynamically by querying the Parascope API for CIs matching the ruleset's filter criteria. The collector extracts IP addresses from each CI's network data.

Example: Collect from all running Proxmox VMs

{
  "targets": {
    "mode": "whitelist",
    "ci_types": ["proxmox.vm"],
    "filters": [
      {"field": "config.status", "op": "eq", "value": "running"}
    ]
  }
}

Example: Collect from OpenStack instances in a specific project

{
  "targets": {
    "mode": "whitelist",
    "ci_types": ["openstack.instance"],
    "filters": [
      {"field": "config.status", "op": "eq", "value": "ACTIVE"},
      {"field": "scope_label", "op": "eq", "value": "production-cloud"}
    ]
  }
}

Credential Management

SSH credentials are stored securely in Parascope's credential store and referenced by name in rulesets.

Supported authentication methods:

• SSH private key (RSA, Ed25519): recommended
• SSH password: for legacy systems

Resolution priority per target:

1. Target-specific credential (if configured)
2. Ruleset-level default credential
3. Error (no credentials available)

Reachability Strategies

Jump host example:

{
  "reachability": {
    "strategy": "jump_host",
    "gateway": "10.20.0.1",
    "gateway_port": 22,
    "gateway_credential": "Bastion SSH Key"
  }
}

gateway_port defaults to the collector's SSH port and gateway_credential to the ruleset's own credential, so a bastion on port 22 that shares the fleet key needs only gateway. Naming a gateway_credential the collector cannot resolve (a typo, or one since deleted) fails the source with a credential error instead of quietly presenting the ruleset's own credential to the bastion. A jump_host strategy with no gateway set routes nothing through a bastion; the collector connects directly, and the ruleset is logged as incomplete.

Targets are reached by tunnelling through the gateway, so the bastion is a second SSH hop with its own host key. Verification there is not optional: pre-seed it with gateway_host_keys or gateway_known_hosts_path (see SSH Host Key Verification) before switching a ruleset to Jump Host, or the first collection is refused at the gateway before any session opens. This holds for every ruleset marked Jump Host, including ones that predate the gateway host-key settings: seed the gateway key, or the next collection is refused at the bastion.

On-Demand Collection

In addition to scheduled collection, you can trigger immediate collection for a single target from the CI detail page. This is useful for:

• Verifying a configuration change was applied
• Getting fresh data before an incident investigation
• Testing connectivity to a new target

OS Enrichment

When the OS collector gathers data from a host, it also enriches the parent infrastructure CI (the VM or bare metal node) with a summary of what's running inside. This enrichment appears in the parent CI's detail view.

Enrichment data includes:

This gives you at-a-glance OS information directly on VM and node detail pages, without navigating to the os.linux CI.

Monitoring

The OS collector exposes Prometheus metrics for operational visibility:

Circuit Breaker

Each ruleset has an independent circuit breaker that prevents unhealthy rulesets from blocking healthy ones:

Related Documentation

• Architecture: System design overview
• Correlation Engine: Cross-system relationship discovery
• CI Types: All CI types across sources