Metadata-Version: 2.4
Name: serialcables-hcc
Version: 1.0.0rc1
Summary: UI / REST API / CLI command center for the Serial Cables PCIe Gen6 HYDRA JBOF enclosure
Author: Serial Cables Engineering
License-Expression: LicenseRef-SerialCables-HCC
Classifier: Private :: Do Not Upload
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn[standard]>=0.29
Requires-Dist: nicegui>=2.0
Requires-Dist: typer>=0.12
Requires-Dist: rich>=13.0
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.6
Requires-Dist: pyserial>=3.5
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: cryptography>=42
Provides-Extra: quarch
Requires-Dist: quarchpy>=2.2; extra == "quarch"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff==0.15.18; extra == "dev"
Provides-Extra: e2e
Requires-Dist: playwright>=1.40; extra == "e2e"
Dynamic: license-file

# Hydra Command Center

Interactive **UI / REST API / CLI** platform for managing and validating the
Serial Cables **PCIe Gen6 HYDRA 8-Bay NVMe/CXL JBOF** — one process serving all
three surfaces over the enclosure's single MCU serial link (plus optional
Quarch USB modules). Out-of-band device management, deep protocol access
(NVMe-MI, CXL CCI, MCTP over SMBus **and** I3C), and integrated power /
hot-plug instrumentation, at full parity across UI, API, and CLI.

- **Web UI** — Serial Cables-branded engineering dashboard (NiceGUI +
  Tailwind), light/dark themes, grouped navigation, a global Active-Bay
  context, per-DUT Bay Detail views, a live Quarch scope, an Events audit
  page, and a Run Manifest page. Signature clickable enclosure front panel
  with thermal cues.
- **REST API** — FastAPI with Swagger docs at `/docs`; everything the UI does.
- **CLI** — `hydractl` (Typer + Rich), full parity, `--json` for scripting/CI.
- **Simulator** — `--demo` runs the entire stack — every page and endpoint —
  with no hardware.

## Capabilities

**Slots & environment** — presence, payload power, dual-port, PWRDIS and
PERST#/SMBus reset, locate LEDs, per-bay temps, fans, rails, and buzzer
(`hydractl slot`, `hydractl env`).

**NVMe-MI (out-of-band)** — health and full SMART logs with threshold
auditing, error-log (LID 01h) triage, Device Self-Test (short/extended),
controller/port info, PCIe link verification (gen/width, downtrain
detection), and a capability probe of supported commands (`hydractl mi`).

**CXL (Technology Preview)** — Type-3 CCI over MCTP: device-class autodetect,
Identify, Get Health Info, FW Info, and event-log read/clear, proven on one
CXL memory module class (`hydractl cxl`).

**Diagnostics & device library** — bus/interposer probes and drive-type
detection, plus a deduplicated index of every drive HCC has ever identified,
keyed by serial.

**Validation gates** — server-side acceptance checks against a bay with
CI-friendly output (`hydractl check`), so a bring-up is a pass/fail signal,
not a spreadsheet.

**I3C (Technology Preview)** — bay state, discovery with decoded PID, BCR and
DCR, and a guided pass/fail test on the **I3C (Preview)** page
(`hydractl i3c state|test|daa`). Proven on one drive, on an SC paddle and behind
a Quarch breaker. Reading data from a target is not offered: it doesn't work on
current hardware.

**Recover** — one action for a bay whose management stopped answering: it
releases a held bus, resets and re-checks, without touching drive power or PCIe
(`hydractl slot recover <bay>`, or **Recover** on the Slots page).

**Protocol access** — MCTP transport underpins NVMe-MI and CXL CCI; direct
**I3C** private write / write-read / MIPI CCC (`hydractl i3c`); **VDM**
send/receive of MCTP Vendor Defined Messages (0x7E PCI / 0x7F IANA) with
vendor-ID presets (`hydractl vdm`); and raw NVMe-MI / MCU console passthrough
for anything below the typed surface.

**Firmware updates** — out-of-band slot-device firmware over MCTP:
validate/stage (SHA-256 gate) → chunked download with per-chunk verify and
abort → explicitly confirmed commit (`hydractl fw`). Download and commit have
not yet been run on live hardware: use an expendable drive.

**Quarch instrumentation (optional)** — native integration with Quarch
modules (the install kits include `quarchpy`) across the four sanctioned
capabilities — **Power Analysis, Sideband Analysis, Hot Swap, and Signal
Breaking** — plus:
- **High-rate QIS stream recording** — non-blocking power + sideband capture
  to CSV (1 ms samples on the breaker module), via Quarch's headless QIS
  service.
- **Live scope** — every rail and digital pin over time with labelled event
  markers (plug / pull / glitch) on one trace.
- **Hot-Plug Lab** — orchestrated hot-swap timing and per-signal glitch/break
  (length + PRBS) for connector-emulation testing.

**Run artifacts** — a timestamped command audit log (`hydractl events`),
persistent telemetry (NDJSON + CSV export), and one-shot inventory manifests
(`hydractl inventory`), all under `HCC_DATA_DIR` (default `./hcc_data`).

## Host agents (in-band, optional)

A small stdlib-only companion (`hcc-agent.pyz`, Python 3.8+) for the Linux or
Windows host that owns the PCIe links. It dials out to HCC and joins each
bay's out-of-band identity to what the host sees, keyed by drive serial
(`hydractl host`).

- **Inventory** — device, PCI address, link speed/width, model, firmware.
- **In-band jobs** — typed and allow-listed: Identify, SMART, error and
  firmware logs, get-feature, telemetry and kernel-log captures. Output lands
  as artifacts, with per-agent quotas you can see and prune.
- **Agreement gates** — `host-identity`, `host-link` and `host-smart`
  cross-check out-of-band against in-band inside `hydractl check`.
- **fio** — canned profiles stream IOPS, bandwidth and latency at 1 Hz onto the
  Workloads Monitor and the Quarch scope. Write profiles need a confirm and
  pass an on-host namespace-in-use check.
- **Watches** — opt-in, stopped by hand: error counters, a filtered kernel log,
  block I/O rates and SMART health. Only transitions reach the events timeline.
- **Hot-plug enumeration timing** — joins an enclosure plug to the host's
  link-up and device appearance.
- **Write recipes** — confirm-gated, only on hosts whose admin opted in. They
  refuse busy namespaces and record link, error counters and recovery timing
  before and after.

Agent tokens reach the agent's own routes only, and a drive seen from two hosts
needs an explicit host choice.

| | Linux | Windows (inbox `stornvme`) |
|---|---|---|
| Inventory, NVMe reads, gates, fio | ✅ | ✅ |
| Kernel log | journal | Event Log |
| Error counters | AER | WHEA events (labelled as such) |
| Rescan | ✅ | ✅ |
| Self-test | ✅ start and abort | Start only, once per agent start |
| Telemetry regenerate, per-sensor thresholds | ✅ | — |
| Link retrain, bus reset, controller reset | ✅ | — |
| Admin-queue trace, config-space dump, subsystems, CXL | ✅ | — |

HCC refuses what a host can't do, by capability. See
[`agent/README.md`](agent/README.md) and
[manual chapter 17](docs/manual/17-host-agent.md).

## Workloads

Pick one of sixteen presets (PCIe Gen6, NVMe 2.x, CXL, Quarch), edit it as a
schema-validated definition, save it as a custom template, then arm, start,
watch and stop it. `[BENCH]` marks presets not yet proven on hardware. The
Monitor offers every channel available; selecting one only shows or hides it, and
hovering one highlights it on the chart. **Statistics** give each channel's coverage,
min, mean, max, spread and percentiles, for the whole run or one phase, plus IOPS/W
and J/GB per phase (`hydractl workload stats <run> --phases`).

- **Qualification Report** — one file to file or share for a finished run: summary and
  findings, device and setup, Power vs Performance by phase, health before and after,
  evidence and a sign-off block (`hydractl workload qualify <run>`).
- **Live SMART** — media errors, error-log entries, unsafe shutdowns, thermal throttle
  events and critical warning as tiles, and drive temperature, each sensor, spare and
  life used as channels, on Linux and Windows.

- **Honest by default** — errors are counted and shown, never auto-failed, and
  STOP TEST is never gated. A report says when a snapshot went bad in a run that passed.
- **One parsed log per run** — kernel lines, counter moves, gate checks, guards
  and markers, behind one cursor endpoint and one WebSocket.
- **Exports** — JSON manifest, aligned CSV, raw NDJSON, or a self-contained
  HTML report.

```bash
hydractl workload run <template> --slot N --arm --wait
hydractl workload log <run> --follow
hydractl workload export <run> --format html -o run.html
hydractl workload qualify <run> --document QR-0042 -o report.html
```

**Channels** are the substrate: every live source (fio, watches, MCU telemetry,
Quarch rails, HCC actions) is published on one time base (`GET
/api/v1/channels`, `hydractl channels follow 'aer.*'`), with a file-backed tier
keeping 24 h of 1 s and 10 s rollups (`hydractl channels rollup bay.temp_c
--hours 24`).

See [manual chapter 18](docs/manual/18-workloads.md) and
`examples/workload_run`.

## Feature maturity

| Feature | Status |
|---|---|
| NVMe-MI management, enclosure control, host agents, Workloads, Qualification Report | Supported |
| CXL out-of-band management | Technology Preview |
| Slot-device firmware update | Ships with a caution: download and commit not yet live-run |
| I3C (discovery and identity) | Technology Preview |
| VDM Template Studio | Demonstration |
| SAS page | Experimental |

## Install

HCC requires enclosure MCU firmware **0.9.0 or newer** (units shipped with
0.0.8 must update first); the image and its SHA-256 are on the HYDRA product
page.

**Linux lab machine (the normal deployment)** — unzip the install kit from
the HYDRA product page (`serialcables-hcc-<version>-linux.zip`) and run one
command:

```bash
sudo sh install-hcc.sh                # add --with-agent if this host also owns the PCIe links
```

That creates a private Python environment in `/opt/serialcables/hcc`, then
`hcc-setup` (shipped in the wheel) sets up the `hcc` systemd service, the
token and self-signed TLS certificate in `/etc/hcc`, the Quarch USB rule, the
firewall port, and a `hydractl` that already points at the server. It ends by
printing the browser link; `sudo hcc-setup status` shows it again. Re-run the
same command with a newer kit to upgrade (settings and data are kept);
`--uninstall [--purge]` removes it. Details: [manual §0](docs/manual/00-introduction.md#installation).

**Windows 10/11** — unzip `serialcables-hcc-<version>-windows.zip` and
double-click `install-hcc.cmd` (one UAC prompt). It installs Python 3.12 if
needed, puts HCC in `C:\Program Files\SerialCables\HCC`, and the same
`hcc-setup` registers a boot-time scheduled task (SYSTEM) with settings,
token, certificate, data and log in `C:\ProgramData\SerialCables\HCC`, a
firewall rule, and `hydractl` on the PATH. A double-click asks whether the drives are on
this PC, and if so installs and enrolls the host agent; from a prompt, add
`--with-agent`. It also offers to fix a network marked Public, which would block other
machines. An upgrade keeps an installed agent in step.

**A quick try, any OS** — install the wheel and run the server in the
foreground:

```bash
pip install ./serialcables_hcc-<version>-py3-none-any.whl "quarchpy>=2.2"
hydractl serve --demo                     # simulator
hydractl serve --serial-port COM3         # real hardware
# UI: http://127.0.0.1:8600   API docs: http://127.0.0.1:8600/docs
```

**Developers** (source checkout):

```bash
pip install -e .[dev,quarch]
python agent/build_pyz.py --bundle        # editable installs need the agent bundle staged
hydractl serve --demo
```

```bash
hydractl status
hydractl slot list
hydractl mi health 3 --json
hydractl mi self-test 3 --start short
hydractl cxl health 8            # CXL Type-3 device in a bay
hydractl check 3                 # run the validation gates against a bay
hydractl quarch scan             # discover Quarch modules
hydractl host bay 3              # in-band view of bay 3 from the enrolled host
```

## Examples

Runnable, **stdlib-only** lab-automation scripts live in
[`examples/`](examples/) — smoke test, slot inventory, link check, health
snapshot/trend, validation gates, CXL device check, self-test campaign,
telemetry capture, thermal soak, hot-plug endurance, firmware update, the
host in-band check, and the Quarch bring-up / power-margin / stream-capture /
I3C-probe scripts. Each is
one folder with a README, runs against `--demo`, and uses CI-style exit codes.

## Architecture

```
src/
  hydra_command_center/
    _vendor/             bundled forks — never top-level (PyPI has the same names)
      serialcables_hydra/   serial transport + MCU command set
      serialcables_sphinx/  MCTP framing + NVMe-MI encode/decode
    config.py            settings (env vars HCC_*, CLI flags)
    core/                exceptions
    simulator/           full MCU simulator (demo mode)
    hostsetup/           hcc-setup: service install for Linux and Windows
    services/            shared logic — connection, slots, environment,
                         NVMe-MI, diagnostics, CXL, device library, Quarch
                         (+ QIS recorder), VDM, firmware, validation,
                         telemetry, events, inventory, host agents,
                         Workloads, channel pipeline
    workloads/presets/   the shipped Workload presets
    api/                 FastAPI routers + WebSocket telemetry
    ui/                  NiceGUI pages (one per manual section)
    cli/                 hydractl (an API client)
    server.py            single-process entrypoint
docs/manual/             User's Manual (18 sections; UI + API + CLI each)
examples/                borrowable lab-automation scripts
agent/                   hcc-agent: stdlib-only host companion, Linux and Windows
installer/               install kits (install-hcc.sh, install-hcc.cmd, build_kit.py)
tests/                   pytest: simulator, services, API, CLI, parsers, agent
```

The server owns the one serial connection and serializes MCU access, so the
UI, API, and CLI operate concurrently without port contention. The CLI is an
API client (`--url` / `HCC_URL` to target a remote server). Quarch modules
attach over their **own** USB links, independent of the MCU port. QuarchPy is
**not** vendored — it is an optional PyPI dependency used only for the Quarch
capabilities.

## Development

```bash
pip install -e .[dev]
python -m pytest tests -q
```

Building the release: `python installer/build_kit.py` builds the wheel and
both install kits — `dist/serialcables-hcc-<version>-linux.zip` (wheel +
`install-hcc.sh` + `INSTALL.txt` + LICENSE) and `…-windows.zip` (wheel +
`install-hcc.cmd`/`.ps1` + `INSTALL.txt` + LICENSE). It deletes `build/`
first — setuptools reuses `build/lib/`, so modules moved or deleted since the
last build would otherwise ship again. Post both kits on the product page.

## Documentation

The User's Manual lives in [`docs/manual/`](docs/manual/00-introduction.md) —
one markdown file per UI section (18 sections, through Workloads), each ending with
API and CLI reference addendums.

## License

Proprietary — free of charge, not open source. Copyright (c) 2025-2026 Serial
Cables, LLC; see [`LICENSE`](LICENSE) and
[`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). Distributed only from the
HYDRA product page on the Serial Cables website; `pyproject.toml` carries the
`Private :: Do Not Upload` classifier so PyPI rejects any upload. Every
distributed source file carries the `LicenseRef-SerialCables-HCC` SPDX header
(`tests/test_licensing.py` enforces it — new modules need it too).
