Provider operation and configuration

This reference is for the developer or operator running the provider. For inputs, units, per-atom results and scientific limits, start with the scientific guide.

Provider startup configuration

The binding's result schema is nmrnet.coordinate_shifts.result.v2. Submission instructions may request low, medium or max verbosity and 1–17 significant digits; the defaults are medium and 4. Interpretation records, warnings and scientific limitations remain at every level. Medium includes both coordinate representations; max adds the software-file manifest. The selected output object is frozen before scientific execution. The provider encodes the final API request once and hands those exact bytes to the shared delivery service. Delivery retries preserve the request body; they do not rerender results with new defaults. Source: result planning and publication projection.

The API/GUI provider description is the entry point to the scientific guide. Include its URL in the configured service.description, for example:

NMRNet++ predicts per-atom ¹H and ¹³C chemical shifts (ppm) from 3D molecular coordinates using released models. Supply coordinates or generate conformers from SMILES. Scientific guide

Set the description in the deployment's startup configuration. The operator example provides the summary and operational settings. The scientific offering is maintained once in nmrnet_provider/about/hello.md and installed with the provider. Both deployments publish this Markdown text through Provider Hello. To change the offering, edit that file, rebuild the images and redeploy; deployment TOML contains no offering override.

The internal nmrnet_provider.config.load_provider_config loads one explicit UTF-8 TOML file, up to 64 KiB, into the existing runtime policies. It reads the configured credential, interpreter keys and TLS roots before any HTTP request, native operation or network acquisition. Run its focused, offline container tests with make test/integration/provider-core/config.

Deployment installation reads api from the provider's private api.toml and obtains delivery.socket_path and delivery.socket_group_id from this provider's config/deployments/<name>/deployment.toml shared project and group. The shared SDK derives the socket paths and volume names from those values. The table below describes the generated runtime configuration; keep those derived fields out of the authored provider definition.

Installation writes that runtime file under secrets/deployments/<name>/bundles/generation-*/config/provider.toml. Change the authored inputs and reinstall; do not edit a sealed generation.

The runtime file requires schema_id = "nmrnet.provider.runtime_config.v4" and these closed tables; unknown fields fail startup:

Delivery handoff exchanges use the API exchange and retirement time limits.

scientific_retry.max_retries is 0 or 1. The supplied configuration uses 0. With 1, a model call that raises before returning may be retried once with a fresh worker and the same prepared batch, after the original worker has been retired. Earlier calculations in the batch may have completed and will be repeated; the failed call may also have performed work before raising. A lost reply, a worker crash, or a failure after the model call returned does not trigger a retry. The retry shares the original execution time budget. This setting does not change API or delivery retries.

The supplied definitions reserve two live Attempt slots, 256 KiB for control records and 4 MiB for pending terminal reports. Each slot reserves 128 KiB of control capacity and a 2 MiB encoded-report allowance before START. These are bookkeeping and report allowances, not limits on total process memory. A full registry pauses new START requests and keeps unresolved obligations. The live START policy allows the initial request and at most one permitted replay within 90 seconds of the first send; the API contract and stop request still govern whether a replay is allowed.

Paths are absolute container paths. Startup inputs must be regular files without a final-component symlink. Secrets must belong to the container user and have mode 0400 or 0600; configuration and CA files must belong to root or that user, with no group or other write permission. Parent mounts remain host-controlled. Keys and signing credentials are limited to 16 KiB each; CA files to 1 MiB. Interpreter keys allow one final LF or CRLF; other whitespace is rejected. The API-issued signing credential must match the intended profile and provider. Set API topology explicitly to dev-local, dev or web; it is a different vocabulary from credential profiles (dev-local, dev, run). Every API reply must carry the matching environment marker. Valid header request IDs remain available for authorized diagnostics when a reply is malformed or interrupted; they establish neither acceptance nor a safe retry. Endpoints, models, identities and budgets have no guessed deployment defaults. Generation's cooperative timeout must be below its channel deadline. Job states are an explicit nonempty selection of open and closed; cancelled Jobs stop work. Service presentation is validated before runtime resources are opened. The analysis kind remains the implemented peaks_from_mol; it is not a configuration selector. source_seconds controls the supplied-coordinate reader channel. SMILES generation uses preparation.generation_seconds. These waits do not change the host's worker isolation, resource ceilings or accepted generation settings.

Each HTTP role uses verified system roots unless ca_file supplies replacement roots. Explicit CA bytes are read once and used directly, including when a path changes afterward. Empty or malformed CA files fail; they never select system roots. Ambient SSL_CERT_FILE, SSL_CERT_DIR and SSLKEYLOGFILE are rejected. The analysis kind is the code-owned peaks_from_mol offering, following G01's identity. source_sha256 identifies the provider TOML and ordered endpoint filenames and TOML bytes, not key/CA contents or the complete qualified runtime. Owned error explanations remain public; original file/parser evidence and independent close failures remain protected causes.

This ports SECS's closed configuration and descriptor reader, using NMRPeak's existing policy construction. The port retains causes without exposing private paths, completes short reads, checks secret permissions, and binds TLS trust to the admitted bytes. The ambient TLS rules follow the pinned runtime's CPython 3.12.12 SSL implementation.

Launching the provider

The contained foreground entrypoint is python -m nmrnet_provider.process with required --config, --execution, --models and --diagnostics paths, plus --api-proxy-socket, --interpreter-proxy-socket and --store-proxy-socket. Every foreground HTTP role uses its explicit Unix route; omission or an invalid socket path refuses startup before retained storage or network acquisition. Its installed image is built and checked through the lanes below. Execution and model declarations are bounded to 64 KiB each and admitted before storage, HTTP or native allocation. Execution requires exactly image_id (prediction), geometry_image_id, controller_image_id, repository_revision, upstream_revision, runtime_identity_source = "caller" and nonempty tool_sources. Image IDs use sha256: plus 64 lowercase hexadecimal characters; revisions use 40. Source names are canonical repository-relative paths with exactly integer size and lowercase sha256; empty source files are permitted. These are caller-declared identities, not independently observed image or source-closure attestations. The process adds observed execution_declaration_sha256, prediction_declaration_sha256 and startup_configuration_sha256; callers cannot override them. Model declarations use the released selection. Results contain only dictionary and H/C checkpoint/scaler member names, positive sizes, SHA-256 and their archive SHA-256. Archive acquisition metadata is excluded; other unknown fields are refused. Scientific model-byte verification stays with the native owner.

The provider's private deployment.toml declares interpreter and Upload-store routes. The host's shared API proxy supplies the API route. Each scientific route selects local (a frozen address on an observed internal Docker network), mapped (an explicit host-network IPv4 destination), or external (interface-bound DNS). Each requires an explicit interface and log_level: INFO, DEBUG or WARNING. Mapped/local addresses preserve the configured hostname and port for TLS and signing; they do not fall back to DNS. External interpreter/store routes use the provider settings' resolvers.

Setup downloads use NETWORK_INTERFACE. Running services use the explicit interfaces in their route definitions. The choices may be the same or different; neither supplies the other. See the provider route example and the shared service's operator settings.

Persistent deployment

A named deployment uses Docker Compose and unless-stopped restart policy. Its six containers are the provider, three serial native runners (prediction, structure reader and conformer generator), and two proxies (interpreter and Upload access). The up command finishes after checking their configuration and current readiness. Docker must itself start after a host reboot. A separate shared delivery service on the same Docker host owns outstanding completion and failure requests. It can finish delivery while a provider is stopped or being replaced. The provider supervisor keeps its live Attempt records in memory. Restart does not reconstruct an interrupted Attempt, replay its START request or repeat its calculation. Reports already saved by shared delivery continue independently. A delivery acknowledgement confirms durable local storage; API acceptance is a later event. Sources: service admission and shutdown and the shared lifecycle design. Keep the public provider definition in config/deployments/<name>/provider.toml and private machine settings in config/deployments/<name>/deployment.toml. Copy the latter from the deployment settings example. Use mode 0644 for the caller-owned public definition and 0600 for private machine settings; neither may be writable by group or others. Keep the path and network settings in that private deployment definition. The provider-only Docker volume holds the lifetime lock; final request bodies use the delivery service's separate queue volume. Attempt bookkeeping is in memory and is not restored after the provider process stops.

Keep api.toml, the API credential, interpreter keys and TLS files in secrets/deployments/<name>/. The private api.toml contains the API fields listed above, with credential and CA paths under /srv/nmrnet/input/. Keep the shared proxy/outbox project name and socket group ID in this provider's config/deployments/<name>/deployment.toml. The shared service's operational settings stay in nmr-provider-tools. Follow the shared deployment instructions. Private directories require mode 0700 and secret files 0400 or 0600, owned by the deployment caller. The outbox retains the publication inputs supplied with each accepted result in its own private queue.

# secrets/deployments/<name>/api.toml
origin = "https://api.example.invalid"
profile = "dev-local"
topology = "dev-local"
provider_ref = "provider:example"
credential_file = "/srv/nmrnet/input/signing.private.json"
ca_file = "/srv/nmrnet/input/api-ca.crt"
exchange_seconds = 10
retirement_seconds = 5

Use the actual issued provider identity and API origin. Omit ca_file when the API uses system trust.

Build or load the NMRNet controller, geometry and inference images and the shared runtime image first. Copy config/deployment.toml.example to config/deployments/<name>/deployment.toml. Set its [deployment].project and [deployment].gid to the shared proxy/outbox Compose project and socket group; keep the shared service's operational settings in nmr-provider-tools. No socket grant belongs in the public provider definition.

provider/prepare records the three NMRNet image IDs under .work/provider/, .work/preparation/ and .work/inference/. Install admits those image records and captures the model selection from artifacts/prediction-models.json; up uses the frozen generation.

make -C /path/to/nmr-provider-tools delivery/deployment/up SHARED_SETTINGS=/path/to/nmr-provider-tools/config/deployments/localhost/settings.toml SHARED_RUNTIME_IMAGE=sha256:YOUR_SHARED_IMAGE

make provider/deployment/install DEPLOYMENT=<name> SHARED_RUNTIME_IMAGE=sha256:YOUR_SHARED_IMAGE
make provider/deployment/update DEPLOYMENT=<name> WHEELHOUSE=/path/to/shared-wheelhouse DOCKER_CONTEXT=<local-context>
make provider/deployment/up DEPLOYMENT=<name>
make provider/deployment/status DEPLOYMENT=<name>
make provider/deployment/logs DEPLOYMENT=<name>
make provider/deployment/down DEPLOYMENT=<name>

Install reads this deployment's deployment.toml, freezes provider inputs and shared socket grants into a new generation, switches the selected generation, and retires unused generations. It requires this provider to be stopped and does not start the provider, proxy or outbox. Start the shared proxy and outbox with nmr-provider-tools first. Up requires an empty NMRNet provider Compose project. The shared proxy and outbox use a separate nmr-provider-tools project; leave them running. If NMRNet provider containers remain, inspect their status and run make provider/deployment/down DEPLOYMENT=<name> before starting again. An incomplete or abnormal deployment may require inspection before down can safely remove its dependencies. Up initializes the provider's lifetime-lock volume, starts the proxies and serial native runners, then waits for provider readiness, including an accepted API Hello. This establishes readiness, not scientific accuracy or successful model interpretation. It also checks each running service's immutable image ID against the selected frozen Compose plan. Plain up never builds or pulls. Down requests provider shutdown and removes its dependencies only after confirming that the provider exited successfully.

Use provider/deployment/update for an explicit provider-image update. It builds from committed host-local inputs and the selected shared wheelhouse before downtime, then holds one deployment lock while it drains, installs the new provider image while retaining the selected prediction, geometry and shared runtime image IDs, starts the selected generation and verifies every running image ID. It preserves the durable volumes and leaves the independently managed shared proxy and outbox running. If a phase fails after drain, inspect status and logs; the command does not claim rollback.

Pass RETAIN_PRIOR=1 to install, update or up to preserve previous images and configuration generations for a separate retirement decision.

This down command requests provider stop without an Engine kill deadline. The command waits up to 120 seconds for its stop request. If that wait expires, is interrupted, or cannot confirm a successful exit, down returns nonzero and leaves the provider's dependencies, selected configuration and container logs available. Inspect status and logs, resolve the blocked delivery or cleanup, then retry down. Expiry of the command's wait does not establish that the provider stopped. Shared delivery and its queue remain available throughout.

Install retires unused older input generations. The selected generation and any generation mounted by a container, including a stopped container, remain. To retire unused generations without reconfiguring or stopping the provider:

make provider/deployment/retire-generations DEPLOYMENT=<name>

When moving a stopped deployment from its private API route to a shared route, retire its old API directory before installing the new configuration:

make provider/deployment/retire-api-route DEPLOYMENT=<name>

The command checks the selected recipe and refuses to remove a directory still mounted by any container, including a stopped one. It does not remove shared socket volumes or the interpreter and store routes.

After a successful make provider/deployment/up, image retirement removes obsolete NMRNet images. It preserves images referenced by any container, including stopped containers, all selected deployment recipes and current image records. Tagged images and digest aliases are also retained. Cleanup never forces removal or prunes parent images, build caches, volumes or another provider's images.

Builds and deployment installation hold a shared checkout-directory lock; retirement requires its exclusive lock. Use one canonical NMRNet checkout per Docker engine. The lock coordinates this checkout's supported commands; direct Docker mutations remain outside it. If cleanup is busy or fails, the healthy deployment remains up and reports the incomplete cleanup. Retry it independently with:

make packages/images/retire

make -C /path/to/nmr-provider-tools delivery/deployment/status PROJECT=<project-name>
make -C /path/to/nmr-provider-tools delivery/deployment/logs PROJECT=<project-name>
make -C /path/to/nmr-provider-tools delivery/deployment/down PROJECT=<project-name>

PROJECT is the project value in the shared settings. Status, logs and down inspect that project's actual Docker containers; they do not need a generated host recipe. Up derives the recipe from the operator's inputs and requires the project to be stopped. Keep its referenced input files available.

The independently managed shared proxy and outbox serve several providers. Their socket volumes and the outbox queue belong to the host deployment.

Update and redeploy

From a clean checkout on the Docker host, using the deployment owner's account:

git switch main
git pull --ff-only
git submodule update --init --recursive

Prepare NMRNet's three images and model files. The wheelhouse and setup interface are chosen once:

make provider/prepare WHEELHOUSE=/path/to/shared-wheelhouse NETWORK_INTERFACE=YOUR_SETUP_INTERFACE

For a provider-only update, use the maintained update command. Select the local Docker context and shared wheelhouse explicitly; the saved generation supplies the shared runtime image ID, so no image digest needs to be copied:

make provider/deployment/update DEPLOYMENT=<name> DOCKER_CONTEXT=<local-context> WHEELHOUSE=/path/to/shared-wheelhouse

For a shared-service replacement, coordinate all producers using it, inspect outstanding work, stop the incumbent owner, then start the shared owner with the shared settings and existing queue identity. Leave the shared proxy running. Do not run two delivery owners or discard unresolved data to complete an update. Container removal removes its bounded Docker logs; collect needed diagnostics before stopping it. See delivery ownership and API qualification.

Routine logs and private diagnostics

The provider, native runners, proxies and shared delivery service send stdout and stderr to bounded Docker logs: json-file, three files of at most 10 MiB each per container, with a 1 MiB nonblocking delivery buffer. A stalled sink may drop messages; old files rotate. Container removal deletes its logs; read bounded logs before stopping or reconfiguring a deployment. There are no parallel application log dumps.

make provider/deployment/logs DEPLOYMENT=<name> reads the latest 80 messages per retained component. Logs may include native tracebacks, molecular inputs and completed predictions after a failed reply; restrict Docker access. Scientific results belong to the API after confirmed publication; outstanding request bodies belong to the delivery queue. Missing log messages do not prove that a request failed or that work retired. The historical journald qualification in note 019 does not qualify this Compose logging configuration. Docker documents rotation limits and nonblocking delivery.

Result delivery and storage

The provider decides whether a scientific outcome is suitable for publication and encodes its final completion or failure request. It sends those bytes over its configured Unix socket to the shared service. The service commits them to its private SQLite queue before acknowledging the handoff. Its signing identity and API destination come from deployment configuration, not the submitted body.

The provider can release its in-memory result after that acknowledgement. The service delivers the unchanged body, with fresh request signing and capped backoff for permitted retries. A verified matching API receipt removes the queued body. Completed results are available through the API; the provider keeps no completed-result archive, per-Attempt result file or transcript history.

The provider keeps bounded live Attempt records in supervisor memory: the selected Job and START identity, the acknowledged Attempt, and references to the actual execution and delivery owners. It removes a record only after those owners finish cleanup and any selected report has been durably handed to shared delivery. Physical cleanup alone cannot release an undelivered result. Shutdown stops new admissions and interrupts a pending START. Acknowledged execution and its result handoff continue through shutdown. drain_seconds sets when the provider logs that it is still waiting; it does not cancel the Attempt. Existing execution and API time limits still apply.

The delivery queue has explicit item and byte allowances for each provider. Held items still count against that allowance. A full queue pauses fresh computation; a result already selected for handoff waits and retries the same bytes. Native cleanup proceeds independently of delivery. Unconfirmed cleanup prevents another scientific execution, while an approved result can still reach the API.

After a restart, shared delivery resumes its committed queue entries. The provider excludes relay-owned Jobs from fresh discovery but does not restore its former in-memory records, replay their START requests, inspect each old Attempt, or repeat their science. An unrelated old Attempt does not block a new Job. If the supervisor dies before durable handoff, a result held only in memory can be lost. An absent queue row never authorizes another prediction. Diagnose unresolved Attempts through the API and Docker logs.

Storage and lifecycle owners: durable delivery queue, live Attempt ownership and same-root lifetime exclusion. The design and existing evidence are in note 021; deferred refinements and further testing are tracked in note 022.

Checking API delivery

Reconciling held delivery data

A held item has no verified matching API receipt. Resolve the reported cause and remote Attempt before deciding to discard its local payload. First explicitly stop shared delivery; this affects all providers on this Docker host. Listing and discard run offline under the queue's existing exclusive lock and refuse while the daemon owns it. They never stop or restart the service automatically.

make -C /path/to/nmr-provider-tools delivery/deployment/down PROJECT=<project-name>
make -C /path/to/nmr-provider-tools delivery/deployment/held SHARED_SETTINGS=/path/to/nmr-provider-tools/config/deployments/localhost/settings.toml SHARED_RUNTIME_IMAGE=sha256:YOUR_SHARED_IMAGE API_ORIGIN=<exact-origin> PROVIDER_REF=<provider-ref>
make -C /path/to/nmr-provider-tools delivery/deployment/discard SHARED_SETTINGS=/path/to/nmr-provider-tools/config/deployments/localhost/settings.toml SHARED_RUNTIME_IMAGE=sha256:YOUR_SHARED_IMAGE API_ORIGIN=<exact-origin> PROVIDER_REF=<provider-ref> EXECUTION_ATTEMPT_REF=<attempt-ref> BODY_SHA256=<held-digest>
make -C /path/to/nmr-provider-tools delivery/deployment/up SHARED_SETTINGS=/path/to/nmr-provider-tools/config/deployments/localhost/settings.toml SHARED_RUNTIME_IMAGE=sha256:YOUR_SHARED_IMAGE

The list shows only the selected provider's metadata and reason, without payload bodies or a dump file. Discard requires that exact provider, Attempt and digest and removes only a held row; pending rows remain unchanged. It proves neither API acceptance nor resolution of the remote Attempt. Producer sockets have no deletion route. Source: offline operator command and queue disposition owner.

One scientific Job

The prediction targets create one persistent Job on the selected API. Pasted targets use fixed supplied chloromethane and compare against an existing pinned scientific receipt; generated-SMILES targets check received generation provenance and H/C coverage without an independent shift-value oracle. These targets are excluded from offline test aggregates:

To check an already installed deployment, use make provider/qualify/api/existing-pasted with explicit DEPLOYMENT, PROJECT_REF, USER_ID, CLIENT_CREDENTIAL_DIRECTORY, CA_FILE, SHARED_RUNTIME_IMAGE, OUTPUT, API_CLIENT_IMAGE and RETAINED_RESULT. The client directory must contain the entitled user's registration.json and signing.private.json; the provider key does not authorize user Jobs. Select the historical pasted oracle at the absolute path to artifacts/pasted-scientific-oracle.json as RETAINED_RESULT. Output, credential and CA paths must also be canonical absolute paths.

This lane creates only one finite public client. It derives API, provider, interpreter and current prediction-image expectations from the admitted original deployment; it neither launches a second provider nor opens its Attempt store. Its received result must match the unchanged numerical oracle and the current image pin. Configured correction/fallback is accepted within its existing turn budget; actual turns, corrections and routes stay in the retained result. Use a fresh private OUTPUT, including an ignored qualification leaf if desired. Failed or uncertain submissions are retained and never resubmitted automatically.

If the Job was created but client readback stopped, supply JOB_RECEIPT=/absolute/path/to/original/client/created.json and JOB_IDENTITY=/absolute/path/to/original/source/identity.json to the same existing-pasted target, with a fresh OUTPUT and the original client, project, network and oracle selectors. DEPLOYMENT still selects the local shared proxy grant settings in this mode. Expected execution comes from that original capture, so readback still works after redeployment or producer shutdown. It performs GET requests for that original Job only; it does not create, select or open another Job, or repeat prediction. The received result cannot supply its own expected image identity.

• make provider/qualify/api/pasted uses the isolated scripted model peer.
• make provider/qualify/api/real-pasted uses one explicitly selected real endpoint.
• make provider/qualify/api/real-smiles requests one generated CCl conformer through one explicitly selected real endpoint.
• make provider/qualify/api/real-smiles-multiple requests three generated CCCC conformers and checks each geometry's provenance and H/C coverage.
• make provider/qualify/api/real-upload attaches an XYZ file and compares its supplied atoms and predictions with the pinned scientific fixtures.
• make provider/qualify/api/real-sdf-zip publishes one ZIP containing the authored two-record, carbon-only SDF and creates one API Job. It compares both geometries and eight carbon shifts exactly with the retained R46 result, while checking the new Upload's separate archive/member identities. Its readback, process and host budgets are 900, 1200 and 1220 seconds. This does not exercise browser submission. An uncertain transfer must be reconciled using the original Upload. After its storage owner confirms absent bytes, pass UPLOAD_RECEIPT=/absolute/path/to/original/client/upload-created.json with a fresh OUTPUT to continue that reservation without creating another Upload. A null finalization hash alone does not prove absent bytes. This explicit continuation checks the live identity/state and requests a fresh capability; it does not recover the old capability or retry a transfer automatically. Omitting UPLOAD_RECEIPT creates a new reservation.

Command-line selectors for these targets are:

• API_ORIGIN, TOPOLOGY, PROJECT_REF and USER_ID: the operator's chosen API, expected topology and entitled client/project. Provider identity and credential profile are read from the API-issued JSON file selected by PROVIDER_CREDENTIAL.
• PROVIDER_CREDENTIAL, CLIENT_CREDENTIAL_DIRECTORY and CA_FILE: absolute paths to the issued provider key, the client's original registration/private pair and the API trust certificate. Keys must remain caller-owned and mode 0400/0600.
• ADDRESS and INTERFACE: the explicit mapped Upload-store route for the separately staged real-API targets. API client requests use the shared proxy.
• API_CLIENT_IMAGE: an immutable local API runtime image ID containing the API-owned credential validator/signing example. No implicit pull or installation.
• DEPLOYMENT and SHARED_RUNTIME_IMAGE: qualification selects config/deployments/<name>/deployment.toml using the existing named deployment selector. The file supplies the shared project and socket group. SHARED_RUNTIME_IMAGE is the immutable shared runtime image. The client borrows the existing proxy socket read-only and checks that its exact volume exists first. Staged targets also require the WHEELHOUSE containing the pinned core archive.
• RETAINED_RESULT: an absolute path to the existing scientific comparison receipt; its expected hash remains pinned in the fixture client. It is never regenerated from this run's predictions.
• OUTPUT: a fresh absolute private directory outside the checkout, separate from all credential/trust inputs. It retains configuration and protected receipts. The provider's lifetime lock and delivery service's outstanding terminal bodies use separate named Docker volumes. Preserve both the directory and those volumes; inspect them before any new mutation after failure.

The scripted target additionally requires LOCAL_NETWORK and LOCAL_INTERFACE for its peer's observed internal network. The real targets instead require:

• INTERPRETER_CONFIG: an absolute path to one private endpoint TOML file. Its key must be in the separate key_file; optional ca_file supplies model trust roots.
• EXTERNAL_INTERFACE and RESOLVERS: the explicit model-egress interface and space-separated literal IPv4 DNS resolvers. These remain separate from the API publication's ADDRESS/INTERFACE.

Each real target first inspects fragment metadata without a key grant. It then grants only the referenced individual key/optional-CA files to offline staging, uses the existing configuration admission, and freezes their exact bytes under the private output. It selects no fallback endpoint and starts no scripted peer. The existing foreground host and native launchers own execution and retirement; no API signing credential or provider configuration is sent to the model.

The maintained profiles use these finite qualification limits, in seconds:

Real staging sets the preparation budget to 240 seconds. Native prediction prepare, execute and channel closure each have their own 90-second policy; host retirement and API publication are separate operations. A deadline does not prove unsettled work retired: the original owners remain held until release is confirmed. See tools/provider_api_qualification.sh and tests/integration/provider_runtime/{stage_api_config,api_prediction}.py.

Job creation and subsequent mutations are sent once. Failure or uncertainty retains the original requests and receipts; it does not automatically resend a mutation or failed model request. Each profile defines its expected interpreter operations, scientific comparisons and confirmed-cleanup requirements in the API fixture client. A result can therefore have been accepted by the API even when the target exits nonzero and writes no client/passed.json.

Diagnostics and shutdown

Diagnostics require an absolute directory, mode 0700, owned by the container user. Process readiness uses expiring status observations tied to the current process identity. Causal error reports go to stderr/Docker logs without a file fallback. SIGINT/TERM stops new work and requests bounded settlement. If an original borrower remains active, the process reports requires_host_retirement = true, keeps its original owners alive and waits for the host to retire it; a normal stop signal does not cancel the same work again. Ordinary deployment down preserves that held process rather than imposing a forced-stop deadline. Emergency termination can lose a report that shared delivery has not acknowledged; it does not establish successful drain. An unadmitted diagnostics directory receives no files; safe reasons or OS errno still reach Docker's stderr logging. Neither diagnostics nor the readiness observations establish successful science. Run the focused subprocess proof with make test/integration/provider-core/process.

Execution failures include the acknowledged Job and Attempt references in the protected report. Native setup can fail before an Attempt exists; when a Job was already selected, that setup failure reports its Job reference alone. Source-only setup without Job context leaves those references absent. These fields identify affected work; they do not establish dispatch or acceptance of a result.

Images and network isolation

To acquire the provider dependencies and model files and build its three images in one invocation:

make provider/prepare WHEELHOUSE=/path/to/shared-pool NETWORK_INTERFACE=YOUR_SETUP_INTERFACE

Omit either selection to choose it once on the terminal. The command runs the existing owners in sequence and stops on failure. It reuses cached downloads and keeps existing extraction trees with their receipts; selected model bytes are checked against pinned hashes by the provider before decoding. It requires the initialized upstream submodule and the pinned Python base, provisioned with make packages/base-images/python/download. That explicit bootstrap uses Docker daemon routing. Build/load the shared runtime through nmr-provider-tools. API/interpreter credentials and service configuration are subsequent steps.

Build and check the installed controller offline:

make test/unit/provider/image
make provider/image/build WHEELHOUSE=/path/to/shared-pool
make test/integration/provider-runtime

The build uses committed input bytes and the released core wheel and locked runtime dependencies selected from the shared WHEELHOUSE root. It first requires the pinned Python base locally; it never downloads a missing base. The resulting image contains a wheel-installed provider-core and image-owned binding modules in an isolated, pip-free environment. Startup requires no source or wheel mounts and installs nothing. The recorded immutable image ID and committed-input label are checked before image tests. /opt/provider/dependency-inputs.json retains lock and wheel hashes from pip's actual installation reports; the source/lock label alone does not identify those platform-wheel bytes. Installed-image checks cover the full dependency set, native signing, import origins and isolated CLI startup. They contact no API or model and perform no prediction.

The shared provider package maintains the service proxy integration test.

The controller's HTTPS transport can use a fixed Unix CONNECT socket. The service proxy admits only explicitly granted DNS-name/IPv4 authorities before resolution or connection. Local routing requires a complete frozen host-to-IPv4 mapping; external routing uses the existing explicit interface binding. Both preserve destination TLS. IPv6 literal destinations are refused. The proxy receives no API/model credentials. These checks cover policy, private listener startup, diagnostics and actual loopback TLS. The host launcher must still bind those components to the intended networks and confirm exact container retirement; these component tests establish neither deployment nor external model access.

Attempt and service ownership

nmrnet_provider.runtime.assemble_attempt connects these settings to the existing input interpreter, source acquisition, NMRNet session and core controller. The caller selects a never-attempted Job from signed ProviderAPI.jobs and supplies the live claims, stop event and scientific allocation factories. The controller reserves bounded live capacity before START. An acknowledged START precedes the Job input read, interpretation and scientific allocation. The returned controller runs that same pending start and owns its session's cleanup. Each Attempt has separate lifecycle and source HTTP exchanges. The provider holds its lifetime lock until the actual owners are ready for shutdown.

The controller selects the final request before handoff. Native retirement and delivery are independent: a cleanup failure holds worker reuse, while the approved request can still be delivered. Every session receives an explicit preparation owner; production supplies the configured interpreter and source readers before the shared scientific execution and delivery path.

nmrnet_provider.service.ProviderService coordinates one Attempt at a time: retained-work/capacity and live-inventory checks precede Hello, fresh discovery and allocation. Verified relay-owned Attempts do not block unrelated Jobs, and those Jobs are excluded from new computation. It refreshes the current announcement through independent I/O, drains admitted work on stop, and retains unsettled tasks/controllers. Startup acceptance does not establish scientific readiness.

Startup admission and idle discovery automatically retry reviewed GET failures after the original API I/O retires. Each retry repeats complete admission and discovery from the first page, before allocating work. Initial and periodic Hello announcements also recover from reviewed availability failures: each send rebuilds the latest complete announcement, including changed or withdrawn offerings. The first accepted Hello still gates discovery and allocation; a periodic Hello outage does not cancel an active prediction. The initial delay is service.idle_seconds; it doubles after each failed cycle, capped by the required service.api_backoff_max_seconds, which must be at least idle_seconds. Each local wait is randomly selected between half and all of that delay. A valid API Retry-After delay is added to it and can exceed the local cap. A complete successful read cycle or accepted Hello resets its own backoff. Requests keep their configured deadlines; an outage may wait until recovery or stop, and stop interrupts the backoff wait. A startup request already in flight still has its exchange/retirement deadlines.

During Upload selection and acquisition, eligible temporary failures can be retried within the same Attempt. Each source operation allows at most three attempts, including the first. The provider retires the previous exchange and checks that the Job's Upload membership is unchanged before requesting another read grant. All attempts share the original request, byte and time limits; stop interrupts the wait. Docker diagnostics report the original failure and any uncertain earlier grant. Recovery does not repeat interpretation or prediction. Source: shared Upload owner.

Safe reads and fresh complete Hello announcements can recover after rate limits and HTTP 500, 502, 503 or 504, without treating an unverified error body as API authority. The verified API refusals request_body_timeout (HTTP 408), authentication_window_closed and signature_expired confirm that this request did not run; earlier unanswered requests remain unresolved. Existing operation owners may retry with a fresh signature after transport retirement: service reads and Hello keep their paced, stop-responsive retry loops; retained writes keep their own route, command and recovery-window checks. These read and Hello permissions exclude invalid credentials, future-dated signatures and reused nonces. Explicit wrong environment, malformed success, storage/ownership errors and admission holds still stop. Sources: API timing-refusal contract, body-timeout refusal, this-send effect policy and operation-specific retry permissions.

Shared delivery retries retained completion and failure requests after rate limits and recognised API authentication or authorization refusals, as well as network and gateway failures covered by its retry policy. Each attempt preserves the request body and uses a fresh signature. Capped exponential backoff includes jitter; a valid Retry-After adds a minimum wait before that jitter. Stop interrupts the wait. Repeated refusals remain visible in Docker logs: check the reported credentials, clock or API permissions. A matching API receipt clears the queued body; durable local acceptance alone does not establish API publication. Conflicting outcomes remain held; trust, storage and ownership failures require operator attention. Sources: terminal replay permissions, delivery pacing and receipts and HTTP Retry-After.

Active Attempt observations also recover within the required finite lifecycle.observation_gap_seconds window. Each fresh-observation wait includes exchange-lock acquisition, reads, transport retirement and backoff. Its initial delay is the smaller of lifecycle.poll_seconds and service.api_backoff_max_seconds, then doubles with the same half-to-full jitter and cap, clipped by remaining time. The original interpretation or prediction may continue during that window; another phase or an accepted result still requires a fresh allowed observation. A reply that arrives after the window cannot grant that permission.

Window expiry starts scientific retirement independently of HTTP cleanup. The original read remains owned until it settles; neither elapsed time nor requested cancellation proves that work stopped. Expiry, observed server stop and other active Attempt failures still stop the service; preserve unresolved control identities and settle their original owners before restart. No allocation, Attempt mutation or science is retried by the read loop. A Hello receipt confirms the new announcement, without resolving whether an earlier send was accepted. Between settled Jobs, the provider clears handed-off control metadata after cleanup. Unresolved producer control blocks fresh work; delivery-owned requests stay in the shared queue until publication or explicit operator disposition.

The foreground process emits bounded, redacted api_read_retry and api_read_recovered notices to stderr/Docker logs; Hello uses api_hello_retry and api_hello_recovered. Their latest_hello_send_confirmed flag describes the last sent snapshot's receipt; it does not attest to later supplier or directory state. Their failure evidence describes the preceding failed send. Shutdown reports unconfirmed Hello acceptance through provider_shutdown_observation log records, including stop during initial backoff or a later interrupted send. Pipe notices fit the atomic-write limit; an oversized notice never leaves partial JSON. Notice writes and final error reports do not wait for pipe capacity. The latest failed notice and its output cause remain in memory for shutdown reporting; there is no file fallback. A notice-output failure alone does not terminate the service or change science's outcome. A failed final error write retains original, cleanup and output causes in the owned exception; unresolved work stays held.

Active-read recovery uses api_observation_retry and api_observation_recovered, with the acknowledged Job and Attempt references. Recovery confirms a fresh API observation, not completion or publication; observed server stop still wins. If reporting fails, these notices use the same bounded in-memory diagnostic owner. This observation recovery does not retry mutations. Final request delivery belongs to the shared service after durable handoff. It follows the existing API owner's exact-request replay policy, freshly signing the same body and retiring each original exchange before a permitted retry. Reviewed authentication/rate-limit responses remain in paced retry with fresh signatures; repeated refusals require credential, clock or permission investigation. Explicit environment/identity mismatches and unusable original publication inputs pause publication for that provider. Inspect the retained work and reconcile its original identity and trust; do not silently substitute another credential. An item-specific conflict holds its request. These facts remain visible in Docker logs, without repeating interpretation, geometry generation or prediction. The replay authority is the API contract fixture and its terminal POST x-nmr-recovery/HTTP 500 descriptions, identified in that manifest.

For the fixed preparing and running labels, a reviewed delivery failure can leave progress unconfirmed while this Job proceeds. The service first confirms original API exchange retirement, emits the protected api_progress_unconfirmed notice, and obtains a fresh allowed Attempt/Job observation before dependent work. The notice includes the phase, progress_delivery_confirmed=false, real Job and Attempt references, and safe cause/correlation facts. It does not mean work has continued yet. That read does not confirm progress delivery; no failed label is resent. Authentication, wrong environment, conflicts, malformed received receipts, cancellation and unconfirmed cleanup still stop this path. Session condition reports keep their acknowledgement requirement. Failed notice output uses the existing protected final-report fallback. No extra deadline or retry setting is needed: the fresh read uses lifecycle.observation_gap_seconds and the configured API I/O bounds. The pinned progress PUT's x-nmr-recovery describes why stale progress must not be replayed; see the contract fixture manifest.

An awaited condition update can retry a reviewed temporary delivery failure while that same sequential report is still current. It settles the original exchange before retrying the same body with fresh signing, and dependent work waits for the matching receipt. The retry-admission window starts at the first failed send and uses lifecycle.observation_gap_seconds, including retirement, notices, lock waits and backoff. An admitted exchange still owns its I/O until settled. The existing lifecycle monitor can stop work during backoff. Protected api_condition_retry and api_condition_recovered notices identify the phase, condition and Attempt; only the latter reports condition_receipt_received=true. This recovery does not retry a model call or scientific calculation. Authentication, conflicting state, malformed receipts, cancellation and unsettled cleanup remain operational failures. Configure only one progress writer for an Attempt: local lifetime/claim locks do not fence an independently deployed provider. Source: ConditionRecovery and AttemptExecutor, can_retry_condition, and the same pinned progress PUT recovery contract above.

When the original worker exits during preparation or prediction, a positive observation made before local stop and within the Attempt's execution permission produces a worker_crashed failure for that exact acknowledged Attempt. The provider does not repeat its calculation. The public failure gives the phase and verified exit status or signal. The provider attempts to log the underlying diagnostic for the operator. A signal alone does not establish an out-of-memory diagnosis. Local stop requests, channel loss and missing records do not establish a crash. An already selected completion or failure keeps its own delivery responsibility.

With scientific_retry.max_retries = 0, an Attempt dispatches prediction at most once. The provider attempts to retire the worker after a failure; if retirement cannot be confirmed, it retains the cleanup responsibility. It writes bounded diagnostics to Docker logs on a best-effort basis. If the log sink cannot accept a notice, the provider keeps the latest failed notice in memory and tries to report it during shutdown; that report can also fail. A process restart loses the in-memory notice.

The policy is implemented by NMRNetSession, NativeAttemptFactory and NativeOperation. The installed consumer tests exercise real child exits during reader preparation and prediction startup, exact failure handoff before delayed cleanup, and the absence of a scientific rerun.

The opt-in offline target make test/integration/provider/prediction-crash runs a synthetic child under a fixed 512 MiB limit. It checks one original SIGKILL observation, one SDK-selected failure, no replacement execution, completed native cleanup and exactly one additional kernel OOM kill in the runner's cgroup. The public failure uses the verified signal and phase; the cgroup counter supplies the separate OOM evidence. API and relay receipts are authored local fixtures, so this probe establishes neither deployed API delivery nor durable queue storage. It uses the recorded provider image and loads no released models. The probe and launcher define its checks and retained qualification evidence.

The Job's own preparation.seconds limit is different. Once its original model, source and native resources have retired, expiry produces the public failure input_selection_time_limit. Protected evidence retains the configured limit, received model response facts and any uncertainty about a cancelled model POST. Durable relay acceptance of that FAIL and confirmed native cleanup permit the service to accept a later Job; API delivery proceeds independently. A foreign timeout, independently observed operation failure or unconfirmed cleanup remains operational and stops the service. In particular, an Upload session's cumulative deadline is not reclassified as this Job limit. No model request or scientific operation is repeated to recover from selection expiry.

After a reviewed START availability failure, HTTP 500 or unverified pre-head protocol response, the same starter can resend the exact retained Job/key/body with fresh signing. It first settles the original exchange and checks the API-owned replay permission. The live policy allows at most lifecycle.start_replays replays within lifecycle.start_seconds of the first START: one replay within 90 seconds in the supplied configurations. Capped jittered waits respect that limit and stop. Each exchange and its cleanup remain bounded; no prediction child is allocated while START or input interpretation is still pending. Authentication/topology faults, malformed success, cancellation and cleanup failure stop automatic START replay. An unresolved START remains owned in memory and does not authorize science. A terminal replay admission retires the session without starting science.

Protected api_start_retry notices identify only the known Job. Once an acknowledgement is verified, api_start_recovered also identifies its actual Attempt. Notices use the bounded stderr channel collected by Docker. The focused signed local TLS service witness loses its first accepted START reply, returns the same admission on exact replay, and performs one interpretation and prediction. Its scientific/model peers are authored fixtures; this is local lifecycle evidence. START replay authority is the same pinned API fixture's START POST x-nmr-recovery and HTTP 500 description, not a blanket permission to repeat mutations.

preparation.max_worker_calls limits each Job's attempts to read or generate molecular coordinates. Once exhausted, no additional source grant or worker allocation begins. That Attempt receives input_worker_call_limit with the limit and safe next steps; protected evidence retains actual call counts and prior interpreter diagnostics. After the relay durably accepts that FAIL and original resources retire, the service can accept another Job within available capacity. The limit is separate from an invalid input diagnosis. Unsettled producer cleanup, a full live registry or unresolved lifecycle I/O can block fresh work. A shared publisher outage can leave outcomes queued while other Jobs proceed within the queue limits.

If an Upload remains unavailable, observation of an active Attempt is lost for too long, or a prediction returns no usable confirmed result, the provider may retain that Attempt without reporting a result or repeating the calculation. After the original work and its resources have settled, it can accept another Job if capacity remains. The provider attempts to write an attempt_held notice to Docker logs with the affected Job and available failure details. Not every failure permits continuation: an independent cleanup or control failure still stops the service. Held Attempts occupy live capacity. A provider restart loses these in-memory holds; it does not reconstruct those Attempts, resume their calculations or report a guessed outcome. Results already accepted by the delivery service continue through that service.

The focused service lane also verifies the actual signed failure request: a model-reported input problem reaches the local API recipient with unverified attribution and owned guidance for unclear units, conformer count or Upload record selection. Its freeform explanation stays in protected evidence. An operational science error keeps its original cause and sends neither a failure nor a completion. This does not qualify GUI presentation or an operator diagnostic renderer.

Numerical qualification evidence uses the production factory and installed serial runners. It includes supplied chloromethane with explicit nm units, RDKit-generated CCl geometries, 137 decoded predictions and the original 64 reference positions, plus an H128/C128 capacity case. Source/image identities, result hashes and scope limits are recorded in note 021 and its linked evidence. Capacity is distinct from chemical accuracy, and scripted interpretation or local API peers do not establish live external-model behavior. Further delivery-refactor testing is tracked separately in note 022.

Prediction, supplied-geometry reading and SMILES generation each have a persistent serial runner. Each operation gets a fresh child and private scratch. The native supervisor owns child termination, reaping and scratch cleanup. A stable lease and correlated generation/operation identify that ownership; a socket closing or a container becoming ready alone does not prove it retired.

The native factory owns source/operation handoff and interrupted setup; the runner owns its connection, cancellation and retirement. Producer control metadata and the delivery queue have the separate storage responsibilities described above.

These checks ran actual RDKit and released NMRNet CPU inference. Interpreter replies were scripted and the signed API peer was local; live LLM interpretation, external API delivery requires separate evidence. GUI behavior is outside this repository's system under test. The source/image identities, unchanged scientific inputs, result hashes and limits are recorded in note 021 and its linked evidence packets.

Check the interpreter's file-format choices

provider/qualify/interpreter/formats sends three fixed requests to the selected real model: one MOL structure, PDB models 2 then 1, and MOL2 records 2 then 1. The model sees authored instructions and direct-file Upload metadata. The probe captures its actual prepare_upload arguments, then checks the named source, format, units and record order. It creates no API Job, acquires no source bytes, parses no molecular structure and computes no shifts. No prepared structure or scientific result is fabricated. Production interpreter instructions/tools stay unchanged; a wrong choice stops the batch without coaching a later choice.

make provider/qualify/interpreter/formats \
  INTERPRETER_CONFIG=/absolute/private/interpreter.toml \
  EXTERNAL_INTERFACE=eth0 RESOLVERS='192.0.2.53' \
  OUTPUT=/absolute/private/fresh-format-probe

This is an opt-in paid model workflow, outside default offline aggregates. It uses the existing single-endpoint carrier, individually selected key/trust grants, explicit interface-bound proxy and exact retirement. Its deadline derives from three cases and the selected turn/exchange/retirement limits. It stops on the first failed case, makes no whole-case retry and keeps unresolved cleanup held. Process streams use Docker logs; private case receipts retain bounded redacted arguments, interpretation facts, comparison refusal and independent failures.

interpreter_formats.json and the fixture provenance fix the cases. This evidence is only model argument choice; existing R37/R46 reader and science evidence supplies the separate deterministic part. It does not establish authenticated source transfer or GUI packaging for these three formats. Current qualification status and exact scopes are recorded in R51's probe design and R54's observed outcomes.