Docker-backed process logging
Dated design and evidence record. Original wording, line order and historical source identities are preserved; use the current guides for operation.
Record sections
Docker-backed process logging ============================= Decision -------- The user's 2026-10-23 instruction supersedes the earlier private-stderr split: "All should go to bounded docker logs. Remove the file dump path and the dumped files." Both stdout and stderr now go directly to Docker journald, including raw tracebacks and completed scientific outcomes attached to exceptions. The user accepted that these logs can contain molecular data. Existing credential redaction and RDKit's scoped diagnostic capture remain unchanged. Use standard Python logging, owned INFO by default, explicit LOG_LEVEL=DEBUG or WARNING, and a dependency WARNING floor enforced at the root handler even when a dependency enables its own INFO/DEBUG logger. Imports remain passive. Do not add twig2, stream replacement, a collector, a queue or a parallel file fallback. Ownership --------- tools/provider_logging.sh supplies the shared journald policy: nonblocking delivery, a 1m buffer, and nmrnet-{{.FullID}} tag. Normal workers, controller, proxies and finite helper roles all use it. Production launch owners admit the observed LogConfig alongside their existing isolation/resource posture. Journald support is required; there is no alternate-driver fallback. The 1m limit bounds pending delivery, not retained disk. Host journald owns shared disk/age limits and access. No host configuration is changed. The inspected existing auto-cleanup-limits.conf declares SystemMaxUse=300M, SystemKeepFree=1G and MaxRetentionSec=14day. This is not guaranteed 14-day or per-Job retention. Messages can be dropped or expire. Missing logs establish no lifecycle outcome. Ordinary process failure reports have one redacted stderr sink. There is no failure.json or ordinary per-process shutdown-summary file. A failed write keeps original operation, cleanup and output causes in ProviderProcessFailed; the existing held-owner finally still prevents release of unresolved work. Clean shutdown can report unconfirmed Hello/notice-delivery observations without falsely declaring operation failure. Output failures do not trigger another API or scientific effect, a file fallback or automatic message resend. Actual data/control artifacts remain with their owners: - admitted startup marker; - stopped recovery's functional inspect/replay result.json; - scientific outputs and resource snapshots; - exact CID, engine-operation, retirement, consumption and replay receipts. The shared fresh-directory admission still refuses prior result.json because stopped recovery uses that same directory contract. Startup helpers previously used stdout for 17/7 NUL-delimited control fields. They now require --handoff and atomically publish those exact bytes as the exclusive mode-0600 host-fields artifact using the existing _publish owner. Successful wait and exact removal still precede adoption. Field counts, bounds, private ownership and byte-equal recovery admission are unchanged. No binary protocol is emitted into logs and no new parser or artifact framework is added. Operator and qualification paths -------------------------------- make provider/logs CONTAINER_ID=<full 64-character ID from worker.cid> reads the latest 200 journal messages, including a removed container's entries. This is an operator read filter, not a second receipt-admission authority. make test/integration/provider/logging uses the installed image, captured sources and actual service launch/removal owners for synthetic normal, fatal and host-stop cases. It checks both streams, fatal traceback/outcome notes and explicit DEBUG after removal. It queries logs in memory with finite deadlines; it does not create journal exports or stdout/ stderr dump files. Control receipts remain. No credentials, network, API Jobs, models or scientific accuracy reruns are needed for the logging proof. All worker/service/prediction-recovery qualification wrappers also inherit both streams under the same logging policy. Their outer host launchers no longer write foreground/inspect/recovery/restart log dumps. Superseded proof and rationale ----------------------------- The earlier split was qualified at dab4942243c13d8dcb0a0895315d698ccc9d5d7f, image f0cb7b4651f9f799b66ec3f2139667689b0510adaf56b5ae5c45f310b6157ef6, receipt .work/qualification/logging-k2QrFdWx. It correctly proved its then-current private-stderr contract, which the user subsequently rejected. Its source and lifecycle receipts remain useful historical evidence; it does not qualify the current both-stream contract. Current source/proof and deletion evidence follow. Current focused validation and cleanup ------------------------------------- The no-dump change passed contained provider/host 26, provider/allocation 51, provider-core/config 30, process/report 63, recovery 24, interpreter 38, provider-core repository 49, operator-input repository 25 and worker-startup 12 tests. Ruff 0.15.15, bash -n and diff whitespace pass. The process suite was rerun after fixing the independent review's one context finding: shutdown observations must preserve the original failed notice's event, redacted Job/Attempt identity, phase/code/receipt facts, outage and actual output cause. Its regression now receives the successful shutdown record after the initial notice write fails. Removed 138 reviewed ordinary stream dumps/exports and unused normal-process shutdown summaries. The exact relative paths and pre-removal hashes are retained as deletion metadata (not copies of the logs): .work/qualification/log-dump-removal.VrjrxHSD/removed.sha256 SHA256 4b6f7964923395acce4aa5b8e68008f591c9e2bf9bba156fe32585e8fd7b68a5 Seven historical binary handoffs, real recovery outputs, scientific/comparison observations, resource snapshots and engine/control receipts were deliberately preserved. No broad *.stdout/*.stderr/*.log deletion was used. The optional fuser probe failed under sandbox and established no open-descriptor fact; a separate Docker mount inspection found no active container mount under this checkout's .work. No running service or unrelated container was stopped for cleanup. Installed no-dump proof ---------------------- make test/integration/provider/logging and test/integration/provider-runtime both passed with committed source bdc2487724438203ee95eb5022dc75a168f36675 and image: sha256:35fb8e2a820de7305a8adfc7d9b1f7bf6139e68052ecc4218f1f9117cdaaabeb Receipt .work/qualification/logging-05rf2C1v retains source/control facts only. Captured input manifest SHA256: fc7915483f2242e72bd33f01d7946811b609938dabb4f9d2d836af3ffe6c5749 Captured source-hash manifest SHA256: 119bf7ecb2f87202775009a17ddc91525a98e3d0628d917121ef766696242c79 All three exact containers were removed before journal/operator reads: success: 68172610c2663d4299a648a35cb79b6077053fe84a54f9467b813baec89ea954 failure: 74e2d5ba190f248d93721a323fe0d0265994a54e4fad2b955c09ff9542ab180b host stop: 2d6cc452b5ee100689c41a308a3b91d5a76780398bd9d8cfbfbfc031d861d4db Normal/fatal wait statuses were 0/1. The proof received stderr's synthetic native marker and the fatal traceback/completed-outcome notes in Docker logs; DEBUG was visible only when explicitly selected, and dependency INFO remained absent. Every diagnostics directory remained empty: no stream dump or exported journal file was created. The actual operator command retrieved both streams after removal. The installed-runtime lane's four tests passed in 0.751 seconds. An independent read-only audit confirmed the actual post-removal journals, source/image/posture/removal receipts, all 138 manifest paths absent, and the preserved functional handoff/recovery/scientific artifacts. No mismatch remained. This is a synthetic logging/lifecycle proof, not new science, API/GUI qualification or an observed journal vacuum/retention-duration experiment. Sources ------- Inspected Magnet d1412b1847f5bf689ef0396cde32abd983f462cb: magnet_provider/provider_process.py:718-743: dependency filtering belongs at the sink; the root logger level alone does not filter propagated records. models/g_01_nmr_forward/runner/serving_worker.py:254-255: standard logging, root WARNING and owned INFO. Use serving practice, not twig2's demo queue. NMRNet's existing owners before this correction, 1c44e5a: nmrnet_inference/worker.py::serve_connection: failed result-send notes retain completed_outcomes. They now naturally reach Docker through raw stderr. nmrnet_provider/geometry_worker.py and smiles_worker.py: native failure notes. nmrnet_provider/smiles.py::generate_conformers and mol.py: scoped RDKit capture. nmrnet_provider/process.py::_publish: existing exclusive artifact publication. tools/provider.sh::provider_host_adopt_plan: exact host-field admission. tools/provider_workers.sh::provider_workers_retire/archive: resource and lifecycle receipts remain distinct from logs. Primary references inspected 2026-10-22 UTC: https://docs.python.org/3/library/logging.html#logging.Logger.propagate https://docs.python.org/3/library/logging.html#logging.basicConfig https://docs.docker.com/engine/logging/drivers/journald/ https://docs.docker.com/engine/logging/configure/#configure-the-delivery-mode-of-log-messages-from-container-to-log-driver https://docs.docker.com/engine/logging/log_tags/ https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html