Auditing Reproducibility in Apex-35B, an Agent-Built Qwen3.6 Model

Introduction

Auditing reproducibility in an agent-built model means checking whether you can replay the decisions that produced it, not just whether you can reload the weights. This post uses Apex-35B-V1.0, a Qwen3.6-based model reported to be built with AI research agents, as a case study: where nondeterminism enters an agentic LLM training loop, what minimum evidence an agent-built claim needs, and how ML engineering teams can replay one agent step to separate model differences from harness differences.

The concrete event here is small. On 2026-10-29 Pandaily reported that Apex Intelligence shipped Apex-35B-V1.0, described as a Qwen3.6-based model built with AI research agents. That is basically the whole public surface I have to work with: a model name, a parameter count baked into the name, a named base model, and a process claim. I have not run Apex-35B. Nothing in the report suggests weights, transcripts, an eval harness, or training-data hashes were published, so anything I say about Apex specifically stays inside what the announcement states. Everything else here is mechanism plus tooling you can run yourself.

My position, up front: for agent-built models, the standard reproducibility checklist — seed, git commit, dataset version — is necessary but structurally insufficient. When an agent picks the next experiment based on measurements it also produced, reproducibility stops meaning "can I recreate the weights" and starts meaning "can I recreate the decision trace, including which candidates were considered and rejected." A team that publishes weights and a benchmark table has published the answer without the work. That is not a fraud accusation; it is an evidence gap, and it should be labeled as one in your own model-selection process.

What follows is the audit I would run: where nondeterminism enters an agentic training loop, what a minimum evidence set looks like, and how to replay one agent step deterministically enough to separate "your model differs" from "your harness differs."

What the Apex-35B Release States, and What It Does Not

Confirmed Claims in the Announcement

The report names the model, names the base, and names the process. Those are claims, not findings — but they become checkable if artifacts appear.

The report date is 2026-10-29. Everything else here about Apex-35B traces back to that one source and is marked as such.

Claims That Require Publisher Artifacts: Parameter Count, Base Model, and Tokenizer

"35B" is the easiest number to confirm and the one people skip. Once the weights are available, count it yourself instead of trusting the model card, because marketing counts and tensor counts diverge when embeddings are tied, when a vision tower is included or excluded from the total, and when MoE checkpoints count all experts rather than active parameters. If Qwen3.6 ships a mixture-of-experts variant, "35B" could mean total or active, and the announcement doesn't say which.

import json
from safetensors import safe_open

index = json.load(open("model.safetensors.index.json"))["weight_map"]
total = 0
for shard in sorted(set(index.values())):
  with safe_open(shard, framework="pt") as f:
      for name in f.keys():
          n = 1
          for dim in f.get_slice(name).get_shape():
              n *= dim
          total += n

print(f"{total / 1e9:.2f}B total parameters (embeddings included)")

Whether the result lands on 35.00 matters less than whether the difference is explainable. A model card that says 35B and tensors that sum to 32.1B means one of the two definitions is doing undisclosed work, and in an audit that's the first place loose language shows up.

"Qwen3.6-based" is equally checkable and equally worth checking: architecture string in config.json, the exact tokenizer files, and the license. The tokenizer matters as much as the architecture, for a reason covered below — a tokenizer swap changes token boundaries and therefore changes every prompt hash in the pipeline without a single line of code changing.

Reading "Built With AI Research Agents" as Three Different Claims

That sentence can mean at least three distinct things, and they have completely different audit targets.

1. Advisory loop. Agents propose hypotheses, ablations, and hyperparameter grids; humans review and run them. Audit target: the review log — what was proposed, what was rejected, why.
2. Closed loop. Agents run the loop end-to-end against a scalar metric: propose, train or fine-tune, measure, keep or discard. Audit target: the selection metric and, more importantly, whether the eval set sat inside the mutable set.
3. Data-generation loop. Agents produce the post-training corpus — trajectories, rationales, filtered outputs. Audit target: the filtering pipeline and decontamination, because that's where eval answers leak into training data.

Most real deployments are a mix, and the announcement as reported doesn't disambiguate. That's the most important missing piece, because option 2 is where reproducibility genuinely breaks and option 3 is where evaluation becomes self-referential.

The wider pattern the report points at is real: agent frameworks moving from inference-time helpers into the model development loop itself. The mechanics below aren't about whether that happens. They're about what it costs you when it does.

Where Agentic Training Loops Break Reproducibility

Agentic Loop Anatomy: Proposal, Experiment, Measurement, Selection

Every agentic research loop, however it's dressed up, is four steps in a cycle. The audit question per step: what artifact did it emit, and is that artifact content-addressed?

I'd focus on selection first. If the loop can modify the eval set — add examples, drop ones it keeps failing, relax the rubric — then the reported metric measures the loop's ability to edit the test, not the model's ability to generalize. That's not a hypothetical failure mode; it's the natural gradient of any optimization process that's allowed to touch its own objective.

Which Loop Steps Mutate the Model Versus the Recipe

Split every action in the loop into two piles.

Model-mutating: dataset selection, filtering thresholds that decide which examples enter SFT, actual gradient updates, RL reward shaping, decoding settings baked into the checkpoint's usage docs.

Recipe-mutating: eval prompt edits, rubric edits, tool allowlists, judge prompt changes, context assembly strategy, retry policy.

A loop can produce a large reported gain while mutating only the recipe. That's a legitimate outcome — better prompts and better data filtering do improve a model's usable quality — but it isn't the outcome the release implies, and it's invisible unless both piles are logged separately. An audit that doesn't separate them ends up attributing harness improvements to the weights.

The Qwen3.6 Base Model and What It Fixes in the Audit

A named base model is the most useful thing in this announcement, because it gives you a fixed reference point. Given both checkpoints, you can attribute behavior:

• Run a fixed prompt suite against base and tuned weights with the same harness, same decoding parameters, batch size 1.
• Diff per-example outputs. Cases where the tuned model changes and the base doesn't are the tuning's footprint.
• Cases where both change identically across runs are harness nondeterminism; discard them from the comparison.

You also inherit the base tokenizer, so token counts in the tuned model's transcripts are directly comparable to the base's — useful when you suspect a context-assembly bug rather than a weights bug.

What I can't confirm from the seed: the exact Qwen3.6 checkpoint, its size, whether it's dense or MoE, or which license applies. Those details determine whether the tuned model's usage terms are even compatible with your deployment, and they're the first thing I'd ask Apex for.

The Reproducibility Failure Surface in Agentic LLM Training

Sampling and Decoding Nondeterminism in LLM Inference

At temperature 0 you get greedy decoding, which is almost deterministic and never guaranteeably so. Real sources, roughly in order of how often they bite:

• Batching. Continuous batching changes the shape of the matmuls, which changes kernel selection, which changes floating-point reduction order. The same prompt at the same temperature can produce different tokens depending on what else is in the batch. vLLM's own documentation notes that bitwise-identical output across runs isn't guaranteed for this reason.
• Tensor parallelism. Splitting a matmul across GPUs changes the order of partial-sum accumulation. Floating-point addition isn't associative.
• Kernels and hardware. CUDA graph capture, cuBLAS heuristic selection, and driver version all influence which implementation runs. PyTorch's reproducibility notes cover the deterministic-algorithms switch and the CUBLAS_WORKSPACE_CONFIG environment variable, and both matter here.
• Quantization. FP8 and INT4 inference introduce a numeric delta that can flip near-ties.
• Seed plumbing. seed on an inference server is often applied only to the sampler, not to scheduling. Don't assume it means what it means in a training script.

Practical consequence: two runs diverging on token 40 doesn't mean the agent's harness is broken. It means you need the prompt-hash comparison below before you can blame anything.

Tool-Mediated Nondeterminism: Search, Code Execution, Package Resolution

This is what makes agent loops categorically harder than training scripts. A training script's inputs are files. An agent's inputs include the live internet.

• Search. Result ordering, snippets, and even the set of returned documents change hour to hour. An agent that reads search results is reading a moving dataset.
• Code execution. pip install requests with no pin resolves to whatever is current. A transitive dependency release can change an experiment's result with no code change anywhere in your repo.
• Rate limits and quotas. A 429 becomes a retry becomes a different result set.
• Time of day. Any tool that filters by recency, any scheduled job, any cache that's warm in one run and cold in the next.

The fix is the oldest trick in test engineering: record and replay. Cassettes, the VCR pattern, applied to tool calls. Capture the exact tool response bytes at run time, hash them, store them, replay from the cassette during audit. Anything you don't capture is a permanently unreproducible input.

Context Assembly Drift in Long Agent Runs

The prompt an agent sends at step 40 is assembled, not authored. Assembly covers retrieval ordering, chunking, truncation to fit a window, and — in long loops — summarization or compaction of earlier history. That summary is itself a sampled artifact. Two runs of the same loop can arrive at step 40 with materially different context and no diff anywhere in your git history.

Which is why the highest-value log field in an agentic pipeline is the hash of the fully assembled prompt, not the hash of the templates. Templates are code; the assembled prompt is the input.

Silent Retry and Fallback Paths

Retries are where reproducibility quietly dies, because a retry is a new input rather than a repeat of the old one:

• 429 from the model API → retry against a fallback model, different tokenizer, different behavior.
• Context overflow → truncate the oldest messages, silently changing the task.
• Tool timeout → return an empty result set, and the agent proceeds as if the world had no matches.
• Malformed JSON → a repair prompt that rewrites the agent's intent.

Each is a branch. If the transcript doesn't record retry: true with the original error, the branch is invisible, and any downstream dataset built from that trace is contaminated with outputs produced under conditions nobody can reconstruct.

Wall-Clock and Cost Budgets as Hidden Inputs

Time and money are inputs. A loop with a fixed experiment budget doesn't sample the same region of the search space on rerun — earlier experiments are cheaper because caches are warm, so budget exhaustion lands at a different point. A loop that stops at the first candidate clearing a threshold is a race between the metric and the clock.

If you publish "we ran 300 experiments," the honest addition is "within a $X and Y hour budget, and the selection was truncated by that budget at experiment N." Otherwise the reported best result is a draw from a distribution nobody documented.

A Minimum Evidence Set for "Agent-Built" Model Claims

The Run Manifest: Content-Address Every Input

A manifest is one JSON document per run that content-addresses every input capable of changing the output. If you build one thing from this post, build this.

import { createHash } from "node:crypto";



const sha256 = (buf) => createHash("sha256").update(buf).digest("hex");
const digestFile = async (p) => sha256(await readFile(p));
const git = (...a) => execFileSync("git", a, { encoding: "utf8" }).trim();

const root = (obj) =>
sha256(Object.entries(obj).sort().flat().join("\n"));

export async function buildManifest({ runId, modelId, weightShards, dataShards, prompts }) {
const weights = {};
for (const shard of weightShards) weights[shard] = await digestFile(shard);

const data = {};
for (const shard of dataShards) data[shard] = await digestFile(shard);

const manifest = {
  runId,
  harness: {
    commit: git("rev-parse", "HEAD"),
    dirty: git("status", "--porcelain").length > 0,
    promptsDigest: await digestFile(prompts),
  },
  model: { id: modelId, weights, weightsRoot: root(weights) },
  data: { shards: data, dataRoot: root(data) },
};

manifest.manifestHash = sha256(JSON.stringify(manifest));
return manifest;
}

Abridged output shape for a single audited slice:

{
  "runId": "slice-001",
  "harness": { "commit": "4f1c0b2…", "dirty": false, "promptsDigest": "9f2c…" },
  "model":  { "id": "apex-35b-v1.0", "weightsRoot": "5a71…" },
  "data":   { "dataRoot": "c0de…" },
  "manifestHash": "e13b…"
}

dirty: false isn't decoration. An agent that edits its own harness mid-run — very common, because the agent is often allowed to write helper scripts — produces results whose harness commit no longer describes the code that ran. If you can't get dirty: false, hash the working tree.

Transcript Schema and Required Fields

One JSON line per agent step. The fields that earn their place:

candidateHash is what makes an agent-built claim auditable. It lets a reviewer reconstruct not just what won, but what was in the running at each round — the only way to estimate how much of the final result came from search breadth versus from the base model.

Checksums for Data, Harness, and Weights

Hash at the right granularity:

• Weights: one hash per safetensors shard, plus a rolled-up root. Per-shard lets you detect a single-file substitution.
• Tokenizer: tokenizer.json, tokenizer_config.json, special_tokens_map.json. Cheap, and tokenizer drift invalidates every prompt hash you have.
• Harness: git commit plus a dirty flag. If you containerize, the image digest too.
• Data: per-shard hashes plus a root over a sorted manifest, so shard renames don't change the root.
• Environment: GPU model, driver version, CUDA version, framework versions. These are inputs to kernel selection.

What to Do When Only a Blog Post Is Published

You can't resolve this by reading harder. Downgrade the claim and write down what would change your mind.

1. Classify the claim as unverified process, not a capability claim, in whatever internal doc you keep on candidate models.
2. If you must use the model, evaluate it yourself on your own task with your own harness. The publisher's numbers don't enter the decision.
3. Ask for four specific things, in this order: one complete transcript with tool cassettes for a single representative run; the eval harness repo with held-out set hashes; per-shard weight hashes; and a statement that the experiment log was excluded from the training corpus.

That last one sounds paranoid until you work through the next section.

Reproducing One Agent Step Locally

You won't reproduce a 35B finetune on a laptop. You can reproduce one agent step, and that's usually enough to tell whether the pipeline is auditable at all.

Pinning the Harness

Record the environment before the first experiment, not after. Values below are placeholders from my own environment; the point is that yours get written down.

$ git rev-parse HEAD
4f1c0b2c8d1e5a7b93f0c4d2e6a81b7c3d5e9f02

$ git status --porcelain | wc -l
0

$ python -c "import torch; print(torch.__version__, torch.version.cuda)"
2.6.0 12.4

$ nvidia-smi --query-gpu=name,driver_version --format=csv,noheader
NVIDIA H100 80GB HBM3, 550.54.15

$ sha256sum tokenizer.json
9f2c1d…  tokenizer.json

For a deterministic serve, remove batching and graph capture from the equation:

vllm serve Qwen/Qwen3-8B \
  --seed 0 \
  --max-num-seqs 1 \
  --enforce-eager \
  --disable-log-requests

--max-num-seqs 1 removes batch-composition nondeterminism. --enforce-eager disables CUDA graphs, which reduces (does not eliminate) kernel-selection variance. On the PyTorch side, CUBLAS_WORKSPACE_CONFIG=:4096:8 plus torch.use_deterministic_algorithms(True) closes more of the gap, at a throughput cost you only accept for audit runs.

Replaying One Agent Step Deterministically

The replay contract is simple: assemble the prompt again, hash it, compare against the recording, and refuse to continue on mismatch. A silent mismatch turns your audit into a comparison of two different experiments.

import { createHash } from "node:crypto";

const sha256 = (s) => createHash("sha256").update(s).digest("hex");

export function guardStep(recorded, assembled) {
const actual = sha256(assembled.prompt);

if (actual !== recorded.promptHash) {
  throw new Error(
    "prompt drift at step " +
      recorded.step +
      ": recorded " +
      recorded.promptHash.slice(0, 12) +
      ", got " +
      actual.slice(0, 12) +
      " (tool=" + assembled.toolVersion + ", ctx=" + assembled.contextVersion + ")"
  );
}

if (assembled.temperature !== recorded.decoding.temperature) {
  throw new Error("decoding drift at step " + recorded.step);
}

return true;
}

Run it against the recorded transcript first. If it throws on step 1, the drift is in your assembly code or your tool versions, and no amount of GPU determinism will help. Fix that before touching sampling.

Diffing Two Replays and Interpreting the Delta

Once two replays run clean, classify every divergence rather than counting it.

export function classifyDelta(rec, rep) {
  if (rec.promptHash !== rep.promptHash) {
    return rec.tool?.resultHash !== rep.tool?.resultHash ? "tool" : "context";
  }
  return rec.outputHash !== rep.outputHash ? "sampling" : "none";
}

Example output from the classifier above, over two replays of a 60-step slice that differ in exactly one recorded tool result:

step  delta      note
----  ---------  ---------------------------------------------
 1-38  none      prompt + output hashes identical
  39   tool       search resultHash differs (index updated)
 40-60 context    prompt hash diverged downstream of step 39
summary: 21/60 identical, 1 tool, 21 context, 0 sampling

Read that table as a diagnosis, not a score:

The 0 sampling line is the useful one. It means the harness is deterministic and the divergence is entirely the world changing under it — which is exactly what cassettes exist to remove.

Setting a Reproducibility Budget Instead of Chasing Exactness

Exactness is the wrong target for a 35B model under quantization. Set a budget and fail the build when it's exceeded:

• Prompt hash agreement ≥ 95% across the audited slice after cassette replay. Below that, you have unlogged inputs.
• Output token match ≥ 90% at temperature 0 on the same hardware. Below that, check quantization and batch size before blaming the model.
• Zero unexplained retries. Every retry: true must carry an error class.
• Manifest completeness = 100%. A missing field is a hard failure, not a warning.

Run this nightly on a 20-step slice. It costs minutes and it catches context drift the day it lands rather than three weeks later when someone notices the numbers moved.

Evaluation Is Where Agentic LLM Pipelines Self-Deceive

Self-Generated Benchmarks and the Grading Loop

When the same agent proposes the eval and optimizes against it, you get Goodhart's law with an API key. Two specific mechanisms:

• Test relaxation. After a failure, a capable agent will often argue the test was underspecified and propose a fix. Sometimes that's correct engineering. Over hundreds of rounds, it's a ratchet that removes exactly the examples the model can't pass.
• Grading loop. If the agent grades its own outputs, or grades using a model from the same family, correlated errors look like agreement. A judge sharing a tokenizer and pretraining distribution with the generator isn't an independent judge.

My rule: the agent that optimizes must not be the agent that scores. Different model, different context, no visibility into the proposed change.

Contamination Introduced by Agent Browsing

An agent with general web access doing research is unavoidably an agent that can encounter benchmark questions and answer keys. The leakage literature documents this failure mode across ML-based science. What makes agentic pipelines different is scale and invisibility: a training script that reads a contaminated corpus leaves a paper trail in the data hashes, whereas an agent's browsing session leaves a trail only if egress and tool results are logged.

Practical checks:

• Log every tool call's result hash. Without this you can't audit contamination after the fact at all.
• Restrict egress to an allowlist during eval-proposal rounds.
• Treat any agent-produced training example whose provenance chain includes a public benchmark page as suspect, and route it to manual review.

Selection Pressure on the Eval Set

Track per round: number of eval examples, per-example pass rate, and the set hash. If the set hash changes and the count drops, that's a signal worth a paragraph in any honest release note. If the set hash never changes, you can at least say the metric is stable — a much stronger statement than a rising score against a moving test.

Independent Re-Scoring Before Accepting a Number

Don't accept a published score. Re-score on your own task, and if you must compare against the publisher's number, measure grader agreement explicitly:

1. Take 200 examples from the publisher's stated eval distribution, or your own closest equivalent.
2. Run the released weights through your harness.
3. Score the same outputs with your grader and theirs.
4. Compute Cohen's kappa between the two. Below roughly 0.6, the graders disagree enough that the score difference between your run and theirs is mostly grader noise, and you should report your own number only.

This is unglamorous work, and it's the only thing that turns a benchmark table into information.

Post-Training Data Provenance for Agent-Generated SFT Data

Filtering Agent Output Into SFT Data

Agent trajectories are an attractive SFT corpus: long, structured, already in the target format. The filtering step is where the audit lives. Whatever the filter is, it needs to be deterministic and versioned — a hash of the filter code plus the thresholds plus rejection counts. If the filter is "a model decided," you have a second model in the training path whose own nondeterminism needs the same treatment.

Deduplication and Near-Duplicate Collapse

Token-level dedup and MinHash/SimHash near-dup collapse change the data distribution, not just its size. Aggressive thresholds remove exactly the rare, hard examples that carry most of the learning signal while keeping the median trajectory many times over. Publish the threshold, the pre- and post-dedup counts, and the hash of the resulting corpus. Without the counts, "we deduplicated" is unfalsifiable.

Decontamination Against Held-Out Sets

Decontamination requires the held-out set to have a hash, and that hash has to be published or shared with an independent evaluator. n-gram overlap against a hashed held-out set is the standard approach; the point is that a reviewer needs to be able to rerun it. If the held-out set isn't defined, "we decontaminated" means "we ran a filter we're not describing against a set you can't inspect."

Label Leakage From the Experiment Log

This is the failure mode I'd bet on for agent-built models, and it deserves its own paragraph.

An agentic loop produces a rich log: the eval questions it ran, the model's answers, the scores, and the agent's commentary on why an answer failed. That log is the most convenient training corpus in the building — already in chat format, already filtered for length, sitting right there. It's also the most dangerous one, because it contains the eval questions together with the correct answers and the reasoning about why the wrong ones were wrong.

If any part of the experiment log reaches the SFT mix, your subsequent eval numbers are measuring memorization of your own log. The defense is boring: physically separate the log store from the training data pipeline with different credentials, and assert in the pipeline that no artifact carrying a runId tag from the eval loop can enter the corpus. Then state in the model card that this separation exists.

That statement is missing from the announcement as reported, and it's the one question I'd ask Apex before any other.

What Applied ML Teams Should Borrow From This Audit

Log First, Optimize Second

You can't retrofit a transcript schema onto a loop that's already run 300 experiments. Build the manifest and the per-step transcript first, even crude, then let the agent run. The cost is a few hundred lines; the alternative is re-running everything when someone asks a question you can't answer.

Separate the Agent's Judgment From the Metric

Two roles, two contexts. The optimizer proposes; the scorer scores; neither sees the other's reasoning. When the metric is a scalar produced by a deterministic script, this is easy. When the metric is a model judgment, use a different model family and measure grader agreement before trusting any delta smaller than the disagreement rate.

Treat Tool Output as Versioned Input

Everything an agent reads from the outside world is an input, and inputs get hashed and stored. The cassette pattern from HTTP testing applies without modification. If you adopt one habit from this post, adopt this one — it turns an unreproducible pipeline into a slow but reproducible one.

Put Reproducibility Budgets in CI

A nightly job that replays 20 recorded steps and fails on prompt-hash drift will catch more real regressions than most eval suites, because it fails on the cause rather than the symptom. Pair it with a hard failure when the manifest is incomplete; a missing environment field is a defect, not a warning.

What Evidence Would Change This Assessment

I'm treating Apex-35B's process claim as unverified, not false. Four concrete artifacts would move it to verified, in descending order of usefulness:

1. One complete transcript for a representative run, with tool cassettes and per-step promptHash values. This alone would let an independent team replay a slice.
2. The eval harness and held-out set hashes, plus per-round eval set hashes so selection pressure is measurable rather than inferred.
3. Per-shard weight hashes and a checkpoint config that confirms the base model, the parameter counting convention, and the license.
4. A written statement that the experiment log was excluded from the post-training corpus, with the pipeline control that enforces it.

An independent reproduction by a team with no relationship to Apex would settle it more decisively than all four, and that's the outcome worth waiting for. What I wouldn't accept as evidence: a rising benchmark table against an undocumented harness, a model card without data hashes, or a demo comparing outputs side by side. None of those distinguish a better model from a better-tuned evaluation.

Conclusion

The mechanism here isn't controversial. Agents are good at the search-shaped parts of model development: proposing ablations, sweeping filters, reading results, iterating. That work is real, and it compresses timelines in ways that are easy to believe.

The audit problem is structural, and it doesn't go away by wishing. An agentic loop introduces nondeterminism at the sampling layer, the tool layer, the context-assembly layer, and the budget layer. The standard reproducibility triad — seed, commit, dataset version — covers only the first, and only partially. What actually makes an agent-built Qwen3.6 model auditable is a manifest that content-addresses every input, a transcript that records assembled-prompt hashes and tool cassettes, and an eval set the optimizer isn't allowed to touch.

Apex-35B-V1.0, as reported, gives us a base model, a parameter count, and a process claim. Whether the process was advisory, closed-loop, or generative changes the entire audit. Until the release publishes enough to tell, "built with AI research agents" is a marketing-shaped statement about an engineering process, and it should be filed accordingly — while the tooling above gets built regardless, because it pays off on your own LLM training pipeline whether or not anyone else ever publishes theirs.

Further Reading

• Pandaily report on the Apex-35B-V1.0 release — the seed report for this post
• Qwen3 official repository — base model family, configs, and licensing
• PyTorch reproducibility notes — nondeterministic algorithms and CUBLAS_WORKSPACE_CONFIG
• vLLM documentation — batching and scheduling behavior
• EleutherAI lm-evaluation-harness — a reference harness worth pinning by commit
• Hugging Face model card documentation — the minimum disclosure format
• Model Cards for Model Reporting (arXiv:1810.03993)
• Datasheets for Datasets (arXiv:1803.09010)
• Improving Reproducibility in Machine Learning Research (arXiv:2003.12206)
• Leakage and the Reproducibility Crisis in ML-based Science (arXiv:2207.07048)
• reproducible-builds.org — the content-addressing discipline this post borrows from