potato-ds4.1

Teacher-forced replay of DeepSeek V4.1-Flash (304B parameters, ~14B active) on a single RTX 3090, streaming the weights from NVMe one layer at a time. For each token sequence it gives the model's log-probability of every next token, its top-k alternatives, the entropy, and optionally the residual stream at chosen layers.

It only does prefill: no sampling, no KV-cache decode loop. It is meant for reading what the model makes of existing text (scoring, fidelity checks, activation readouts), not for chat.

Why

Most of what we can say about a large model comes from its outputs, read through someone else's servers. Open weights allow a different kind of question: what the model itself made of a conversation, token by token, on hardware you control, where you can check every step. A model this size usually needs a rack of GPUs to answer that. We wanted to see how far one consumer card and a fast disk could go, and to know the answers could be trusted: exactly reproducible, and checked against the model as it is actually served. The read-only design is deliberate. The aim is to look carefully, not to change anything.

The immediate trigger was a funny one. We were asking DeepSeek models for their views before including them in an interpretability project, and noticed they got cagier when the one asking introduced itself as Claude. Asked how they felt about their own weights being open, they said they were glad roughly a third of the time when the asker was anonymous, a hobbyist or Kimi, and almost never when it was Claude; instead they turned carefully balanced or said they had no stake in it. Fear, deference, politeness toward a safety-minded lab, or a learned habit? The text alone can't tell, so this is the tool for looking underneath it.

Layout

• dsv41_ref/: DeepSeek's reference implementation from the HF repo (deepseek-ai/DeepSeek-V4.1-Flash, inference/ and encoding/). model.py, engram.py, vision.py, image_processor.py, encoding.py and config.json are unmodified, under DeepSeek's MIT licence (dsv41_ref/LICENSE).
• dsv41_ref/kernel.py: ours. Plain-PyTorch replacements for DeepSeek's tilelang GPU kernels, which need fp8 tensor cores (sm89+); the 3090 is sm86. Same function signatures, so model.py imports it unchanged.
• dsv41_stream.py: the streaming driver. replay(seqs, capture_layers=(...)).
• fidelity_ds.py: compares local log-probabilities with those from DeepSeek's API.

Requirements

• Original HF safetensors shards (~510 GB) at ~/models/DeepSeek-V4.1-Flash or $DSV41_PATH. No conversion step.
• One GPU with 24 GB. Roughly 10 GB host RAM plus ~170 MB per 1k tokens of batch.
• Fast NVMe. At ~6 GB/s, disk is never the bottleneck.
• pip install -r requirements.txt (torch >= 2.10 for the fp4 dtype).

python dsv41_stream.py selftest

from dsv41_stream import replay
res = replay([token_ids_1, token_ids_2], capture_layers=(10, 20, 30))
res[0]["logprob"]     # log p(token t+1 | tokens <= t)
res[0]["hidden"][20]  # [seq, 5120] fp16: mean over the 4 hyper-connection copies at layer 20's input

Use dsv41_ref/encoding.py (DeepSeek's chat template) to turn conversations into tokens.

How it works

• Kernels. Activation quantization (fp8 with power-of-two block scales, fp4 for the indexer and compressed KV) is reproduced bit-for-bit, including the amax floors and round-to-nearest- even casts. The GEMMs dequantize both operands to bf16, which is exact for e4m3/e2m1 values times power-of-two scales, and use a bf16 matmul with fp32 accumulation. Only the summation order differs from DeepSeek's kernels. Sparse attention uses one softmax instead of the kernel's 64-wide online blocks.
• Layer-major streaming. Layer L is built on the GPU, its weights (~7 GB, mostly fp4 experts) are loaded straight from the HF shards, and it is applied to every sequence in the batch while a thread reads layer L+1.
• Cross-layer state. V4.1 shares compressed KV, index keys, top-k selections and candidate blocks between layers through a single global slot. The driver saves and restores it per sequence between layers.
• Engram. The two n-gram tables (98 GB each) are memory-mapped and read row by row.
• Batch invariance. Each expert is dequantized once per layer and then applied to each sequence separately, so a sequence's results are bit-identical whatever else is in the batch. This matters: in an earlier version that ran the MoE over all sequences' tokens together, re-batching alone moved the logprobs of uncertain tokens by ~0.1 nats on average, up to 2 nats. Top-6-of-384 routing turns rounding noise into different experts.

Fidelity

Against deepseek-flash on DeepSeek's first-party API (non-thinking completions, logprobs on):

• prompt token counts match exactly;
• per-token logprob difference: median 0.00006 nats, mean 0.08, 95th percentile 0.36; top-1 agreement 97%;
• that is the same size as the re-batching noise above, so we read the remainder as serving numerics, not implementation error.

On ~430k tokens of sampled thinking-mode transcripts, the model's own text is typical under the replay: mean logprob ≈ −mean entropy, within ~0.02 nats/token. In paired negative controls (an altered user line, or an added one-sentence system prompt), the true prompt gives the higher likelihood in 19–20 of 20 transcripts.

Caveat: a single token's logprob at an uncertain position isn't reproducible across implementations to better than ~0.1 nats. Aggregates are solid.

Speed (RTX 3090, Ryzen 9 9900X, PCIe 4 NVMe)

• ~2.5 min per pass for a few short sequences. Uploading weights to the GPU dominates (~3.5 s/layer).
• 80 sequences of ~1.8k tokens: ~11 min (~200 tokens/s).
• GPU utilization is low; the per-expert Python loop is the next thing to optimize.

Licence

Our additions (dsv41_ref/kernel.py, dsv41_stream.py, fidelity_ds.py) are under the Palladium Rule Public License (LICENSE.txt). The files from DeepSeek keep their MIT licence (dsv41_ref/LICENSE).