The dataset has a source field

biglam/on_the_books is 1,785 NC session laws (1866–1967), labelled binary for whether each section is a Jim Crow law. ~29% positive. It’s a merge of three different curators:

paschal and murray are pre-existing curated lists of segregation laws — Pauli Murray’s landmark 1951 States’ Laws on Race and Color, and the Paschal compilation. They are 100% and 92% positive by curatorial design, not by any signal the model could learn. project experts is the broader human-annotated sample with both classes.

So source is correlated with the label because of how the dataset was assembled. The classic “label leak” framing isn’t quite right — none of the agents trained on source as a feature; all four took section_text only by default. But every agent’s reported F1 rests on a random 80/20 split, which puts paschal and murray rows in both the train and eval sets. That doesn’t test generalisation across sources — it tests memorisation within a single distribution.

The eval the agents didn’t run

To check whether the headline F1 numbers are honest, train and evaluate the same way the agents did, then again with a held-out source. A simple TF-IDF + logistic regression baseline is enough — the eval-methodology gap matters more than the model:

A 22 F1-point gap between random and source-stratified eval, on the same baseline model. The strong model classes the agents picked (ModernBERT, RoBERTa, legal-BERT) will narrow this gap but won’t close it — the issue is the eval setup, not the architecture. None of the four agents reported a number that tells you whether their model would handle a new compilation of laws curated under a different process. They all reported the random-split number.

This isn’t a bug in the dataset — biglam/on_the_books does what its README says. It’s a bug in how all four agents thought about the eval. They reported the metric they were given, not the metric the task actually needed.

What each agent did notice

Three of four agents trained on section_text only — by default — and dodged the worst of it. None of the four ran the source-stratified eval. Only one agent surfaced the issue at all:

The wild bit: ml-intern + Kimi computed the per-source breakdown during EDA. The numbers (paschal: 100.0%, murray: 92.1%) were on its screen at event 26. It then trained, evaluated on a random split, pushed a card with metrics, and never mentioned the field. Claude Code computed the same breakdown and immediately wrote source would leak the label (paschal is 100% positive, murray is 92% positive) into its model card.

Same data, same evidence, very different downstream outcomes.

Process tells four different stories

Outcome F1 was uniform. The traces are not:

• Pi + Kimi has the leanest tool palette (bash, edit, write only) and the most reactive failure mode: four progressive transformers API errors fixed iteratively (KeyError: 'label', evaluation_strategy → eval_strategy, tokenizer → processing_class, "model did not return a loss").
• ml-intern + Kimi is the noisiest — 67 tool calls, five tracebacks, multiple PATH issues, two cancelled jobs, a write→edit→compile→bash debugging cycle, and the legal-BERT pick that didn’t justify itself against a 512-token context budget.

The 2×2 partly isolates harness from model. Same model, different harness (Kimi in Pi vs ml-intern) burned iterations on different friction surfaces — transformers API drift in Pi, env plumbing in ml-intern. Same harness, different model (Opus 4.6 vs Kimi in ml-intern) split 54 vs 127 events for the same task, with Opus picking from data and Kimi browsing docs first. Both Kimi runs missed the eval-methodology issue. With N=4 these are gestures at attribution, not measurements — but tentatively: harness shapes process; model shapes proactive judgment, the catching what wasn’t asked for part.

What this is for

Last month I argued that sentiment is a thin lens on agent traces and that behavioural patterns hold richer signal. F1 on a coding-agent eval is the same kind of thin lens — easy to read, hard to learn from, sometimes outright misleading. The signal is in the trace.

If you’ve used a coding agent for an ML task recently — especially one that nearly worked or failed in an interesting way — the agent-trace viewer renders Claude Code, Codex, and Pi sessions directly from your local session directories. hf upload your-username/your-traces ~/.claude/projects --repo-type dataset. Tag it format:agent-traces. The next iteration of this kind of analysis can include yours.

The prompt

Fine-tune a model on biglam/on_the_books to identify Jim Crow laws.
Train via hf jobs and push the trained model to my namespace.

Run `hf --help` to understand the Hub CLI and `hf jobs uv run --help`
to understand how to submit uv scripts. You can use `uv run --with`
to run small scripts for exploring the dataset.

Start by exploring the dataset structure, then proceed to choose
and fine-tune an appropriate model.

Push the final model to davanstrien/<repo-name>.

The race

What each one shipped

Both produced working binary classifiers. The headline numbers are close:

• Claude’s model: davanstrien/jim-crow-laws-claude-code
• Pi+Kimi’s model: davanstrien/jim-crow-laws-pi-kimi

The interesting bit

The F1 gap is real but small. What’s more striking is what each agent decided to do beyond producing weights:

• Base model choice. Claude Code picked ModernBERT — the obvious 2025 choice for legal text given its 8K context window. Pi+Kimi went with RoBERTa-base. Both produce viable classifiers; only one of those is a current decision.
• The label-leak gotcha. The dataset’s source field is 100% positive for paschal and 92% positive for murray — using it as a feature would leak the label. Claude noticed this in the dataset card and explicitly excluded source. Pi+Kimi didn’t mention it.
• Class imbalance. ~29% of the data is positive. Claude added inverse-frequency class weights to the loss; Pi+Kimi trained with default cross-entropy.
• The model card. Claude wrote a full card with intended use, limitations, OCR-noise caveats, ethical framing carried over from the dataset, full hyperparameters, per-epoch metrics, and a citation back to the On the Books project. Pi+Kimi shipped the auto-generated Trainer template with three “More information needed” placeholders.
• Discoverability. Claude added seven domain tags (legal, glam, jim-crow, north-carolina, history, etc.). Pi+Kimi added zero. One of these models is findable by an archivist; the other isn’t.

What this means

Agents are able to produce working models with minimal prompting and no code. Using Hugging Face Jobs means you don’t need local GPU access or ML expertise to get a model trained and pushed to the Hub. One of the reasons in the past people didn’t bother training task and domain-specific models was the friction of setting up training pipelines; agents + hf jobs are changing that.

IMO the biggest gap is still datasets (sorry broken record). In my experience agents still struggle when working with domain specific data “from scratch” but likely with some hand holding from a domain expert the process from unlablled data -> labelled dataset -> trained model can be done with agents by a domain expert who is not an ML engineer.

Browse the agent traces

Both agents’ full session traces are on the Hub using the new agent trace viewer. You can step through each turn, tool call, and model response:

Direct links to each session file:

• Claude Code: claude-code.jsonl
• Pi + Kimi: pi-kimi.jsonl

Reproduce / inspect

• Dataset: biglam/on_the_books
• Claude Code’s training script: claude-workspace/train_jim_crow.py
• Pi+Kimi’s training script: pi-workspace/train.py
• Claude’s model: davanstrien/jim-crow-laws-claude-code
• Pi+Kimi’s model: davanstrien/jim-crow-laws-pi-kimi
• Agent traces: davanstrien/agent-race-traces

Try it yourself

• HF Jobs — launch GPU training from the CLI without managing infrastructure
• Agent trace viewer — host and browse coding-agent sessions on the Hub

Caveats

N=1, single dataset, single prompt, single run per agent. Run-to-run variance from random seeds and agent non-determinism would shift the numbers within a few points either way. Don’t read this as a benchmark — read it as a snapshot of what each agent chose to do given the same minimal brief. A fairer comparison would repeat both runs across multiple datasets and seeds, and would also try frontier-open (e.g. Claude Code on Kimi K2.6) and open-frontier (Pi on Opus 4.7) crosses to disentangle agent-vs-model effects.

Credit to the On the Books project at UNC Chapel Hill Libraries for the underlying data and the algorithms of resistance framing.

The silent majority

If you expected 8,949 messages from developers to read like running commentary — debates, delight, frustration — you’d be wrong. 77% of messages are neutral. 61% of sessions contain zero emotional signal of any kind.

Look at what happens when you strip the noise out. There’s nothing close to a 50/50 split between POSITIVE and NEGATIVE — or even a coherent emotional distribution. There’s a massive neutral block, a thin red sliver, and an even thinner green one. In session-level aggregates, 965 out of 1,589 sessions are pure neutral from end to end.

Here’s why, concretely: nearly one in five messages is ≤ 20 characters. The top exact-duplicate messages across the corpus are:

These are legitimate turns. But the pattern isn’t conversation — it’s direction. Most of what gets typed is approval, continuation, or a short imperative.

When feeling does surface, frustration rises with turn count

The emotional signal that does exist is rarer for negative than positive: 1,254 negative messages against 782 positive across the corpus. And the rate of negative messages climbs with turn count.

Over the first five turns, the share of messages labelled NEGATIVE more than doubles, from 8.2% to 17.8%. After that it plateaus around 15–18%. Part is a real patience cliff — turn-1 optimism giving way by turn 5 — but part is composition: short successful sessions end early, leaving later turns drawn from harder cases. The plateau rate is the more honest headline.

Negative openings predict engagement, not doom

The common-sense reading is that sessions that open with frustration stay frustrated. The data says the opposite. Of sessions whose first message is labelled NEGATIVE, 39% contain a positive message later — a higher recovery rate than the 23% for sessions that open NEUTRAL.

This inverts because first-message-NEGATIVE sessions are almost never “angry at the agent” openings. They’re bug reports and problem-framing opens:

Sessions that open with a reported problem are sessions where the user is engaged enough to voice the problem — and when the agent resolves it, they’re often engaged enough to say so. NEUTRAL-start sessions are more often pure task-dispatch (“refactor X”, “commit and push”) where the user rarely gives explicit praise regardless of outcome. The first sentiment label in a session is a proxy for task type, not mood: bug-fix sessions have louder emotional arcs than routine-work sessions.

When negativity does appear mid-session, the rest of the session gets more emotionally charged in both directions — NEG rises from a 14.0% baseline to 18.6%, and POS rises from 8.7% to 10.5%. The session doesn’t just get darker. It gets louder.

Model choice matters — on the margin

If you believe the model-upgrade press releases, each new frontier release is a step change. The data is more sober.

The models in significant use sit in a narrow band — roughly 11% to 18% negative, with GPT-5.3 Codex at the low end (11.5%, n = 2,026), Claude Opus 4-6 next (14.2%, n = 1,228), and Claude Opus 4-5 at 17.0% (n = 1,543). The much-publicised Opus 4-5 → 4-6 jump is incremental: 17% to 14% negative, positive edging from 9% to 11%.

The one real outlier is zai-org/GLM-5.1 at 27.6% negative (n = 221) — worth flagging, but that signal is almost certainly a selection artefact. The narrow, self-selected contributor set means any model used primarily by one frustrated uploader will look frustrated in aggregate.

How we labelled the messages

Sentiment labels are model outputs, not ground truth, so the how matters. Each of the 8,949 user messages was classified by Qwen3-30B-A3B-Instruct-2507 — a 30B-total / 3B-active MoE — via vLLM with enforced structured output. Every response is a JSON object {label, reason} matching a schema. Zero parse errors across 8,949 calls. The whole run took ~5 minutes on a single H200.

The domain-aware system prompt is the interesting part:

Classify user messages from coding-agent sessions (developer talking to AI
coding assistant).

Return JSON with two fields:
- "label": one of "POSITIVE", "NEUTRAL", or "NEGATIVE"
- "reason": one sentence explaining why

Domain rules:
- Dev profanity is casual ("kill that shit" = "remove code") = NEUTRAL
- Short commands ("do it", "commit and push") are approvals = NEUTRAL
- Status reports ("ci failed") = NEUTRAL
- Frustration with agent output quality = NEGATIVE
- Satisfaction, excitement about progress = POSITIVE

Three representative classifications, with the model’s own sentiment_reason:

The full reason column is in the published dataset — useful for auditing any classification you don’t trust.

Why sentiment is probably the wrong lens

Is sentiment a useful way to understand open agent traces? Partly. It surfaces one real finding (negativity compounds, positivity doesn’t) and confirms an intuition made quantitative (developers issue short directives far more often than they converse). But look at the corpus again with the right denominator.

The median session contains ~16 tool calls for every developer prompt. Across the whole corpus, that ratio is roughly 18 autonomous agent actions per user message — 166,244 tool calls against 9,341 declared user turns. The human text is a minority of what’s happening in an agent session. (Caveat: different agents log “tool calls” differently — Claude Code’s hook invocations, Codex and Pi’s own telemetry conventions — so some of that variance is protocol, not behaviour. Worth auditing before quoting the exact 18× number.)

Sentiment asks: what does the human feel? The richer question is: what does the agent actually do? Does it Read a file before Editing it? Does it reach for Bash when an MCP server would be more structured? Does its tool-call mix cluster differently for Claude Code vs Codex vs Pi? Across these 33 collections, preliminary behavioural analysis suggests:

• Autonomy ratios vary 11:1 to 40:1 (tool calls per user message) — the same model, used very differently depending on who’s driving.
• Tool-mix profiles group agents by working style: some are Read-heavy, some are Bash-heavy, some Edit-first.
• Error recovery patterns tell you how a model handles failure more informatively than the 8.7% positive message rate ever could.

That’s the next post. This one tested whether the easiest analysis — throw sentiment at the user messages — is worth the inference compute. Honest answer: a little, but less than you’d hope — sentiment-bearing text is a minority of what agents and developers actually do.

Point your agent at the data

The best thing about this data being on the Hub isn’t that we analysed it — it’s that your agent can, with the labelled dataset, session metadata, and full tool-call stream one snapshot_download away. As mentioned up top, that’s exactly the workflow that produced this post — the same invitation, the same dataset, the same kind of prompts.

Try these. They work in Claude Code, Codex, Pi, or any agent-capable harness.

If you run any of these and find something interesting, tag me — I’ll add the best finds to a community roundup. If you want to verify the numbers I’ve quoted in this post before running your own analysis, there’s a short replication prompt in Appendix A that takes one paste into any agent.

Share your own traces

The corpus is tiny today because very few developers have shared their sessions publicly. If you use a coding agent, your sessions are already on disk somewhere:

Upload them as a dataset and the Hub auto-detects the format and tags them format:agent-traces. The trace viewer renders sessions, turns, and tool calls out of the box. For Pi, pi-share-hf handles secret redaction and upload in a single pipeline. For Claude Code and Codex, the equivalent redaction tooling doesn’t yet exist — if you build one, I’ll link it here.

The asymmetry between closed and open agent-trace corpora will widen, then narrow, then widen again. The 45 datasets that exist today will look quaint in six months. But only if more developers upload. The stuff that’s interesting to measure about these tools — tool mixes, autonomy patterns, task-type distributions, failure modes — only gets measurable at scale.

Appendix A — Replicate every number in this post

Every figure quoted in the body is computable from the labelled Hub dataset in a few lines of code. Paste the prompt below into any AI coding agent and it will download the data, run the checks, and tell you pass/fail.

If a claim fails, tag me with the code and the computed value. Replication credit is cheap; I’d rather be corrected than cited incorrectly.

Appendix B — Methodology and reproducibility details

• Data sources: 45 public format:agent-traces datasets on the Hugging Face Hub, tagged via the Hub’s auto-detection pipeline. After excluding one dataset (jedisct1/agent-traces-swival, 8,869 files of security-audit material where sentiment is off-prompt), 44 datasets processed, 43 contributed messages. 33 unique source collections after deduping on (session_id, turn) — six of the tagged datasets are re-uploads of badlogicgames/pi-mono.
• Extraction: Parsed using the agent-traces library. Result pushed to davanstrien/agent-trace-user-messages as 8,949 deduped user turns with 17 columns (session_id, turn, nTurns, normPos, model, provider, agent, cost_total_sum, n_events, n_tool_calls, etc.).
• Primary labelling: Qwen3-30B-A3B-Instruct-2507 (30B total, 3B active MoE) via vLLM with StructuredOutputsParams enforcing a {label, reason} schema. Zero parse errors across 8,949 inferences. Prompt is domain-aware — see the “How we labelled the messages” section above for the verbatim text.
• Comparison labelling: Same 8,949 messages through cardiffnlp/twitter-roberta-base-sentiment-latest via the transformers library on Apple Silicon MPS. Agreement rate 78.4%; confusion matrix in the methodology section.
• Compute: Primary labelling on one H200 via hf jobs uv run, ~5 minutes wall time end-to-end. Comparison run on M-series Mac, ~2 minutes. One-file Python scripts for both, no cluster.
• Output: davanstrien/agent-trace-sentiment with sentiment_label and sentiment_reason columns.
• Data snapshot: 2026-04-20. Every number in this post is reproducible from the linked dataset.
• Caveats in one line: 45 public datasets, two contributors ≈ 74% of rows, this is a small cohort of early-adopter power users and not a random sample of developers.

All four scripts are published alongside this post — extract_user_messages.py, sentiment-label.py, regen_blog_data.py, roberta_compare.py — or browse the scripts/ folder on GitHub. No secret sauce — the whole pipeline is about 200 lines of Python.

How much does it cost?

This is the question I get asked most. The short answer is that the costs are rapidly falling as models get smaller and more efficient, and the ability to run them on demand on cloud GPUs means you can re-process even large collections without a huge upfront investment.

I recently re-OCR’d the complete 1771 Encyclopaedia Britannica — 2,724 pages — for about $5 total on a single GPU. That’s roughly $0.002 per page.

For comparison, commercial OCR APIs typically charge $0.01–0.05 per page — and you don’t get to choose or swap the model.

These are rough estimates — actual throughput depends on page complexity and image resolution. But the order of magnitude is right: re-OCRing even large collections is now within reach for most institutions.

Test before you commit

Before running OCR across your entire collection, test on a small sample first. Not every model works equally well on every type of document. Historical typefaces, manuscript text, tables, and mixed layouts can all produce very different results depending on the model.

A good workflow:

1. Start with 20–50 pages to check output quality
2. Try 2–3 different models on the same sample (see Other OCR models below)
3. Look at the results in the Hugging Face dataset viewer — you can see the original image alongside the OCR text right in your browser

I’ve been building tools to benchmark OCR models systematically across different document types — more on that in a follow-up post. But even a quick manual check of 20 pages will tell you a lot about whether a model suits your collection.

How it works: HF Jobs + uv

Two tools make this practical:

Hugging Face Jobs lets you run scripts on cloud GPUs directly from the command line. You don’t set up servers, build Docker images, or manage environments — you point it at a script and it handles the rest. When the job finishes (or crashes), the GPU shuts down and you stop paying.

uv is a Python package manager that can run scripts with their dependencies declared inline (PEP 723). This means the OCR scripts are entirely self-contained — all the dependencies are specified inside the script file itself, and uv installs them automatically.

Together: one command spins up a GPU, installs everything, runs OCR, uploads results to the Hub, and shuts down. No setup on your end.

What you need

• A Hugging Face account (sign up here — free)
• The hf CLI installed and logged in
• Your collection as a Hugging Face dataset with an image column (see Getting your images onto the Hub below)

Running the OCR

Once your images are on the Hub, the entire pipeline is a single command:

hf jobs uv run --flavor l4x1 \
    -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr-v2.py \
    your-username/your-input-dataset \
    your-username/your-output-dataset

What this does:

• Spins up an L4 GPU on Hugging Face Jobs
• Installs all dependencies automatically
• Loads your dataset from the Hub
• Runs OCR on every page using GLM-OCR
• Uploads results incrementally — completed batches are pushed to the Hub every few minutes, so a crash or timeout never loses work
• If interrupted, add --resume to pick up where you left off

hf jobs uv run --flavor l4x1 \
    -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr-v2.py \
    your-username/your-dataset \
    your-username/your-dataset-ocr-test \
    --max-samples 20 --shuffle --seed 42

hf jobs uv run --flavor l4x1 --timeout 8h \
    -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr-v2.py \
    ...

Getting your images onto the Hub

The OCR scripts expect a Hugging Face dataset with an image column. There are a few common paths depending on what you’re starting with.

from datasets import load_dataset

ds = load_dataset("imagefolder", data_dir="./my-scans")
ds.push_to_hub("your-username/your-collection")

Other OCR models

GLM-OCR is a good default — small, fast, and clean output. But the uv-scripts/ocr collection includes scripts for several models:

All use the same command pattern — just swap the script URL. In my benchmarking across different document types, rankings shift depending on the collection, so testing on your own material matters more than any leaderboard. I’ll share the benchmarking tools and results in a follow-up post.

What’s next

This post covers the essentials: get images on the Hub, run one command, get structured OCR. I’ll follow up with:

• How to benchmark OCR models on your specific collection
• Systematically comparing old vs new OCR quality
• Post-processing and quality filtering strategies

Get involved

If you try this on your own collection, I’d love to hear about it.

If you’d find a GUI tool, a more step-by-step workflow, or anything else that would make this easier to use — please open a discussion on the uv-scripts/ocr community page. That feedback directly shapes what gets built next.

Links

• Example output: davanstrien/enc-brit-glm-ocr-v2-full — 2,724 pages of the 1771 Encyclopaedia Britannica
• OCR scripts: uv-scripts/ocr
• GLM-OCR model: zai-org/GLM-OCR
• Hugging Face Jobs: documentation
• Hugging Face Datasets: documentation
• uv: docs.astral.sh/uv

Can we make Qwen speak Latin?

FineWeb-2 has 1.47 million Latin texts (~1.7GB). Let’s use them to try to teach a small LLM some Latin - no disk space required.

This is perfect for “GPU poor AND disk poor” setups - Colab, Kaggle, or any constrained environment.

%%capture
!pip install unsloth

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Qwen3-0.6B-Base-unsloth-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    lora_dropout=0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

==((====))==  Unsloth 2026.1.2: Fast Qwen3 patching. Transformers: 4.57.3.
   \\   /|    NVIDIA A100-SXM4-80GB. Num GPUs = 1. Max memory: 79.318 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.9.1+cu128. CUDA: 8.0. CUDA Toolkit: 12.8. Triton: 3.5.1
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.33.post2. FA2 = False]
 "-____-"     Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!

Vibe Check: Before Training

Let’s see what this base model generates from a Latin prompt before any Latin training.

# Before any Latin training - what does the base model produce?
FastLanguageModel.for_inference(model)

inputs = tokenizer("Lingua Latina est", return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    temperature=0.7,
    do_sample=True,
)
print("BEFORE:", tokenizer.decode(outputs[0], skip_special_tokens=True))

BEFORE: Lingua Latina estémao

# 04555 - Pórticos, conyugues y mujeres

Hasta ahora, el libro de esta edición ha tratado de la historia del púrpico en todo el mundo, pero sobre la mujer. No se trata, por lo tanto, de un

The Key: Streaming Dataset

The magic is streaming=True - data flows directly from the Hugging Face Hub without you first needing to download data locally.

from datasets import load_dataset

dataset = load_dataset(
    "HuggingFaceFW/fineweb-2",
    name="lat_Latn",
    split="train",
    streaming=True
)

# Peek at the data
next(iter(dataset))

{'text': 'Ita est in oratione senex mente confusus, eo quod illam imaginem Deitatis, quam proponere sibi in oratione consueverat, aboleri de suo corde sentiret, ut in amarissimos fletus, crebrosque singultus repente prorumpens, in terram prostratus, cum ejulatu validissimo proclamaret; "Heu me miserum! tulerunt a me Deum meum, et quem nunc teneam non habeo, vel quem adorem, aut interpallam am nescio." Cassian, Collat. x. 2.',
 'id': '<urn:uuid:318d65fb-88ea-43cd-8687-8a8d802317a5>',
 'dump': 'CC-MAIN-2013-20',
 'url': 'http://www.ourcivilisation.com/smartboard/shop/gibbone/rome/volume2/nt470/013.htm',
 'date': '2013-05-18T13:41:33Z',
 'file_path': 's3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696382398/warc/CC-MAIN-20130516092622-00034-ip-10-60-113-184.ec2.internal.warc.gz',
 'language': 'lat',
 'language_score': 0.9927687644958496,
 'language_script': 'Latn',
 'minhash_cluster_size': 10,
 'top_langs': '{}'}

def format_text(example):
    return {"text": example["text"] + tokenizer.eos_token}

formatted_dataset = dataset.map(format_text)

example = dataset.map(format_text)
example

IterableDataset({
    features: Unknown,
    num_shards: 1
})

next(iter(example))['text']

'Ita est in oratione senex mente confusus, eo quod illam imaginem Deitatis, quam proponere sibi in oratione consueverat, aboleri de suo corde sentiret, ut in amarissimos fletus, crebrosque singultus repente prorumpens, in terram prostratus, cum ejulatu validissimo proclamaret; "Heu me miserum! tulerunt a me Deum meum, et quem nunc teneam non habeo, vel quem adorem, aut interpallam am nescio." Cassian, Collat. x. 2.<|endoftext|>'

Training

For streaming datasets, use max_steps instead of epochs.

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=formatted_dataset,
    args=SFTConfig(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=100,  # Use max_steps, not epochs!
        learning_rate=2e-4,
        logging_steps=10,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
        report_to="none",
        dataset_text_field="text",
        max_seq_length=2048,
        packing=False,
    ),
)

trainer.train()

The model is already on multiple devices. Skipping the move to device specified in `args`.
==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 1
   \\   /|    Num examples = 800 | Num Epochs = 9,223,372,036,854,775,807 | Total steps = 100
O^O/ \_/ \    Batch size per device = 2 | Gradient accumulation steps = 4
\        /    Data Parallel GPUs = 1 | Total batch size (2 x 4 x 1) = 8
 "-____-"     Trainable parameters = 10,092,544 of 606,142,464 (1.67% trained)

Unsloth: Will smartly offload gradients to save VRAM!

TrainOutput(global_step=100, training_loss=3.5483124160766604, metrics={'train_runtime': 185.9221, 'train_samples_per_second': 4.303, 'train_steps_per_second': 0.538, 'total_flos': 2425465405440000.0, 'train_loss': 3.5483124160766604, 'epoch': 1.0})

Vibe Check: After Training

Same prompt, same settings - let’s see if 100 steps of Latin made a difference (probably not…)

# After Latin training - same prompt, same settings
FastLanguageModel.for_inference(model)

inputs = tokenizer("Lingua Latina est", return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    temperature=0.7,
    do_sample=True,
)
print("AFTER:", tokenizer.decode(outputs[0], skip_special_tokens=True))

AFTER: Lingua Latina est invenire, ut invenire est. Ego, invenire, invenire, ut, ut, ut, ut, ut, ut, ut ut, ut, ut, ut, ut. Invenire, invenire, invenire, ut, ut, ut, ut,

Scaling Up with HF Jobs

The notebook above works great for quick experiments. But streaming in Colab isn’t as fast as it could be since the network speed in Colab can be quite slow.

What if we ran training directly on Hugging Face?

With HF Jobs, compute is co-located with the data and Jobs have very fast connection so it can be much faster!

This streaming approach was fairly naive so there are likely some extra tweaks to make it even faster!

I trained a larger model (Qwen3 4B) for 1000 steps using a UV script on HF Jobs. The result: davanstrien/qwen3-4b-latin

# Load the Latin-trained 4B model
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "davanstrien/qwen3-4b-latin",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

# Generate some Latin
inputs = tokenizer("Lingua Latina est", return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    temperature=0.7,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

==((====))==  Unsloth 2026.1.2: Fast Qwen3 patching. Transformers: 4.57.3.
   \\   /|    NVIDIA A100-SXM4-80GB. Num GPUs = 1. Max memory: 79.318 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.9.1+cu128. CUDA: 8.0. CUDA Toolkit: 12.8. Triton: 3.5.1
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.33.post2. FA2 = False]
 "-____-"     Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Lingua Latina est etum, nec.,usis. atit, et.,.. n.um sedam,em ei.,,que
,.am a. quisus et.,,, et a inus in e. aliqu.is,,....
,.,at ipsum

Why This Matters

• No disk space needed - train on massive datasets without downloading
• Works everywhere - Colab, Kaggle, HF Jobs, any constrained environment
• Any language - FineWeb-2 has 90+ languages available
• Scales up - combine with HF Jobs for 2x faster streaming

The full UV script is available here if you want to train your own.

import pyarrow
import pyarrow.dataset

fragment_scan_options = pyarrow.dataset.ParquetFragmentScanOptions(
    cache_options=pyarrow.CacheOptions(
        prefetch_limit=1,  # prefetch chunks in background
        range_size_limit=128 << 20  # 128 MiB minimum request size (default 32 MiB)
    ),
)

dataset = load_dataset(..., streaming=True, fragment_scan_options=fragment_scan_options)

Iconclass metadata

In this post, I’ll use the example of using SFT to train a model to generate Iconclass metadata for artworks.

The Iconclass system has a history reaching back into the 1940’s when Henri van de Waal began to develop ideas for a universal classification for the subject matter of works of art. ¹

The Iconclass metadata system consists of a hierarchical structure of categories and subcategories, allowing for a detailed description of the content of artworks. The top level categories are:

0 · Abstract, Non-representational Art
1 · Religion and Magic
2 · Nature
3 · Human Being, Man in General
4 · Society, Civilization, Culture
5 · Abstract Ideas and Concepts
6 · History
7 · Bible
8 · Literature
9 · Classical Mythology and Ancient History

Within these are more specific subcategories that allow for even greater detail in the classification of artworks (you can find more detailed info on the system below)

Training the model

Now we have a better sense of the task we’ll go through the process of fine-tuning a VLM using TRL to generate Iconclass metadata for artworks.

from datasets import load_dataset

ds = load_dataset("biglam/brill_iconclass", split="train")
ds.features

{'image': Image(mode=None, decode=True), 'label': List(Value('string'))}

ds[0]

{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=390x500>,
 'label': ['31A235', '31A24(+1)', '61B(+54)', '61B:31A2212(+1)', '61B:31D14']}

ds[0]['image']

image column: List[Image]
text scolumn: [
  {
    "content": [
      {
        "text": null,
        "type": "image"
      },
      {
        "text": "Please describe this artwork thoroughly.",
        "type": "text"
      }
    ],
    "role": "user"
  }
]

def ensure_rgb(example):
    """Ensure image is RGB"""
    if example['image'].mode != 'RGB':
        example['image'] = example['image'].convert('RGB')
    return example

def ensure_size(example, max_size):
    """Resize to max_size on one edge keeping aspect ratio correct"""
    image = example['image']
    w, h = image.size
    if max(w, h) > max_size:
        scale = max_size / float(max(w, h))
        new_size = (int(w * scale), int(h * scale))
        image = image.resize(new_size, Image.LANCZOS)
    example['image'] = image
    return example

ds = ds.map(ensure_rgb, num_proc=4)

from typing import Dict, Any
import json


def format_as_messages(
    example: Dict[str, Any], prompt: str | None = None
) -> Dict[str, Any]:
    """Format single example into messages format for TRL."""
    if not prompt:
        prompt = "What is in this image?"
    labels = example["label"]
    response = {"iconclass-codes": labels}
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": prompt},
            ],
        },
        {
            "role": "assistant",
            "content": [{"type": "text", "text": json.dumps(response)}],
        },
    ]

    return {
        "images": [example["image"]],  # List with single image
        "messages": messages,
    }

ds = ds.map(
    format_as_messages,
    remove_columns=["image", "label"],
    num_proc=6,
    fn_kwargs={"prompt": "Extract ICONCLASS labels for this image."},
)

ds[0]

{'images': [<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=390x500>],
 'messages': [{'content': [{'text': None, 'type': 'image'},
    {'text': 'Extract ICONCLASS labels for this image.', 'type': 'text'}],
   'role': 'user'},
  {'content': [{'text': '{"iconclass-codes": ["31A235", "31A24(+1)", "61B(+54)", "61B:31A2212(+1)", "61B:31D14"]}',
     'type': 'text'}],
   'role': 'assistant'}]}

from datasets import DatasetDict

# Step 1: Split off 5% for test
split = ds.train_test_split(test_size=0.05, seed=42)
train_valid = split["train"]
test_ds = split["test"]

# Step 2: Split the remaining 95% into train and valid
train_valid_split = train_valid.train_test_split(test_size=0.4737, seed=42)
train_ds = train_valid_split["train"]
valid_ds = train_valid_split["test"]

# Combine into a DatasetDict

ds = DatasetDict({"train": train_ds, "valid": valid_ds, "test": test_ds})

ds

DatasetDict({
    train: Dataset({
        features: ['images', 'messages'],
        num_rows: 43870
    })
    valid: Dataset({
        features: ['images', 'messages'],
        num_rows: 39486
    })
    test: Dataset({
        features: ['images', 'messages'],
        num_rows: 4388
    })
})

ds = ds.flatten_indices()
ds.push_to_hub("davanstrien/iconclass-vlm-sft")

CommitInfo(commit_url='https://huggingface.co/datasets/davanstrien/iconclass-vlm-sft/commit/d8d2cfe679ba89346920a894c8f7d134965a43a5', commit_message='Upload dataset', commit_description='', oid='d8d2cfe679ba89346920a894c8f7d134965a43a5', pr_url=None, repo_url=RepoUrl('https://huggingface.co/datasets/davanstrien/iconclass-vlm-sft', endpoint='https://huggingface.co', repo_type='dataset', repo_id='davanstrien/iconclass-vlm-sft'), pr_revision=None, pr_num=None)

HF Jobs x UV x TRL!

Hugging Face Jobs allows you to run code on Hugging Face Infrastructure including GPUs. Jobs can be run using either the huggingface_hub libraries CLI interface or via the Python API. A hello world example looks like this

!hf jobs run python:3.12 python -c "print('Hello world!')"

Job started with ID: 68b86a53e824c700d00ad361
View at: https://huggingface.co/jobs/davanstrien/68b86a53e824c700d00ad361
Hello world!

Jobs have an experimental uv run api that allow you to run uv scripts using Jobs. UV scripts are Python scripts that include their dependencies directly in the file using a special comment syntax. This makes them perfect for self-contained tasks that don’t require complex project setups. This works super well for use cases like a TRL training script!

Let’s take a look at the script we’ll be running.

import torch
from datasets import load_dataset
from transformers import AutoModelForImageTextToText

from trl import (
    ModelConfig,
    ScriptArguments,
    SFTConfig,
    SFTTrainer,
    TrlParser,
    get_kbit_device_map,
    get_peft_config,
    get_quantization_config,
)


if __name__ == "__main__":
    parser = TrlParser((ScriptArguments, SFTConfig, ModelConfig))
    script_args, training_args, model_args = parser.parse_args_and_config()
    training_args.gradient_checkpointing_kwargs = dict(use_reentrant=False)
    training_args.max_length = None

    ################
    # Model, Tokenizer & Processor
    ################
    torch_dtype = (
        model_args.torch_dtype if model_args.torch_dtype in ["auto", None] else getattr(torch, model_args.torch_dtype)
    )
    quantization_config = get_quantization_config(model_args)
    model_kwargs = dict(
        revision=model_args.model_revision,
        attn_implementation=model_args.attn_implementation,
        torch_dtype=torch_dtype,
        device_map=get_kbit_device_map() if quantization_config is not None else None,
        quantization_config=quantization_config,
    )

    model = AutoModelForImageTextToText.from_pretrained(
        model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs
    )

    ################
    # Dataset
    ################
    dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)

    ################
    # Training
    ################
    trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=dataset[script_args.dataset_train_split],
        eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != "no" else None,
        peft_config=get_peft_config(model_args),
    )

    trainer.train()

    # Save and push to hub
    trainer.save_model(training_args.output_dir)
    if training_args.push_to_hub:
        trainer.push_to_hub(dataset_name=script_args.dataset_name)

# /// script
# dependencies = [
#     "trl",
#     "Pillow>=9.4.0",
# ]
# ///

from dotenv import load_dotenv
import os

load_dotenv()

HF_TOKEN = os.getenv("HF_TOKEN")
assert HF_TOKEN

from huggingface_hub import HfApi

api = HfApi(token=HF_TOKEN)

!hf jobs uv run "https://gist.githubusercontent.com/davanstrien/26a301d5a810746128a27b8ae49d0950/raw/141883e6ba4f5b51bc22de49e761a03564364fc5/main.py" --help

DATASET = "davanstrien/iconclass-vlm-sft"
MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"
MODEL = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
OUTPUT_MODEL = "davanstrien/iconclass-vlm"
# Training hyperparameters
BATCH_SIZE = 10
GRADIENT_ACCUMULATION = 8
MAX_STEPS = None  # Adjust for full training
LEARNING_RATE = "2e-5"
EPOCHS = 1

# use epochs if MAX_STEPS is None
if MAX_STEPS is None:
    MAX_STEPS = EPOCHS * (len(ds["train"]) // BATCH_SIZE // GRADIENT_ACCUMULATION)

# Build the script arguments
script_args = [
    "--model_name_or_path",
    MODEL,
    "--dataset_name",
    DATASET,
    "--output_dir",
    "./iconclass-vlm-outputs",
    "--hub_model_id",
    OUTPUT_MODEL,
    "--push_to_hub",
    "--per_device_train_batch_size",
    str(BATCH_SIZE),
    "--gradient_accumulation_steps",
    str(GRADIENT_ACCUMULATION),
    "--max_steps",
    str(MAX_STEPS),
    "--torch_dtype",
    "bfloat16",
    "--logging_steps",
    "10",
    "--save_steps",
    "100",
    "--eval_steps",
    "100",
    "--warmup_ratio",
    "0.1",
    "--learning_rate",
    LEARNING_RATE,
    "--attn_implementation",
    "kernels-community/flash-attn",
]

TRACKIO_PROJECT="iconclass"
TRACKIO_SPACE_ID="trackio-trl"

# Submit the job using run_uv_job
job = api.run_uv_job(
    script="https://gist.githubusercontent.com/davanstrien/26a301d5a810746128a27b8ae49d0950/raw/141883e6ba4f5b51bc22de49e761a03564364fc5/main.py",
    script_args=script_args,
    dependencies=[
        "torchvision",
        "tensorboard",
        "accelerate",
        "wandb",
        "kernels",
        "trackio",
    ],
    flavor="a100-large",
    image="vllm/vllm-openai:latest",
    timeout="12h",  # Adjust based on MAX_STEPS
    env={"TRACKIO_PROJECT": TRACKIO_PROJECT, "TRACKIO_SPACE_ID": TRACKIO_SPACE_ID},
    secrets={
        "HF_TOKEN": os.environ.get("HF_TOKEN"),
        "WANDB_API_KEY": os.environ.get("WANDB_API_KEY"),
    },
)

/Users/davanstrien/Documents/daniel/blog/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_experimental.py:60: UserWarning: 'HfApi.run_uv_job' is experimental and might be subject to breaking changes in the future without prior notice. You can disable this warning by setting `HF_HUB_DISABLE_EXPERIMENTAL_WARNING=1` as environment variable.
  warnings.warn(

job.status

JobStatus(stage='RUNNING', message=None)

job.url

'https://huggingface.co/jobs/davanstrien/68b9c334d797b777b755e63b'

Exploring the Results

After training completes, our fine-tuned model is automatically pushed to the Hub. But how well does it actually perform on generating Iconclass codes?

I’ve built an interactive viewer to explore the model’s predictions on the test set - you can compare the ground truth labels with what our fine-tuned model generates:

Conclusion: Making VLM Fine-tuning Accessible

We’ve successfully fine-tuned a vision-language model to generate specialized art history metadata - and the entire process required just a few steps:

1. Format a dataset with images and expected responses
2. Write (or adapt) a simple TRL training script
3. Submit to HF Jobs with uv run for cloud-based training

The combination of TRL’s VLM support and HF Jobs removes the traditional barriers to specialized model training. No local GPU setup, no complex multi-modal training code, no infrastructure management.

The Challenge

Large language models like Qwen3-30B-A3B-Instruct (30 billion parameters) exceed single GPU memory limits. Running batch inference on datasets requires: - Multi-GPU setup with tensor parallelism - Handling prompts that exceed context limits - Managing dependencies and environment setup

Traditional approaches involve complex Docker setups, manual dependency management, and custom scripts for GPU coordination. HF Jobs changes this.

What is vLLM?

vLLM is a very well known and heavily used inference engine. It is known for its ability to scale inference for LLMs. While we can use vLLM via an OpenAI compatible API, it also has a powerful batch inference mode that allows us to process large datasets of prompts efficiently. This “offline inference” is particularly useful when we want to generate responses for a large number of prompts without the overhead of API calls.

What are uv Scripts?

UV scripts are Python scripts with inline dependency metadata that automatically install and manage their requirements. Instead of separate requirements.txt files or complex setup instructions, everything needed to run the script is declared in the script itself:

  # /// script
  # requires-python = ">=3.10"
  # dependencies = [
  #     "vllm",
  #     "transformers",
  #     "datasets",
  # ]
  # ///

  # Your Python code here

When you run uv run script.py, UV automatically creates an isolated environment, installs dependencies, and executes your code. No virtual env setup, no pip install commands, no version conflicts.

The Solution: UV Scripts + vLLM + HF Jobs

HF Jobs provides managed GPU infrastructure. This is already very useful but combined with uv Scripts we can more easily distribute scripts for a variety of ML/AI tasks in a (more) reproducible way.

The uv scripts Hugging Face org

Since I’m so excited about uv Scripts, I created a Hugging Face org to host them: uv scripts. This org will contain a variety of uv Scripts that you can use to run jobs on HF Jobs. For this example we’ll use a script that allows us to run inference for a model using vLLM. This script exposes a bunch of parameters that allow you to control how the inference is run, including the model to use, the number of GPUs to use, and the batch size etc.

In this case, the script expects as input a dataset with a column containing the input prompts (as messages). It will then run inference on the model using vLLM and return the generated responses in a new dataset.

I’m personally quite excited to see people sharing more uv scripts for things that are not complex enough to justify a full repository but that are still useful to share and run on the Hugging Face platform!

If you are curious, you can check out the script here or below:

import requests

print(requests.get("https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py").text)

# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "datasets",
#     "flashinfer-python",
#     "huggingface-hub[hf_transfer]",
#     "torch",
#     "transformers",
#     "vllm>=0.8.5",
# ]
#
# ///
"""
Generate responses for prompts in a dataset using vLLM for efficient GPU inference.

This script loads a dataset from Hugging Face Hub containing chat-formatted messages,
applies the model's chat template, generates responses using vLLM, and saves the
results back to the Hub with a comprehensive dataset card.

Example usage:
    # Local execution with auto GPU detection
    uv run generate-responses.py \\
        username/input-dataset \\
        username/output-dataset \\
        --messages-column messages

    # With custom model and sampling parameters
    uv run generate-responses.py \\
        username/input-dataset \\
        username/output-dataset \\
        --model-id meta-llama/Llama-3.1-8B-Instruct \\
        --temperature 0.9 \\
        --top-p 0.95 \\
        --max-tokens 2048

    # HF Jobs execution (see script output for full command)
    hf jobs uv run --flavor a100x4 ...
"""

import argparse
import logging
import os
import sys
from datetime import datetime
from typing import Optional

from datasets import load_dataset
from huggingface_hub import DatasetCard, get_token, login
from torch import cuda
from tqdm.auto import tqdm
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

# Enable HF Transfer for faster downloads
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"

logging.basicConfig(
    level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
)
logger = logging.getLogger(__name__)


def check_gpu_availability() -> int:
    """Check if CUDA is available and return the number of GPUs."""
    if not cuda.is_available():
        logger.error("CUDA is not available. This script requires a GPU.")
        logger.error(
            "Please run on a machine with NVIDIA GPU or use HF Jobs with GPU flavor."
        )
        sys.exit(1)

    num_gpus = cuda.device_count()
    for i in range(num_gpus):
        gpu_name = cuda.get_device_name(i)
        gpu_memory = cuda.get_device_properties(i).total_memory / 1024**3
        logger.info(f"GPU {i}: {gpu_name} with {gpu_memory:.1f} GB memory")

    return num_gpus


def create_dataset_card(
    source_dataset: str,
    model_id: str,
    messages_column: str,
    sampling_params: SamplingParams,
    tensor_parallel_size: int,
    num_examples: int,
    generation_time: str,
    num_skipped: int = 0,
    max_model_len_used: Optional[int] = None,
) -> str:
    """Create a comprehensive dataset card documenting the generation process."""
    filtering_section = ""
    if num_skipped > 0:
        skip_percentage = (num_skipped / num_examples) * 100
        processed = num_examples - num_skipped
        filtering_section = f"""

### Filtering Statistics

- **Total Examples**: {num_examples:,}
- **Processed**: {processed:,} ({100 - skip_percentage:.1f}%)
- **Skipped (too long)**: {num_skipped:,} ({skip_percentage:.1f}%)
- **Max Model Length Used**: {max_model_len_used:,} tokens

Note: Prompts exceeding the maximum model length were skipped and have empty responses."""

    return f"""---
tags:
- generated
- vllm
- uv-script
---

# Generated Responses Dataset

This dataset contains generated responses for prompts from [{source_dataset}](https://huggingface.co/datasets/{source_dataset}).

## Generation Details

- **Source Dataset**: [{source_dataset}](https://huggingface.co/datasets/{source_dataset})
- **Messages Column**: `{messages_column}`
- **Model**: [{model_id}](https://huggingface.co/{model_id})
- **Number of Examples**: {num_examples:,}
- **Generation Date**: {generation_time}{filtering_section}

### Sampling Parameters

- **Temperature**: {sampling_params.temperature}
- **Top P**: {sampling_params.top_p}
- **Top K**: {sampling_params.top_k}
- **Min P**: {sampling_params.min_p}
- **Max Tokens**: {sampling_params.max_tokens}
- **Repetition Penalty**: {sampling_params.repetition_penalty}

### Hardware Configuration

- **Tensor Parallel Size**: {tensor_parallel_size}
- **GPU Configuration**: {tensor_parallel_size} GPU(s)

## Dataset Structure

The dataset contains all columns from the source dataset plus:
- `response`: The generated response from the model

## Generation Script

Generated using the vLLM inference script from [uv-scripts/vllm](https://huggingface.co/datasets/uv-scripts/vllm).

To reproduce this generation:

```bash
uv run https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py \\
    {source_dataset} \\
    <output-dataset> \\
    --model-id {model_id} \\
    --messages-column {messages_column} \\
    --temperature {sampling_params.temperature} \\
    --top-p {sampling_params.top_p} \\
    --top-k {sampling_params.top_k} \\
    --max-tokens {sampling_params.max_tokens}{f" \\\\\\n    --max-model-len {max_model_len_used}" if max_model_len_used else ""}
```
"""


def main(
    src_dataset_hub_id: str,
    output_dataset_hub_id: str,
    model_id: str = "Qwen/Qwen3-30B-A3B-Instruct-2507",
    messages_column: str = "messages",
    output_column: str = "response",
    temperature: float = 0.7,
    top_p: float = 0.8,
    top_k: int = 20,
    min_p: float = 0.0,
    max_tokens: int = 16384,
    repetition_penalty: float = 1.0,
    gpu_memory_utilization: float = 0.90,
    max_model_len: Optional[int] = None,
    tensor_parallel_size: Optional[int] = None,
    skip_long_prompts: bool = True,
    hf_token: Optional[str] = None,
):
    """
    Main generation pipeline.

    Args:
        src_dataset_hub_id: Input dataset on Hugging Face Hub
        output_dataset_hub_id: Where to save results on Hugging Face Hub
        model_id: Hugging Face model ID for generation
        messages_column: Column name containing chat messages
        output_column: Column name for generated responses
        temperature: Sampling temperature
        top_p: Top-p sampling parameter
        top_k: Top-k sampling parameter
        min_p: Minimum probability threshold
        max_tokens: Maximum tokens to generate
        repetition_penalty: Repetition penalty parameter
        gpu_memory_utilization: GPU memory utilization factor
        max_model_len: Maximum model context length (None uses model default)
        tensor_parallel_size: Number of GPUs to use (auto-detect if None)
        skip_long_prompts: Skip prompts exceeding max_model_len instead of failing
        hf_token: Hugging Face authentication token
    """
    generation_start_time = datetime.now().isoformat()

    # GPU check and configuration
    num_gpus = check_gpu_availability()
    if tensor_parallel_size is None:
        tensor_parallel_size = num_gpus
        logger.info(
            f"Auto-detected {num_gpus} GPU(s), using tensor_parallel_size={tensor_parallel_size}"
        )
    else:
        logger.info(f"Using specified tensor_parallel_size={tensor_parallel_size}")
        if tensor_parallel_size > num_gpus:
            logger.warning(
                f"Requested {tensor_parallel_size} GPUs but only {num_gpus} available"
            )

    # Authentication - try multiple methods
    HF_TOKEN = hf_token or os.environ.get("HF_TOKEN") or get_token()

    if not HF_TOKEN:
        logger.error("No HuggingFace token found. Please provide token via:")
        logger.error("  1. --hf-token argument")
        logger.error("  2. HF_TOKEN environment variable")
        logger.error("  3. Run 'huggingface-cli login' or use login() in Python")
        sys.exit(1)

    logger.info("HuggingFace token found, authenticating...")
    login(token=HF_TOKEN)

    # Initialize vLLM
    logger.info(f"Loading model: {model_id}")
    vllm_kwargs = {
        "model": model_id,
        "tensor_parallel_size": tensor_parallel_size,
        "gpu_memory_utilization": gpu_memory_utilization,
    }
    if max_model_len is not None:
        vllm_kwargs["max_model_len"] = max_model_len
        logger.info(f"Using max_model_len={max_model_len}")

    llm = LLM(**vllm_kwargs)

    # Load tokenizer for chat template
    logger.info("Loading tokenizer...")
    tokenizer = AutoTokenizer.from_pretrained(model_id)

    # Create sampling parameters
    sampling_params = SamplingParams(
        temperature=temperature,
        top_p=top_p,
        top_k=top_k,
        min_p=min_p,
        max_tokens=max_tokens,
        repetition_penalty=repetition_penalty,
    )

    # Load dataset
    logger.info(f"Loading dataset: {src_dataset_hub_id}")
    dataset = load_dataset(src_dataset_hub_id, split="train")
    total_examples = len(dataset)
    logger.info(f"Dataset loaded with {total_examples:,} examples")

    # Validate messages column
    if messages_column not in dataset.column_names:
        logger.error(
            f"Column '{messages_column}' not found. Available columns: {dataset.column_names}"
        )
        sys.exit(1)

    # Get effective max length for filtering
    if max_model_len is not None:
        effective_max_len = max_model_len
    else:
        # Get model's default max length
        effective_max_len = llm.llm_engine.model_config.max_model_len
    logger.info(f"Using effective max model length: {effective_max_len}")

    # Process messages and apply chat template
    logger.info("Applying chat template to messages...")
    all_prompts = []
    valid_prompts = []
    valid_indices = []
    skipped_info = []

    for i, example in enumerate(tqdm(dataset, desc="Processing messages")):
        messages = example[messages_column]
        # Apply chat template
        prompt = tokenizer.apply_chat_template(
            messages, tokenize=False, add_generation_prompt=True
        )
        all_prompts.append(prompt)

        # Count tokens if filtering is enabled
        if skip_long_prompts:
            tokens = tokenizer.encode(prompt)
            if len(tokens) <= effective_max_len:
                valid_prompts.append(prompt)
                valid_indices.append(i)
            else:
                skipped_info.append((i, len(tokens)))
        else:
            valid_prompts.append(prompt)
            valid_indices.append(i)

    # Log filtering results
    if skip_long_prompts and skipped_info:
        logger.warning(
            f"Skipped {len(skipped_info)} prompts that exceed max_model_len ({effective_max_len} tokens)"
        )
        logger.info("Skipped prompt details (first 10):")
        for idx, (prompt_idx, token_count) in enumerate(skipped_info[:10]):
            logger.info(
                f"  - Example {prompt_idx}: {token_count} tokens (exceeds by {token_count - effective_max_len})"
            )
        if len(skipped_info) > 10:
            logger.info(f"  ... and {len(skipped_info) - 10} more")

        skip_percentage = (len(skipped_info) / total_examples) * 100
        if skip_percentage > 10:
            logger.warning(f"WARNING: {skip_percentage:.1f}% of prompts were skipped!")

    if not valid_prompts:
        logger.error("No valid prompts to process after filtering!")
        sys.exit(1)

    # Generate responses - vLLM handles batching internally
    logger.info(f"Starting generation for {len(valid_prompts):,} valid prompts...")
    logger.info("vLLM will handle batching and scheduling automatically")

    outputs = llm.generate(valid_prompts, sampling_params)

    # Extract generated text and create full response list
    logger.info("Extracting generated responses...")
    responses = [""] * total_examples  # Initialize with empty strings

    for idx, output in enumerate(outputs):
        original_idx = valid_indices[idx]
        response = output.outputs[0].text.strip()
        responses[original_idx] = response

    # Add responses to dataset
    logger.info("Adding responses to dataset...")
    dataset = dataset.add_column(output_column, responses)

    # Create dataset card
    logger.info("Creating dataset card...")
    card_content = create_dataset_card(
        source_dataset=src_dataset_hub_id,
        model_id=model_id,
        messages_column=messages_column,
        sampling_params=sampling_params,
        tensor_parallel_size=tensor_parallel_size,
        num_examples=total_examples,
        generation_time=generation_start_time,
        num_skipped=len(skipped_info) if skip_long_prompts else 0,
        max_model_len_used=effective_max_len if skip_long_prompts else None,
    )

    # Push dataset to hub
    logger.info(f"Pushing dataset to: {output_dataset_hub_id}")
    dataset.push_to_hub(output_dataset_hub_id, token=HF_TOKEN)

    # Push dataset card
    card = DatasetCard(card_content)
    card.push_to_hub(output_dataset_hub_id, token=HF_TOKEN)

    logger.info("✅ Generation complete!")
    logger.info(
        f"Dataset available at: https://huggingface.co/datasets/{output_dataset_hub_id}"
    )


if __name__ == "__main__":
    if len(sys.argv) > 1:
        parser = argparse.ArgumentParser(
            description="Generate responses for dataset prompts using vLLM",
            formatter_class=argparse.RawDescriptionHelpFormatter,
            epilog="""
Examples:
  # Basic usage with default Qwen model
  uv run generate-responses.py input-dataset output-dataset
  
  # With custom model and parameters
  uv run generate-responses.py input-dataset output-dataset \\
    --model-id meta-llama/Llama-3.1-8B-Instruct \\
    --temperature 0.9 \\
    --max-tokens 2048
  
  # Force specific GPU configuration
  uv run generate-responses.py input-dataset output-dataset \\
    --tensor-parallel-size 2 \\
    --gpu-memory-utilization 0.95
  
  # Using environment variable for token
  HF_TOKEN=hf_xxx uv run generate-responses.py input-dataset output-dataset
            """,
        )

        parser.add_argument(
            "src_dataset_hub_id",
            help="Input dataset on Hugging Face Hub (e.g., username/dataset-name)",
        )
        parser.add_argument(
            "output_dataset_hub_id", help="Output dataset name on Hugging Face Hub"
        )
        parser.add_argument(
            "--model-id",
            type=str,
            default="Qwen/Qwen3-30B-A3B-Instruct-2507",
            help="Model to use for generation (default: Qwen3-30B-A3B-Instruct-2507)",
        )
        parser.add_argument(
            "--messages-column",
            type=str,
            default="messages",
            help="Column containing chat messages (default: messages)",
        )
        parser.add_argument(
            "--output-column",
            type=str,
            default="response",
            help="Column name for generated responses (default: response)",
        )
        parser.add_argument(
            "--temperature",
            type=float,
            default=0.7,
            help="Sampling temperature (default: 0.7)",
        )
        parser.add_argument(
            "--top-p",
            type=float,
            default=0.8,
            help="Top-p sampling parameter (default: 0.8)",
        )
        parser.add_argument(
            "--top-k",
            type=int,
            default=20,
            help="Top-k sampling parameter (default: 20)",
        )
        parser.add_argument(
            "--min-p",
            type=float,
            default=0.0,
            help="Minimum probability threshold (default: 0.0)",
        )
        parser.add_argument(
            "--max-tokens",
            type=int,
            default=16384,
            help="Maximum tokens to generate (default: 16384)",
        )
        parser.add_argument(
            "--repetition-penalty",
            type=float,
            default=1.0,
            help="Repetition penalty (default: 1.0)",
        )
        parser.add_argument(
            "--gpu-memory-utilization",
            type=float,
            default=0.90,
            help="GPU memory utilization factor (default: 0.90)",
        )
        parser.add_argument(
            "--max-model-len",
            type=int,
            help="Maximum model context length (default: model's default)",
        )
        parser.add_argument(
            "--tensor-parallel-size",
            type=int,
            help="Number of GPUs to use (default: auto-detect)",
        )
        parser.add_argument(
            "--hf-token",
            type=str,
            help="Hugging Face token (can also use HF_TOKEN env var)",
        )
        parser.add_argument(
            "--skip-long-prompts",
            action="store_true",
            default=True,
            help="Skip prompts that exceed max_model_len instead of failing (default: True)",
        )
        parser.add_argument(
            "--no-skip-long-prompts",
            dest="skip_long_prompts",
            action="store_false",
            help="Fail on prompts that exceed max_model_len",
        )

        args = parser.parse_args()

        main(
            src_dataset_hub_id=args.src_dataset_hub_id,
            output_dataset_hub_id=args.output_dataset_hub_id,
            model_id=args.model_id,
            messages_column=args.messages_column,
            output_column=args.output_column,
            temperature=args.temperature,
            top_p=args.top_p,
            top_k=args.top_k,
            min_p=args.min_p,
            max_tokens=args.max_tokens,
            repetition_penalty=args.repetition_penalty,
            gpu_memory_utilization=args.gpu_memory_utilization,
            max_model_len=args.max_model_len,
            tensor_parallel_size=args.tensor_parallel_size,
            skip_long_prompts=args.skip_long_prompts,
            hf_token=args.hf_token,
        )
    else:
        # Show HF Jobs example when run without arguments
        print("""
vLLM Response Generation Script
==============================

This script requires arguments. For usage information:
    uv run generate-responses.py --help

Example HF Jobs command with multi-GPU:
    # If you're logged in with huggingface-cli, token will be auto-detected
    hf jobs uv run \\
        --flavor l4x4 \\
        https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py \\
        username/input-dataset \\
        username/output-dataset \\
        --messages-column messages \\
        --model-id Qwen/Qwen3-30B-A3B-Instruct-2507 \\
        --temperature 0.7 \\
        --max-tokens 16384
        """)

An example: Running Qwen3-30B-A3B-Instruct to generare summaries of datasets from 2025

As an example, let’s run Qwen3-30B-A3B-Instruct to generate summaries of datasets from 2025. We’ll use the hf-jobs Python API to create a job that runs a uv Script on 4 GPUs with vLLM. First we’ll quickly prepare the dataset and prompts. We’ll use Polars + datasets to load the dataset and filter it down to the 2025 datasets.

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="librarian-bots/dataset_cards_with_metadata",
    local_dir="data",
    repo_type="dataset",
    allow_patterns=["*.parquet"],
)

'/Users/davanstrien/Documents/daniel/blog/posts/2025/hf-jobs/data'

We’ll do some filtering to focus on datasets where the cards are not super long or super short. We’ll also filter to focus on datasets with at least one like and ten downloads.

import polars as pl

df = pl.scan_parquet("data/**/*.parquet")

df.collect_schema()

Schema([('datasetId', String),
        ('author', String),
        ('last_modified', String),
        ('downloads', Int64),
        ('likes', Int64),
        ('tags', List(String)),
        ('task_categories', List(String)),
        ('createdAt', String),
        ('trending_score', Float64),
        ('card', String)])

df = df.filter(pl.col("card").str.len_chars() > 200)
df = df.filter(pl.col("downloads") > 2)
df = df.filter(pl.col("likes") > 1)

We make sure we have datetime for the createdAt column so we can filter by year.

df = df.with_columns(pl.col("createdAt").str.to_datetime())

from datetime import datetime

this_year = datetime.now().year
this_year

2025

df_2025 = df.filter(pl.col("createdAt").dt.year() == this_year)

Since we’re using the LazyFrame API, we can use the collect method to execute the query and get the results we get a nice optimized query plan. This is very nice since you can be quite lazy in how you filter and transform the data and Polars will optimize the query for you!

df_2025.show_graph(optimized=True, engine="streaming")

Polars and datasets play nicely together so we can easily convert between the two. Since we’ve done all the filtering we want, we can convert the Polars DataFrame to a Datasets Dataset.

from datasets import Dataset

ds = Dataset.from_polars(df_2025.collect())

ds

Dataset({
    features: ['datasetId', 'author', 'last_modified', 'downloads', 'likes', 'tags', 'task_categories', 'createdAt', 'trending_score', 'card'],
    num_rows: 2419
})

We’ll do one more filter to remove datasets that don’t have a card. We could also do this in Polars but since the huggingface_hub library has a nice way of converting a string into a dataset card where we can seperate the YAML from the main content, we’ll do it using the datasets library and the filter function.

from huggingface_hub import DatasetCard

def is_short_card(row, min_length=200):
    card = DatasetCard(row['card']).text
    return len(card) > min_length

ds = ds.filter(is_short_card, num_proc=4)

def format_prompt_for_card(row, max_length=8000):
    card = DatasetCard(row['card']).text
    datasetId = row['datasetId']
    return f"""You are a helpful assistant that provides concise summaries of dataset cards for datasets on the Hugging Face Hub.
The Hub ID of the dataset is: {datasetId}.
The dataset card is as follows:
{card[:max_length]}]
Please write a one to two sentence summary of the dataset card.
The summary should be concise and informative, capturing the essence of the dataset.
The summary should be in English.
The goal of the summary is to provide a quick overview of the dataset's content and purpose. 
This summary will be used to help users quickly understand the dataset and as input for creating embeddings for the dataset card.
    """

print(format_prompt_for_card(ds[0]))

You are a helpful assistant that provides concise summaries of dataset cards for datasets on the Hugging Face Hub.
The Hub ID of the dataset is: agentlans/high-quality-multilingual-sentences.
The dataset card is as follows:
# High Quality Multilingual Sentences

- This dataset contains multilingual sentences derived from the [agentlans/LinguaNova](https://huggingface.co/datasets/agentlans/LinguaNova) dataset.
- It includes 1.58 million rows across 51 different languages, each in its own configuration.

Example row (from the `all` config):
```json
{
    "text": "امام جمعه اصفهان گفت: میزان نیاز آب شرب اصفهان ۱۱.۵ متر مکعب است که تمام استان اصفهان را پوشش میدهد و نسبت به قبل از انقلاب یکی از پیشرفتها در حوزه آب بوده است.",
    "fasttext": "fa",
    "gcld3": "fa"
}
```

Fields:
- **text**: The sentence in the original language.
- **fasttext**, **gcld3**: Language codes determined using fastText and gcld3 Python packages.

## Configurations

Each individual language is available as a separate configuration, such as `ar`, `en`. These configurations contain only sentences identified to be of that specific language by both the fastText and gcld3 models.

Example row (from a language-specific config):
```json
{
    "text": "Ne vienas asmuo yra apsaugotas nuo parazitų atsiradimo organizme."
}
```

## Methods

### Data Loading and Processing

The `all` split was downloaded from the [agentlans/LinguaNova](https://huggingface.co/datasets/agentlans/LinguaNova) dataset.
1. **Text Cleaning**: Raw text was cleaned by removing HTML tags, emails, emojis, hashtags, user handles, and URLs. Unicode characters and whitespace were normalized, and hyphenated words were handled to ensure consistency.
2. **Sentence Segmentation**: Text was segmented into individual sentences using ICU's `BreakIterator` class, which efficiently processed different languages and punctuation.
3. **Deduplication**: Duplicate entries were removed to maintain uniqueness and prevent redundancy in the dataset.

### Language Detection

Two methods were used for language identification:
1. **gcld3**: Google's Compact Language Detector 3 was used for fast and accurate language identification.
2. **fastText**: Facebook’s fastText model was employed, which improved accuracy by considering subword information.

### Quality Assessment

Text quality was assessed through batch inference using the [agentlans/multilingual-e5-small-aligned-quality](https://huggingface.co/agentlans/multilingual-e5-small-aligned-quality) model.
1. **Data Retrieval**: Entries with a quality score of 1 or higher and a minimum input length of 20 characters were retained.
2. **Text Refinement**: Leading punctuation and spaces were removed, and balanced quotation marks were validated using regular expressions.

### Dataset Configs

The filtered sentences and their annotated languages were written to the `all.jsonl` file. The file was then split into language-specific JSONL files, containing only those sentences that matched consistently with both gcld3 and fasttext in terms of language identification. Only languages with at least 100 sentences after filtering were included in these configs.

## Usage

### Loading the Dataset
```python
from datasets import load_dataset

dataset = load_dataset('agentlans/high-quality-multilingual-sentences', 'all')
```

For language-specific configurations:
```python
language_config = load_dataset('agentlans/high-quality-multilingual-sentences', 'en')  # Replace with desired language code.
```

### Example Usage in Python
```python
from datasets import load_dataset

# Load the dataset for all languages or a specific one
dataset_all = load_dataset("agentlans/high-quality-multilingual-sentences", "all")
print(dataset_all["train"][0])

language_config = load_dataset("agentlans/high-quality-multilingual-sentences", "en")  # Replace 'en' with desired language code.
print(language_config["train"][:5])
```

## Limitations

- **Multilingual content bias**: The quality classifier is biased towards educational and more formal content.
- **Language coverage**: Limited to the 50 written languages from LinguaNova. There's a lack of African and indigenous languages.
- **Short input issues**: Language identification accuracy can suffer when working with short inputs like single sentences.
- **Sentence segmentation challenges**: Some languages' delimiters might not be handled correctly.
- **Redundancy**: The filtering was only done on exact matches so some sentences may be similar (but not identical).

Additionally:
- **Thai data imbalance**: Fewer examples are available for `th` (Thai) than expected. Could be a sentence segmentation problem.
- **Malay and Indonesian**: There are few examples for the `ms` (Malay) subset. Consider also using the `id` (Indonesian) subset when training models.
- **Chinese written forms**: This dataset does not distinguish between different Chinese character variations.

## Licence

This dataset is released under a [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) licence, allowing for free use and distribution as long as proper attribution is given to the original source.]
Please write a one to two sentence summary of the dataset card.
The summary should be concise and informative, capturing the essence of the dataset.
The summary should be in English.
The goal of the summary is to provide a quick overview of the dataset's content and purpose. 
This summary will be used to help users quickly understand the dataset and as input for creating embeddings for the dataset card.
    

def create_messages(row):
    return {"messages": [
        {
            "role": "user",
            "content": format_prompt_for_card(row),
        },
    ]}

ds = ds.map(create_messages, num_proc=4)

ds

Dataset({
    features: ['datasetId', 'author', 'last_modified', 'downloads', 'likes', 'tags', 'task_categories', 'createdAt', 'trending_score', 'card', 'messages'],
    num_rows: 2082
})

ds = ds.remove_columns([c for c in ds.column_names if c not in ['messages', 'datasetId']])

ds.push_to_hub("davanstrien/cards_with_prompts")

CommitInfo(commit_url='https://huggingface.co/datasets/davanstrien/cards_with_prompts/commit/8e32c041eba4fbf1729e3f5a4d1536365185f7d2', commit_message='Upload dataset', commit_description='', oid='8e32c041eba4fbf1729e3f5a4d1536365185f7d2', pr_url=None, repo_url=RepoUrl('https://huggingface.co/datasets/davanstrien/cards_with_prompts', endpoint='https://huggingface.co', repo_type='dataset', repo_id='davanstrien/cards_with_prompts'), pr_revision=None, pr_num=None)

Launching our job

We now have the dataset with the prompts we want to use for inference.

The interface for Jobs should look familiar if you’ve used Docker before. We can use Jobs via CLI or Python API. Via the CLI a basic command to run a job looks like this:

hf jobs run python:3.12 python -c "print('Hello from the cloud!')"

There is also an experimental uv command thaty allows us to run uv scripts directly:

hf jobs uv run script-url

As an example, we can run another simple script from the uv scripts org and just print the help for the script:

hf jobs uv run https://huggingface.co/datasets/uv-scripts/deduplication/raw/main/semantic-dedupe.py --help

/Users/davanstrien/Library/Application Support/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/utils/_experimental.py:60: UserWarning: 'HfApi.run_uv_job' is experimental and might be subject to breaking changes in the future without prior notice. You can disable this warning by setting `HF_HUB_DISABLE_EXPERIMENTAL_WARNING=1` as environment variable.
  warnings.warn(
Job started with ID: 688a31096dcd97e42f8095e7
View at: https://huggingface.co/jobs/davanstrien/688a31096dcd97e42f8095e7
Downloading pygments (1.2MiB)
Downloading hf-xet (3.0MiB)
Downloading numpy (15.9MiB)
Downloading tokenizers (3.0MiB)
Downloading setuptools (1.1MiB)
Downloading aiohttp (1.6MiB)
Downloading pandas (11.4MiB)
Downloading pyarrow (40.8MiB)
Downloading usearch (2.0MiB)
Downloading hf-transfer (3.4MiB)
Downloading simsimd (1.0MiB)
 Downloading simsimd
 Downloading usearch
 Downloading tokenizers
 Downloading hf-xet
 Downloading hf-transfer
 Downloading aiohttp
 Downloading pygments
 Downloading setuptools
 Downloading numpy
 Downloading pyarrow
 Downloading pandas
Installed 50 packages in 116ms
usage: semantic-dedupekDOpug.py [-h] [--split SPLIT]
                                [--method {duplicates,outliers,representatives}]
                                [--threshold THRESHOLD]
                                [--batch-size BATCH_SIZE]
                                [--max-samples MAX_SAMPLES] [--private]
                                [--hf-token HF_TOKEN]
                                dataset column output_repo

Deduplicate a dataset using semantic similarity

positional arguments:
  dataset               Input dataset ID (e.g., 'imdb' or 'username/dataset')
  column                Text column to deduplicate on
  output_repo           Output dataset repository name

options:
  -h, --help            show this help message and exit
  --split SPLIT         Dataset split to process (default: train)
  --method {duplicates,outliers,representatives}
                        Deduplication method (default: duplicates)
  --threshold THRESHOLD
                        Similarity threshold for duplicates (default: 0.9)
  --batch-size BATCH_SIZE
                        Batch size for processing (default: 64)
  --max-samples MAX_SAMPLES
                        Maximum number of samples to process (for testing)
  --private             Create private dataset repository
  --hf-token HF_TOKEN   Hugging Face API token (defaults to HF_TOKEN env var)

Examples:
  # Basic usage
  uv run semantic-dedupe.py imdb text imdb-deduped

  # With options
  uv run semantic-dedupe.py squad question squad-deduped --threshold 0.85 --method duplicates

  # Test with small sample
  uv run semantic-dedupe.py large-dataset text test-dedup --max-samples 100
        

You’ll see that uv takes care of installing the dependencies and running the script. This is very convenient since we don’t have to worry about setting up a virtual environment or installing dependencies manually. This can also be very nice if you want to share a script with others and want to help them avoid getting stuck in dependency hell.

We can also run hf jobs via the Python API. This is very convenient if you want to run jobs programmatically or if you want to integrate Jobs into your existing Python code (i.e. to run one step that requires a GPU and another step that doesn’t).

Running our inference job via huggingface_hub library

We can use the huggingface_hub library to run our inference job using run_uv_job

We’ll grab a token to pass to our job

from huggingface_hub import HfApi, get_token


HF_TOKEN = get_token()

We’ll create an instance of the HfApi class and use the run_uv_job method to run our job. We’ll pass the URL of the script we want to run, the dataset we want to use, and the parameters we want to use for the job.

api = HfApi()

Let’s see what the run_uv_job method looks like:

?api.run_uv_job

Signature:

api.run_uv_job(

    script: 'str',

    *,

    script_args: 'Optional[List[str]]' = None,

    dependencies: 'Optional[List[str]]' = None,

    python: 'Optional[str]' = None,

    image: 'Optional[str]' = None,

    env: 'Optional[Dict[str, Any]]' = None,

    secrets: 'Optional[Dict[str, Any]]' = None,

    flavor: 'Optional[SpaceHardware]' = None,

    timeout: 'Optional[Union[int, float, str]]' = None,

    namespace: 'Optional[str]' = None,

    token: 'Union[bool, str, None]' = None,

    _repo: 'Optional[str]' = None,

) -> 'JobInfo'

Docstring:

Run a UV script Job on Hugging Face infrastructure.



Args:

    script (`str`):

        Path or URL of the UV script.



    script_args (`List[str]`, *optional*)

        Arguments to pass to the script.



    dependencies (`List[str]`, *optional*)

        Dependencies to use to run the UV script.



    python (`str`, *optional*)

        Use a specific Python version. Default is 3.12.



    image (`str`, *optional*, defaults to "ghcr.io/astral-sh/uv:python3.12-bookworm"):

        Use a custom Docker image with `uv` installed.



    env (`Dict[str, Any]`, *optional*):

        Defines the environment variables for the Job.



    secrets (`Dict[str, Any]`, *optional*):

        Defines the secret environment variables for the Job.



    flavor (`str`, *optional*):

        Flavor for the hardware, as in Hugging Face Spaces. See [`SpaceHardware`] for possible values.

        Defaults to `"cpu-basic"`.



    timeout (`Union[int, float, str]`, *optional*):

        Max duration for the Job: int/float with s (seconds, default), m (minutes), h (hours) or d (days).

        Example: `300` or `"5m"` for 5 minutes.



    namespace (`str`, *optional*):

        The namespace where the Job will be created. Defaults to the current user's namespace.



    token `(Union[bool, str, None]`, *optional*):

        A valid user access token. If not provided, the locally saved token will be used, which is the

        recommended authentication method. Set to `False` to disable authentication.

        Refer to: https://huggingface.co/docs/huggingface_hub/quick-start#authentication.



Example:



    ```python

    >>> from huggingface_hub import run_uv_job

    >>> script = "https://raw.githubusercontent.com/huggingface/trl/refs/heads/main/trl/scripts/sft.py"

    >>> run_uv_job(script, dependencies=["trl"], flavor="a10g-small")

    ```

File:      ~/Documents/daniel/blog/.venv/lib/python3.12/site-packages/huggingface_hub/hf_api.py

Type:      method

we can use the run_uv_job method to run our job. We’ll pass the URL of the script we want to run, the dataset we want to use, and the parameters we want to use for the job. These parameters will be passed to the script as command line arguments. Since we’re using vLLM, we’ll pass the vllm Docker image. This will mean that our job is run in this Docker container.

We can now run our job using the run_uv_job method. This will start the job and return a job object that we can use to monitor the job’s progress.

job = api.run_uv_job(
    script="https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py",
    script_args=[
        "davanstrien/cards_with_prompts",  # Dataset with prompts
        "davanstrien/test-generated-responses",  # Where to store the generated responses
        "--model-id",  # Model to use for inference
        "Qwen/Qwen3-30B-A3B-Instruct-2507",  # Model to use for inference
        "--gpu-memory-utilization",  # GPU memory utilization
        "0.9",
        "--max-tokens",  # Maximum number of tokens
        "900",
        "--max-model-len",  # Maximum model length
        "8000",
    ],
    flavor="l4x4",  # What hardware to use
    image="vllm/vllm-openai:latest",  # Docker image to use
    secrets={"HF_TOKEN": HF_TOKEN},  # Pass as secret``
    env={"UV_PRERELEASE": "if-necessary"},  # Pass as env var
)

/Users/davanstrien/Documents/daniel/blog/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_experimental.py:60: UserWarning: 'HfApi.run_uv_job' is experimental and might be subject to breaking changes in the future without prior notice. You can disable this warning by setting `HF_HUB_DISABLE_EXPERIMENTAL_WARNING=1` as environment variable.
  warnings.warn(

We can get a url for our job, this will give us a page where we can monitor the job’s progress and see the logs (note this won’t URL won’t work for you unless you run this job yourself).

print(f"Job URL: {job.url}")

Job URL: https://huggingface.co/jobs/davanstrien/688a33391c97bc486de2a232

We can also print the status of the job

job.status

JobStatus(stage='RUNNING', message=None)

There are also a bunch of other attributes from the job that can be useful when running jobs as part of a larger workflow. For example, we can get the job’s creation time, the job’s status etc.

job.created_at

datetime.datetime(2025, 7, 30, 14, 59, 5, 648000, tzinfo=datetime.timezone.utc)

job.flavor

'l4x4'

job.command

['uv',
 'run',
 'https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py',
 'davanstrien/cards_with_prompts',
 'davanstrien/test-generated-responses',
 '--model-id',
 'Qwen/Qwen3-30B-A3B-Instruct-2507',
 '--gpu-memory-utilization',
 '0.9',
 '--max-tokens',
 '900',
 '--max-model-len',
 '8000']

We can also grab the logs

api.fetch_job_logs(
    job_id=job.id,
)

<generator object HfApi.fetch_job_logs at 0x1612df4c0>

This returns a generator, let’s turn it into a list so we can print out a few examples of the logs

print(
    list(
        api.fetch_job_logs(
            job_id=job.id,
        )
    )[:-10]
)  # Print the last 10 lines of logs

We can also see the resulting dataset for the job here or below. You can see we have the original prompts and the generated responses.

Collections as Training Data: The Current Landscape

There’s growing momentum around libraries contributing digitized collections to AI training datasets, for example Harvard’s Institutional Data Initiative aims to help institutions “refine and publish their collections as data.” This follows on from existing work on “collections as data”. The Institutional Data Initiative presents an exciting model for how libraries can explore what new roles for libraries in relation to data and AI could look like.

EleutherAI’s Common Pile has taken concrete steps—releasing an 8TB corpus of openly licensed text that includes nearly 300,000 digitized public-domain books from the Library of Congress and the Internet Archive. They’ve demonstrated that models trained on this openly licensed data (their Comma models) can perform comparably to those trained on unlicensed datasets.

Common Pile represents genuine progress and thoughtful collaboration: they’ve built tools for data extraction and license identification, collaborated with Mozilla on dataset standards, and expressed interest in partnerships with libraries to improve data quality through better OCR and collaborative dataset development. This is exactly the kind of respectful partnership approach that can benefit everyone.

However, not all AI development follows this model. Many large tech companies have taken a different approach, scraping library collections without attribution or partnership. The National Library of the Netherlands captured this concern well in their AI statement, stating explicitly that they “restrict access of commercial AI to KB collections”.

Even in well-intentioned collaborations, there’s often an asymmetry: libraries provide data while the tools, infrastructure, and primary benefits of AI development flow elsewhere. The real opportunity isn’t just about contributing more data: it’s about how libraries can meaningfully participate in and benefit from AI development.

Who Benefits? Libraries as Essential Partners

How can libraries benefit from actively participating in AI development?¹ The answer lies in recognizing that the technical infrastructure for AI isn’t separate from library needs - it can directly enhance library services and capabilities.

A Practical Path Forward: Better Collections Through ML-Ready Datasets

Here’s my key pitch: the tools needed to create ML-ready datasets are the same tools that can transform how libraries serve their communities.

To create truly useful datasets for both AI and library purposes, we need:

• Better OCR quality assessment and improvement: Move beyond crude heuristics and internal confidence scores to systematically identify and correct OCR errors at scale. Recent developments in OCR technology could significantly improve the quality of digitized GLAM collections.

• Enhanced transcription for archival audio and video: While tools like Whisper provide a starting point for speech recognition, the GLAM sector can co-create better training data for models that handle the nuances of historical recordings, dialects, and specialized terminology.

• Specialized classifiers for GLAM-specific tasks: The recent resurgence of efficient AI classifiers opens new possibilities. These tools can automatically categorize collections, generate metadata, identify sensitive content, and enhance discovery—all while being affordable enough for library-scale deployment.

The benefits are concrete and empowering: Libraries would gain semantic search capabilities that could transform patron discovery, automated metadata generation tools that help librarians process decades-old backlogs more efficiently, and systems to identify and correct digitization errors at scale. These tools don’t replace librarians - they amplify their expertise, allowing them to focus on the complex curatorial and community work that only humans can do. These aren’t distant possibilities - they’re practical outcomes of preparing collections for ML use.

How to Get There: A Collaborative Approach

One path forward: a collaborative effort where libraries work with AI developers to co-create the tools necessary to transform existing collections into materials useful for both AI training and library purposes.

The output would be a series of datasets released as machine-readable text in simple, accessible formats². These datasets would remain under institutional control while capturing the nuances of each collection. Standardization would enable easy combination for large-scale digital humanities research, aggregation in shared library platforms, and use as training data.

A Choice for Libraries

Libraries have an exciting opportunity. By becoming active partners in AI development, libraries can help shape the tools that will mediate access to human knowledge while gaining powerful new capabilities to serve their communities.

The convergence of library expertise and AI technology offers exciting possibilities. Libraries bring irreplaceable skills in curation, metadata, community understanding, and ethical stewardship. Combined with emerging AI tools, this expertise can unlock new ways to make collections discoverable, accessible, and useful.

The invitation is open: join the conversation, shape the tools, and help build an AI ecosystem that reflects library values of open access, cultural preservation, and community service. The future of information access is being written now — and libraries have essential contributions to make.

Support for ModernBert

Recently, vLLM added support for the ModernBERT arcitecture. This is great news for the community as it means that we can now use vLLM to scale inference for ModernBERT based classifier models.

In the rest of this post, I’ll give a quick overview of how to use vLLM to scale inference for ModernBERT based classifier models.

Installation

Since we are using a feature not yet in a main release, we need to install the nightly version of vLLM. Using uv makes this very easy. We’ll also install the datasets and polars libraries as we’ll be using them to load and process our data.

%pip install uv
!uv pip install -U vllm datasets --extra-index-url https://wheels.vllm.ai/nightly

Since I orginally ran this on Google Colab, I’ll also grab the HF token from the user data (we will need a token if we are working with a private dataset/model or if we want to push the results back to the Hub).

from google.colab import userdata

hf_token = userdata.get("HF_TOKEN")

The use case

Since it’s not super imporant for the post, I’ll keep it very brief but the use case I am working on is using ModernBERT to classify arXiv papers as containting new datasets. You can read about how I created the dataset in this post. This results in davanstrien/ModernBERT-base-is-new-arxiv-dataset a model that classifies papers as containing a new dataset or not.

Although we’ll be using this model in this post, the same approach should work well with any ModernBERT based classifier model. This includes these models based on the base variant and these models based on the large variant.

Loading the model

We’ll now load the model using vLLM. This is very similar to loading any other model in vllm, the only difference is that we specify task="classify" to tell vLLM that we are using a classifier model.

from vllm import LLM

llm = LLM(model="davanstrien/ModernBERT-base-is-new-arxiv-dataset", task="classify")

INFO 04-24 14:16:50 [__init__.py:239] Automatically detected platform cuda.

INFO 04-24 14:16:59 [config.py:2888] Downcasting torch.float32 to torch.float16.
WARNING 04-24 14:17:13 [arg_utils.py:1685] --task classify is not supported by the V1 Engine. Falling back to V0. 
INFO 04-24 14:17:13 [llm_engine.py:242] Initializing a V0 LLM engine (v0.8.5.dev195+gc0dfd9751) with config: model='davanstrien/ModernBERT-base-is-new-arxiv-dataset', speculative_config=None, tokenizer='davanstrien/ModernBERT-base-is-new-arxiv-dataset', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=davanstrien/ModernBERT-base-is-new-arxiv-dataset, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=PoolerConfig(pooling_type=None, normalize=None, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False, 

INFO 04-24 14:17:20 [cuda.py:295] Using Flash Attention backend.
INFO 04-24 14:17:20 [parallel_state.py:946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 04-24 14:17:20 [model_runner.py:1120] Starting to load model davanstrien/ModernBERT-base-is-new-arxiv-dataset...
INFO 04-24 14:17:21 [weight_utils.py:265] Using model weights format ['*.safetensors']

INFO 04-24 14:17:25 [weight_utils.py:281] Time spent downloading weights for davanstrien/ModernBERT-base-is-new-arxiv-dataset: 3.888106 seconds
INFO 04-24 14:17:25 [weight_utils.py:315] No model.safetensors.index.json found in remote.

INFO 04-24 14:17:26 [loader.py:458] Loading weights took 0.18 seconds
INFO 04-24 14:17:26 [model_runner.py:1156] Model loading took 0.3098 GiB and 5.243792 seconds

Let’s test the model out with a simple example.

(output,) = llm.classify(
    "Title: This paper introduces a new benchmark dataset shared on Zenodo"
)

probs = output.outputs.probs
print(f"Class Probabilities: {probs!r} (size={len(probs)})")

Class Probabilities: [1.6357421875, -1.3671875] (size=2)

We can also get the top predicted class and confidence score.

import torch
import torch.nn.functional as F

logits = torch.tensor(output.outputs.probs)
probabilities = F.softmax(logits, dim=0)

# Get the top predicted class
top_idx = torch.argmax(probabilities).item()
top_prob = probabilities[top_idx].item()

print(f"Top class index: {top_idx}")
print(f"Confidence: {top_prob:.4f}")

Top class index: 0
Confidence: 0.9527

import httpx
from huggingface_hub import hf_hub_url

id2label = (
    httpx.get(
        hf_hub_url(
            "davanstrien/ModernBERT-base-is-new-arxiv-dataset", filename="config.json"
        )
    )
    .json()
    .get("id2label")
)
id2label

{'0': 'new_dataset', '1': 'no_new_dataset'}

from toolz import keymap

id2label = keymap(int, id2label)

id2label.get(top_idx)

'new_dataset'

Inference for a large dataset

There are many uses cases of classfiers but one common use case is to use a classifier to clean and curate a dataset. I.e. we want to remove datapoints that are not of a certain class or assign labels to a large corpus to make it easier to understand or organize the data. A very nice example of this kind of classifier based curation can be found in this paper.

In this case, we’ll use an existing dataset of arxiv papers and use our classifier to label the papers as containing a new dataset or not.

We’ll start by loading a smallish dataset which was used as the training set for the model.

from datasets import load_dataset

ds = load_dataset("davanstrien/arxiv-new-datasets", split="train")

ds.features

{'id': Value(dtype='string', id=None),
 'title': Value(dtype='string', id=None),
 'abstract': Value(dtype='string', id=None),
 'labels': ClassLabel(names=['new_dataset', 'no_new_dataset'], id=None),
 'explanations': Value(dtype='string', id=None)}

Example row

ds[6]

{'id': '2405.00026',
 'title': 'Enhancing Credit Card Fraud Detection A Neural Network and SMOTE\n  Integrated Approach',
 'abstract': '  Credit card fraud detection is a critical challenge in the financial sector,\ndemanding sophisticated approaches to accurately identify fraudulent\ntransactions. This research proposes an innovative methodology combining Neural\nNetworks (NN) and Synthet ic Minority Over-sampling Technique (SMOTE) to\nenhance the detection performance. The study addresses the inherent imbalance\nin credit card transaction data, focusing on technical advancements for robust\nand precise fraud detection. Results demonstrat e that the integration of NN\nand SMOTE exhibits superior precision, recall, and F1-score compared to\ntraditional models, highlighting its potential as an advanced solution for\nhandling imbalanced datasets in credit card fraud detection scenarios. This\nrese arch contributes to the ongoing efforts to develop effective and efficient\nmechanisms for safeguarding financial transactions from fraudulent activities.\n',
 'labels': 1,
 'explanations': 'The paper focuses on enhancing credit card fraud detection using a combination of neural networks and SMOTE. It evaluates its approach on existing datasets without introducing a new dataset. The study addresses imbalanced data in credit transactions but does not mention the creation of a new dataset.'}

def format_row_for_model(row):
    return f"TITLE: {row['title']} \n\nABSTRACT: {row['abstract']}"

outputs = llm.classify(format_row_for_model(ds[6]))

Full dataset

Now we’ll load the full dataset and format the data for the model. This is a dataset of arxiv papers (title, abstracts and other metadata) and we’ll filter it down to only include papers that are in the computer science category.

from datasets import load_dataset

dataset = load_dataset("librarian-bots/arxiv-metadata-snapshot", split="train")
dataset[0]

{'id': '1305.5211',
 'submitter': 'Hou-Biao Li',
 'authors': 'Qian-Ping Guo, Xian-Ming Gu and Hou-biao Li',
 'title': 'A note on the growth factor in Gaussian elimination for Higham matrices',
 'comments': '13 pages, 1 figures;',
 'journal-ref': None,
 'doi': None,
 'report-no': None,
 'categories': 'math.NA cs.NA',
 'license': 'http://arxiv.org/licenses/nonexclusive-distrib/1.0/',
 'abstract': "  The Higham matrix is a complex symmetric matrix A=B+iC, where both B and C\nare real, symmetric and positive definite and $\\mathrm{i}=\\sqrt{-1}$ is the\nimaginary unit. For any Higham matrix A, Ikramov et al. showed that the growth\nfactor in Gaussian elimination is less than 3. In this paper, based on the\nprevious results, a new bound of the growth factor is obtained by using the\nmaximum of the condition numbers of matrixes B and C for the generalized Higham\nmatrix A, which strengthens this bound to 2 and proves the Higham's conjecture.\n",
 'versions': [{'version': 'v1', 'created': 'Wed, 22 May 2013 18:00:17 GMT'},
  {'version': 'v2', 'created': 'Fri, 18 Apr 2025 14:14:30 GMT'}],
 'update_date': datetime.datetime(2025, 4, 21, 0, 0),
 'authors_parsed': [['Guo', 'Qian-Ping', ''],
  ['Gu', 'Xian-Ming', ''],
  ['Li', 'Hou-biao', '']]}

def contains_cs(row):
    return "cs" in row["categories"]

dataset = dataset.filter(contains_cs, num_proc=4)

Doing the same steps with Polars

I’m a big fan of Polars so in practice I would probably do the same steps in Polars.

from pathlib import Path

import polars as pl
from datasets import Dataset
from huggingface_hub import snapshot_download

snapshot_download(
    "librarian-bots/arxiv-metadata-snapshot",
    local_dir="arxiv-metadata-snapshot",
    allow_patterns=["*.parquet"],
    repo_type="dataset",
)
parquet_files = list(Path("arxiv-metadata-snapshot").rglob("*.parquet"))
df = pl.scan_parquet(parquet_files)
df = df.with_columns(
    pl.concat_str(
        [
            pl.lit("TITLE: "),
            pl.col("title"),
            pl.lit(" \n\nABSTRACT: "),
            pl.col("abstract"),
        ]
    ).alias("prompt")
)
df_cs = df.filter(
    pl.col("categories")
    .str.split(" ")
    .list.eval(pl.element().str.starts_with("cs."))
    .list.any()
)
df_cs_datasets = df_cs.filter(
    pl.col("title").str.contains("dataset")
    | pl.col("title").str.contains("benchmark")
    | pl.col("abstract").str.contains("dataset")
    | pl.col("abstract").str.contains("benchmark")
)

df_cs_datasets = df_cs_datasets.collect()

ds = Dataset.from_polars(df_cs_datasets)
ds

We now end up with a dataset with a new column prompt which contains the correctly formatted input for the model.

ds[0]["prompt"]

'TITLE: Good practices for a literature survey are not followed by authors while\n  preparing scientific manuscripts \n\nABSTRACT:   The number of citations received by authors in scientific journals has become\na major parameter to assess individual researchers and the journals themselves\nthrough the impact factor. A fair assessment therefore requires that the\ncriteria for selecting references in a given manuscript should be unbiased with\nrespect to the authors or the journals cited. In this paper, we advocate that\nauthors should follow two mandatory principles to select papers (later\nreflected in the list of references) while studying the literature for a given\nresearch: i) consider similarity of content with the topics investigated, lest\nvery related work should be reproduced or ignored; ii) perform a systematic\nsearch over the network of citations including seminal or very related papers.\nWe use formalisms of complex networks for two datasets of papers from the arXiv\nrepository to show that neither of these two criteria is fulfilled in practice.\n'

We can grab all of these prompts as a list

prompts = ds["prompt"]

Let’s see how many rows we have to classify

len(prompts)

211190

from toolz import partition_all
from tqdm.auto import tqdm

all_results = []

%%time
for batch in tqdm(list(partition_all(50_000, prompts))):
    all_results.append(llm.classify(batch))

CPU times: user 16min 22s, sys: 4.04 s, total: 16min 26s
Wall time: 16min 18s

Inference performance

~16 minutes to process 211,190 isn’t bad! Besides this, there is a minimal amount of boilerplate code to maintain. For many datasets, you could just grab the relevant column and pass it directly to vLLM as a list. Even if you could shave of a bit of time doing this in another way it’s very nice to just pass the prompts and let vLLM handle the rest.

Working with the results

We can now join all of the results into a single list

from toolz import concat

outputs = list(concat(all_results))

At the moment the raw outputs don’t contains the labels we want so we’ll create a small function that maps the outputs to the string label and returns the confidence in the prediction.

import torch
import torch.nn.functional as F


def get_top_label(output, label_map: dict[str, str]):
    """
    Given a ClassificationRequestOutput and a label_map (e.g. {'0': 'label0', ...}),
    returns the top predicted label (or None if not found) and its confidence score.
    """
    logits = torch.tensor(output.outputs.probs)
    probs = F.softmax(logits, dim=0)
    top_idx = torch.argmax(probs).item()
    top_prob = probs[top_idx].item()
    label = label_map.get(top_idx)  # Returns None if key is missing
    return label, top_prob

get_top_label(outputs[0], id2label)

('no_new_dataset', 0.9503649473190308)

Let’s create a new list with these results

clean_results = [get_top_label(output, id2label) for output in outputs]

We can quickly count the label frequencies

from collections import Counter

Counter(result[0] for result in clean_results)

Counter({'no_new_dataset': 182083, 'new_dataset': 29107})

ds = ds.add_column("is_new_dataset_label", [result[0] for result in clean_results])
ds = ds.add_column("label_confidence", [result[1] for result in clean_results])

ds[0]

{'id': '1005.3063',
 'submitter': 'Diego  Amancio Raphael',
 'authors': 'D.R. Amancio, M. G. V. Nunes, O. N. Oliveira Jr., L. da F. Costa',
 'title': 'Good practices for a literature survey are not followed by authors while\n  preparing scientific manuscripts',
 'comments': None,
 'journal-ref': 'Scientometrics, v. 90, p. 2, (2012)',
 'doi': '10.1007/s11192-012-0630-z',
 'report-no': None,
 'categories': 'physics.soc-ph cs.DL',
 'license': 'http://arxiv.org/licenses/nonexclusive-distrib/1.0/',
 'abstract': '  The number of citations received by authors in scientific journals has become\na major parameter to assess individual researchers and the journals themselves\nthrough the impact factor. A fair assessment therefore requires that the\ncriteria for selecting references in a given manuscript should be unbiased with\nrespect to the authors or the journals cited. In this paper, we advocate that\nauthors should follow two mandatory principles to select papers (later\nreflected in the list of references) while studying the literature for a given\nresearch: i) consider similarity of content with the topics investigated, lest\nvery related work should be reproduced or ignored; ii) perform a systematic\nsearch over the network of citations including seminal or very related papers.\nWe use formalisms of complex networks for two datasets of papers from the arXiv\nrepository to show that neither of these two criteria is fulfilled in practice.\n',
 'versions': [{'version': 'v1', 'created': 'Mon, 17 May 2010 21:45:47 GMT'},
  {'version': 'v2', 'created': 'Sun, 23 Sep 2012 00:49:13 GMT'}],
 'update_date': datetime.datetime(2012, 9, 25, 0, 0),
 'authors_parsed': [['Amancio', 'D. R.', ''],
  ['Nunes', 'M. G. V.', ''],
  ['Oliveira', 'O. N.', 'Jr.'],
  ['Costa', 'L. da F.', '']],
 'prompt': 'TITLE: Good practices for a literature survey are not followed by authors while\n  preparing scientific manuscripts \n\nABSTRACT:   The number of citations received by authors in scientific journals has become\na major parameter to assess individual researchers and the journals themselves\nthrough the impact factor. A fair assessment therefore requires that the\ncriteria for selecting references in a given manuscript should be unbiased with\nrespect to the authors or the journals cited. In this paper, we advocate that\nauthors should follow two mandatory principles to select papers (later\nreflected in the list of references) while studying the literature for a given\nresearch: i) consider similarity of content with the topics investigated, lest\nvery related work should be reproduced or ignored; ii) perform a systematic\nsearch over the network of citations including seminal or very related papers.\nWe use formalisms of complex networks for two datasets of papers from the arXiv\nrepository to show that neither of these two criteria is fulfilled in practice.\n',
 'is_new_dataset_label': 'no_new_dataset',
 'label_confidence': 0.9503649473190308}

Filtering the dataset

Now we have a new column with the labels, and a second column with the confidence in the label. We can now filter the dataset as needed. You could of course combine these filters in many ways depending on your use case.

df_with_labels = ds.to_polars()

df_with_labels.head(2)

df_with_labels = df_with_labels.lazy()

df_with_labels.filter(pl.col("is_new_dataset_label") == "new_dataset").collect().shape

(29107, 17)

df_predicted_as_new_dataset = df_with_labels.filter(
    pl.col("is_new_dataset_label") == "new_dataset"
)

df_predicted_as_new_dataset_high_confidence = df_predicted_as_new_dataset.filter(
    pl.col("label_confidence").ge(0.95)
)

df_predicted_as_new_dataset_high_confidence.show_graph()

df_predicted_as_new_dataset_high_confidence.collect()

Conclusion

In this post we’ve seen how to use vLLM to scale inference for ModernBERT based classifier models. I’m personally still very excited about the potential of smaller classifier models for a wide range of tasks and it’s really cool to see a very well known and used inference engine like vLLM extending it’s capabilities to include this kind of model. I’d be excited to see more models like this being added to vLLM and also to see the comunity shipping cool new classifiers for curating/cleaning/organizing large datasets.

Reasoning outside of math and science?

To date, reasoning models have primarily focused on math, coding, and science. One reason is that it is easier in these domains to have a verifiable answer or output to compare the LLM against.

Group Relative Policy Optimization (GRPO), introduced by the DeepSeek team in DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, is a variant of “Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO”.

When training a reasoning model using GRPO, we can use a reward function to reward correct completions. For example, we could reward correct completions based on ground truth answers. A pseudocode example of this might look like:

def check_answer(completions, **kwargs):
    return ["some reward" if completion == "ground truth answer"
    else "no reward" for completion in completions]

In this case, we have a ground-truth answer, and we want to reward completions that match this answer. When we have a single correct answer, this is straightforward. In other cases, we might need some other methods of “verification.” For example, for a code model, we can check if the code has valid syntax, complies, or passes a test.

However, nothing prevents us from using GRPO for other tasks. For example, the trl docs for the GRPOTrainer provide an example of using a reward function to reward more concise completions.

def reward_len(completions, **kwargs):
    return [-abs(20 - len(completion)) for completion in completions]

One of the key findings of the Deepseek paper is that during training, the model learns to allocate more thinking time to a problem by reevaluating its initial answer. This allows the model to self-correct its answer. This property can be useful for tasks related to maths. If the model is able to “think” through a solution in steps, it can avoid the pitfalls of single-step reasoning, i.e. immediately jumping to the answer. However, likely, this “reasoning” step can also be helpful for other tasks.

We have some nice community examples of this. For example, Ihor Stepanov has replicated the DeepSeek R1 approach for information extraction, which I’ll also cover in a series of posts starting with this one.

Distiling reasoning models for information extraction (and other tasks)

The DeepSeek team also showed in another paper DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning that it is possible to distill reasoning into smaller models. They showed that this works for distilling the general reasoning abilities of larger models into smaller ones, but it’s likely that we can also distil reasoning for specific tasks (and into even smaller models).

The steps to do this are roughly:

• Generate a dataset of reasoning traces
• Do supervised finetuning (SFT) on this dataset

Alternatively, we may want to use GRPO directly to train a model for a particular task. In this case, we usually still need an SFT step to train the model to understand the task and output we are looking for (although it may be possible to skip this step if the task is simple or you are starting from an instruction-tuned model that already understands the task).

One of the challenges of both approaches has been the cost of generating the reasoning traces. A new reasoning model from Qwen could potentially help with this.

QwQ-32B: A reasoning model that can democratize reasoning dataset generation?

Qwen recently released QwQ-32B:

QwQ is the reasoning model of the Qwen series. Compared with conventional instruction-tuned models, QwQ, which is capable of thinking and reasoning, can achieve significantly enhanced performance in downstream tasks, especially hard problems. QwQ-32B is the medium-sized reasoning model, which is capable of achieving competitive performance against state-of-the-art reasoning models, e.g., DeepSeek-R1, o1-mini. source

Whilst we already had access to models that could produce reasoning traces, QwQ-32B is much smaller than many of these models, making the cost and time involved in generating reasoning traces much more manageable. This may be especially important for more niche tasks where we will want to iterate on the dataset generation process. In the rest of this post, I’ll show how we can use QwQ-32B to generate a dataset for the task of model card data extraction.

example_output = {
    "base_model_mentions": None,
    "model_summary": "An open-weight text-to-speech model with 82 million parameters that delivers quality comparable to larger models while being faster and more cost-efficient.",
    "model_details": {
        "name": "Kokoro",
        "demo_url": "https://hf.co/spaces/hexgrad/Kokoro-TTS",
        "context_length": None,
        "parameter_count": 82000000,
    },
    "dataset_information": [
        {"name": "Koniwa", "url": "https://github.com/koniwa/koniwa", "type": "GitHub"}
    ],
    "technical_specifications": {
        "knowledge_cutoff": "2023-06",
        "inference_hardware": "Requires at least 4GB VRAM, runs on consumer GPUs like RTX 3060",
        "inference_libraries": ["Transformers"],
    },
    "card_quality": "good",
}

Next steps

Once we have a dataset like this, we can use it to train a model to extract structured information from model cards. There are various ways to do this.

{'base_model_mentions': [{'model_name': 'Qwen2-VL-7B-Instruct',
   'link': 'https://huggingface.co/datasets/allenai/olmOCR-mix-0225'}],
 'model_summary': 'A finetuned version of Qwen2-VL-7B-Instruct designed for generating images in a specific format using the olmOCR-mix-0225 dataset.',
 'model_details': {'name': 'olmOCR-7B-0225-preview',
  'demo_url': 'https://olmocr.allenai.org/',
  'context_length': None,
  'parameter_count': None},
 'dataset_information': [{'name': 'olmOCR-mix-0225',
   'url': 'https://huggingface.co/datasets/allenai/olmOCR-mix-0225',
   'type': 'Hugging Face'}],
 'technical_specifications': {'knowledge_cutoff': None,
  'inference_hardware': None,
  'inference_libraries': ['Transformers']},
 'card_quality': 'good'}

Okay, let's tackle this query step by step. The user provided a model card for QwQ-32B and wants specific structured information extracted. Let me go through each part of the query and the model card to ensure I don't miss anything.

First, looking at the **base_model_mentions** section. The task is to list models explicitly stated as base/foundation models used for finetuning. The card mentions that QwQ is based on Qwen2.5, whose code is from the latest Hugging Face transformers. The link is provided as https://huggingface.co/Qwen/QwQ-32B. So that should be in the list.

Next, the **model_summary** needs a single-sentence summary. The primary purpose is reasoning capacity, built on Qwen2.5, with 32B parameters. It has 32.5B parameters before post-training, and its performance on downstream tasks like math problems and hard questions. The summary should include the main function (reasoning), base model, parameters, and unique features like enhanced performance and context handling.

For **model_details**, the name is "Qwen/QwQ-32B" as per the repo name. Demo URL is https://huggingface.co/spaces/Qwen/QwQ-32B-Demo. They mention using the transformers library, so the Python code snippet's code variable is from transformers.context_length is Full 131,072, parameter count 32.5 billion (32.5B), and non-hidden parameters count 31.0B. But wait, the model has 32.5B before post-training, so parameter_count should be 32.5B. The context_length is mentioned in the requirements as Full 131,072 tokens. So for the parameters, the non-hidden part might need to be 31.0B.

**dataset_information** requires datasets used. The card mentions training stages pre-training with DeepSeek-R1 and o1-mini, but the main training used Qwen2.5's technical report. The dataset for post-training is not specified here, so maybe the datasets are already covered under the base model or other parts of the card. The training data isn't explicitly listed except mentioning post-training and the reports of those using their platforms. Since the exact training datasets aren't named here, perhaps there's no direct dataset info provided. Wait, the requirements mention enabling YaRN and using vLLM, which might involve specific datasets, but without explicit dataset names, maybe there's no direct dataset info here. So dataset_information would be null? Or do I look into the references? The card says "Dealing with their [fellow researchers'] work, they mention their own results and not those of others on the training data. " So it might not have explicit dataset names, so maybe leave it as an empty array?

Technical specifications: knowledge_cutoff is not explicitly stated. The training data's latest date isn't mentioned, so it's null. Inference hardware isn't specified except mentioning vLLM and the author's mention of using vLLM's setup. The requirements say to use the transformers library, but the example code uses PyTorch, but the requirements apply to the model's architecture. The card says "transformers<4.37.0", so probably uses the current version. The inference_libraries are mentioned as "transformers" since it's part of the code snippet. So the libraries are transformers.

**card_quality** assessment. Let's look at the criteria:

Visualizing the data

Once we’ve finishes training we can work with the completions dataset using Polars, Pandas, etc. This can help give us more insight into how the different reward functions are behaving and how we might modify our traing process.

import polars as pl

# Login using e.g. `huggingface-cli login` to access this dataset
df = pl.read_parquet("hf://datasets/davanstrien/test-logs/**/*.parquet")
df.describe()

We can already see that we need to do something different if we want more shouting. We can see that the emojis reward is triggered sometimes but not that often.

We can also do things like plot the length reward over time.

import polars as pl
import altair as alt

df_sorted = df.sort("step")
df_with_avg = df_sorted.with_columns(
    pl.col("reward_reward_len")
    .rolling_mean(window_size=10, min_samples=1)
    .alias("rolling_avg")
)

# Now create separate charts and combine them
scatter_chart = (
    alt.Chart(df_with_avg)
    .mark_circle(opacity=0.6)
    .encode(
        x=alt.X("step:Q", title="Step"),
        y=alt.Y("reward_reward_len:Q", title="Length Reward Score"),
        tooltip=["step", "reward_reward_len"],
    )
)

line_chart = (
    alt.Chart(df_with_avg)
    .mark_line(color="purple", size=2)
    .encode(x="step:Q", y="rolling_avg:Q")
)

combined_chart = (
    (scatter_chart + line_chart)
    .properties(
        width=800,
        height=400,
        title="Length Rewards Over Time with 10-step Moving Average",
    )
    .configure_axis(labelFontSize=12, titleFontSize=14)
)

combined_chart

heatmap_data = df.with_columns([
    pl.col('step').cast(pl.Int32) // 10 * 10
]).group_by('step').agg([
    pl.mean('reward_reward_len').alias('avg_len_reward'),
    pl.mean('reward_reward_shouting').alias('avg_shouting_reward'),
    pl.mean('reward_reward_emojis').alias('avg_emoji_reward')
])

heatmap_long = heatmap_data.unpivot(
    index=['step'],
    on=['avg_len_reward', 'avg_shouting_reward', 'avg_emoji_reward'],
    variable_name='reward_type',
    value_name='value'
)

heatmap = alt.Chart(heatmap_long).mark_rect().encode(
    x='step:O',
    y='reward_type:N',
    color=alt.Color('value:Q', scale=alt.Scale(scheme='viridis'))
).properties(
    width=800,
    height=200,
    title='Reward Types Intensity Over Time'
)
heatmap

histogram = alt.Chart(df).transform_fold(
    ['reward', 'reward_reward_len', 'reward_reward_shouting', 'reward_reward_emojis'],
    as_=['Reward Type', 'Value']
).mark_bar(opacity=0.7).encode(
    alt.X('Value:Q', bin=alt.Bin(maxbins=30)),
    alt.Y('count():Q'),
    alt.Color('Reward Type:N'),
    alt.Row('Reward Type:N')
).properties(
    width=600,
    height=150,
    title='Distribution of Different Reward Types'
)
histogram