Load, Stream and Filter a Hugging Face Dataset in Python

The datasets library turns any dataset repo on the Hub into a table you can filter and transform in a few lines, either fully downloaded or streamed row by row. Examples use rotten_tomatoes (8,530 training rows) and, for streaming, the multi-terabyte FineWeb.

1. Install and load a split

pip install -U datasets

from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
print(ds)

Downloads the Parquet files once into the Hugging Face cache and prints Dataset({features: ['text', 'label'], num_rows: 8530}). Without split= you get a DatasetDict holding train, validation and test.

2. Look before you process

from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
print(ds.features)         # column types: text is a string, label is ClassLabel(neg, pos)
print(ds[0])               # first row as a dict
print(ds[:3]["text"])      # a slice returns columns

features tells you what each column holds before you write code against it; ClassLabel means the integer 1 is pos.

3. Filter, map, keep only what you need

from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")

positive = ds.filter(lambda row: row["label"] == 1)

def add_word_count(batch):
    return {"n_words": [len(t.split()) for t in batch["text"]]}

positive = positive.map(add_word_count, batched=True)
short = positive.filter(lambda row: row["n_words"] < 12).select_columns(["text", "n_words"])
print(short)

filter keeps rows where the function returns True; map(batched=True) gets 1,000 rows at a time as lists, which is much faster than row by row; select_columns drops the rest. Every call returns a new dataset rather than editing in place.

4. Stream instead of downloading

from datasets import load_dataset

stream = load_dataset("HuggingFaceFW/fineweb", split="train", streaming=True)
english = stream.filter(lambda r: r["language_score"] > 0.95).select_columns(["text", "url"])

for row in english.take(3):
    print(row["url"])

streaming=True returns an IterableDataset: nothing is downloaded up front, rows arrive as you iterate, and filter, map and take(n) run lazily. No random access and no len(), so shuffle with shuffle(buffer_size=...) before take.

5. Save the result

from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
short = ds.filter(lambda row: len(row["text"].split()) < 12)

short.to_parquet("short.parquet")    # columnar, compressed, reads back in pandas/polars/DuckDB
short.to_json("short.jsonl")         # one JSON object per line
short.save_to_disk("short_arrow")    # Arrow folder, reload with load_from_disk()
# short.push_to_hub("your-username/short-reviews")  # needs `hf auth login` with a write token

Writes the filtered rows in three formats; push_to_hub uploads them to a dataset repo as Parquet. See the processing docs for to_csv, to_sql and to_pandas.

Only need the raw files? hf download cornell-movie-review-data/rotten_tomatoes --repo-type dataset --local-dir ./rt fetches the Parquet files with no Python at all; see the hf CLI guide.

If it fails

• ValueError: Config name is missing. The repo has several subsets; the message lists them. Pass one: load_dataset("nyu-mll/glue", "mrpc", split="train").
• DatasetNotFoundError: Dataset '…' doesn't exist on the Hub or cannot be accessed. Check the owner/name spelling; for a gated or private dataset accept its terms on the Hub page and run hf auth login.
• TypeError: object of type 'IterableDataset' has no len() You are streaming. Use take(n) to sample, or drop streaming=True if the data fits on disk.

---

Next: embeddings explained with code · minimal RAG in Python · what an embedding is.