# Load, Stream and Filter a Hugging Face Dataset in Python

> Load a Hub dataset split, inspect its features, stream one too big to download, filter and batch-map it, then save it as Parquet or JSONL.

- Canonical: https://guides-ai.com/guides/hugging-face-datasets-load-stream-filter/
- Plate 16.08 · Topic: Hugging Face (https://guides-ai.com/topics/huggingface/)
- Published: 12 Oct 2026 · 3 min read
- Source site: guides-ai — https://guides-ai.com/

The `datasets` library turns any dataset repo on the Hub into a table you can filter and transform in a few lines, either fully downloaded or streamed row by row. Examples use [rotten_tomatoes](https://huggingface.co/datasets/cornell-movie-review-data/rotten_tomatoes) (8,530 training rows) and, for streaming, the multi-terabyte [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb).

## 1. Install and load a split

```bash
pip install -U datasets
```

```python
from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
print(ds)
```

Downloads the Parquet files once into the Hugging Face cache and prints `Dataset({features: ['text', 'label'], num_rows: 8530})`. Without `split=` you get a `DatasetDict` holding train, validation and test.

## 2. Look before you process

```python
from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
print(ds.features)         # column types: text is a string, label is ClassLabel(neg, pos)
print(ds[0])               # first row as a dict
print(ds[:3]["text"])      # a slice returns columns
```

`features` tells you what each column holds before you write code against it; `ClassLabel` means the integer `1` is `pos`.

## 3. Filter, map, keep only what you need

```python
from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")

positive = ds.filter(lambda row: row["label"] == 1)

def add_word_count(batch):
    return {"n_words": [len(t.split()) for t in batch["text"]]}

positive = positive.map(add_word_count, batched=True)
short = positive.filter(lambda row: row["n_words"] < 12).select_columns(["text", "n_words"])
print(short)
```

`filter` keeps rows where the function returns `True`; `map(batched=True)` gets 1,000 rows at a time as lists, which is much faster than row by row; `select_columns` drops the rest. Every call returns a new dataset rather than editing in place.

## 4. Stream instead of downloading

```python
from datasets import load_dataset

stream = load_dataset("HuggingFaceFW/fineweb", split="train", streaming=True)
english = stream.filter(lambda r: r["language_score"] > 0.95).select_columns(["text", "url"])

for row in english.take(3):
    print(row["url"])
```

`streaming=True` returns an `IterableDataset`: nothing is downloaded up front, rows arrive as you iterate, and `filter`, `map` and `take(n)` run lazily. No random access and no `len()`, so shuffle with `shuffle(buffer_size=...)` before `take`.

## 5. Save the result

```python
from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
short = ds.filter(lambda row: len(row["text"].split()) < 12)

short.to_parquet("short.parquet")    # columnar, compressed, reads back in pandas/polars/DuckDB
short.to_json("short.jsonl")         # one JSON object per line
short.save_to_disk("short_arrow")    # Arrow folder, reload with load_from_disk()
# short.push_to_hub("your-username/short-reviews")  # needs `hf auth login` with a write token
```

Writes the filtered rows in three formats; `push_to_hub` uploads them to a dataset repo as Parquet. See the [processing docs](https://huggingface.co/docs/datasets/process) for `to_csv`, `to_sql` and `to_pandas`.

Only need the raw files? `hf download cornell-movie-review-data/rotten_tomatoes --repo-type dataset --local-dir ./rt` fetches the Parquet files with no Python at all; see the [hf CLI guide](/guides/hugging-face-cli-download-models/).

## If it fails

- **`ValueError: Config name is missing.`** The repo has several subsets; the message lists them. Pass one: `load_dataset("nyu-mll/glue", "mrpc", split="train")`.
- **`DatasetNotFoundError: Dataset '…' doesn't exist on the Hub or cannot be accessed.`** Check the `owner/name` spelling; for a gated or private dataset accept its terms on the Hub page and run `hf auth login`.
- **`TypeError: object of type 'IterableDataset' has no len()`** You are streaming. Use `take(n)` to sample, or drop `streaming=True` if the data fits on disk.

---

Next: [embeddings explained with code](/guides/embeddings-explained-with-code/) · [minimal RAG in Python](/guides/rag-with-python-minimal/) · [what an embedding is](/glossary/#embeddings).