The datasets library turns any dataset repo on the Hub into a table you can filter and transform in a few lines, either fully downloaded or streamed row by row. Examples use rotten_tomatoes (8,530 training rows) and, for streaming, the multi-terabyte FineWeb.
1. Install and load a split
pip install -U datasets
from datasets import load_dataset
ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
print(ds)
Downloads the Parquet files once into the Hugging Face cache and prints Dataset({features: ['text', 'label'], num_rows: 8530}). Without split= you get a DatasetDict holding train, validation and test.
2. Look before you process
from datasets import load_dataset
ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
print(ds.features) # column types: text is a string, label is ClassLabel(neg, pos)
print(ds[0]) # first row as a dict
print(ds[:3]["text"]) # a slice returns columns
features tells you what each column holds before you write code against it; ClassLabel means the integer 1 is pos.
3. Filter, map, keep only what you need
from datasets import load_dataset
ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
positive = ds.filter(lambda row: row["label"] == 1)
def add_word_count(batch):
return {"n_words": [len(t.split()) for t in batch["text"]]}
positive = positive.map(add_word_count, batched=True)
short = positive.filter(lambda row: row["n_words"] < 12).select_columns(["text", "n_words"])
print(short)
filter keeps rows where the function returns True; map(batched=True) gets 1,000 rows at a time as lists, which is much faster than row by row; select_columns drops the rest. Every call returns a new dataset rather than editing in place.
4. Stream instead of downloading
from datasets import load_dataset
stream = load_dataset("HuggingFaceFW/fineweb", split="train", streaming=True)
english = stream.filter(lambda r: r["language_score"] > 0.95).select_columns(["text", "url"])
for row in english.take(3):
print(row["url"])
streaming=True returns an IterableDataset: nothing is downloaded up front, rows arrive as you iterate, and filter, map and take(n) run lazily. No random access and no len(), so shuffle with shuffle(buffer_size=...) before take.
5. Save the result
from datasets import load_dataset
ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
short = ds.filter(lambda row: len(row["text"].split()) < 12)
short.to_parquet("short.parquet") # columnar, compressed, reads back in pandas/polars/DuckDB
short.to_json("short.jsonl") # one JSON object per line
short.save_to_disk("short_arrow") # Arrow folder, reload with load_from_disk()
# short.push_to_hub("your-username/short-reviews") # needs `hf auth login` with a write token
Writes the filtered rows in three formats; push_to_hub uploads them to a dataset repo as Parquet. See the processing docs for to_csv, to_sql and to_pandas.
Only need the raw files? hf download cornell-movie-review-data/rotten_tomatoes --repo-type dataset --local-dir ./rt fetches the Parquet files with no Python at all; see the hf CLI guide.
If it fails
ValueError: Config name is missing.The repo has several subsets; the message lists them. Pass one:load_dataset("nyu-mll/glue", "mrpc", split="train").DatasetNotFoundError: Dataset '…' doesn't exist on the Hub or cannot be accessed.Check theowner/namespelling; for a gated or private dataset accept its terms on the Hub page and runhf auth login.TypeError: object of type 'IterableDataset' has no len()You are streaming. Usetake(n)to sample, or dropstreaming=Trueif the data fits on disk.
Next: embeddings explained with code · minimal RAG in Python · what an embedding is.