Reddit dataset: how to build one from live Reddit data
How to build a Reddit dataset: pick communities and queries, collect posts and comments as JSON, page through results, store the date, and what to check before use.
Piyushh · November 7, 2026 · 2 min read
A Reddit dataset is a table of Reddit posts or comments you collected for a purpose: a model, a study, a market scan. You can download an archive, use a published dataset, or build your own from live data. If you build your own, you choose the exact communities and dates you need.
Three ways to get Reddit data
- A public archive is good for history. See the Reddit archive guide and the comparison of Pushshift, PullPush and Arctic Shift.
- A published research dataset is fixed and citable, but it only holds what its authors collected.
- Live collection gets you fresh data for your own queries, and it is what the rest of this post covers.
Plan the collection first
Write down which subreddits you pulled, the search words, the time window, and why you chose them. That list is what makes your dataset defensible. If you plan to train a model on it, check the terms that cover the content and your use of it. This is not legal advice.
Collect posts and page through results
curl "https://api.chatterapi.com/v1/reddit/search?q=data+engineering&time=year" \
-H "Authorization: Bearer $CHATTERAPI_KEY"
Each response has a data array and a page.nextCursor. Loop until hasMore is false. Save the raw JSON with the date you ran it, since scores and comment counts change over time.
To cover a whole community, call subreddit posts. For comments, call a thread for each post you care about. Data science with the Reddit API covers what to do next.
Turn it into a table
In Python, load the saved responses and run pandas.json_normalize on the data array. Fields such as score, numComments and createdAt become columns. A post can come back from more than one query, so deduplicate on the post id.
What to check before you use it
- Removed content is missing, which can bias the sample.
- Scores are a snapshot from the day you collected them.
- Posts about identifiable people may need an ethics review.
To export a single thread to a file without writing code, use the Reddit comment exporter. Methods notes are covered in the academic research use case.