Twitter dataset: how to collect and save X posts for analysis
How to build a Twitter dataset from live X data: define the query, collect posts as JSON, page with cursors, save each run with its date, and what to check first.
Piyushh · November 7, 2026 · 1 min read
A Twitter dataset is a set of X posts you collected for analysis: sentiment, topics, a class project. You can use a published dataset or collect your own. If you collect your own, you choose the query and can record the day you ran it.
Collect posts by keyword
curl "https://api.chatterapi.com/v1/x/tweets/search?q=open+source+llm" \
-H "Authorization: Bearer $CHATTERAPI_KEY"
The response holds a data array of posts, each with its text, author, time and engagement counts, plus a page.nextCursor. Keep paging with the cursor until you have enough rows. For time filters, see search tweets by date.
Save it properly
Store the raw JSON along with the date and the exact query, because a methods section needs both. Flatten with pandas.json_normalize and deduplicate on the post id.
Other sources of posts
- One account's recent posts: X timeline scraper.
- Replies to a post: replies exporter.
- Followers of an account: follower exporter.
For student or research work, see Twitter API for students and for academic research. If you're writing code, see Twitter API in Python.
Limits
Results are whatever X returns on the day you make the call, so deleted posts and protected accounts won't be in them. Check your institution's rules before you collect posts about named people.