Skip to content

How to collect social media data from Reddit and X

How to collect social media data from Reddit and X: four methods compared, from manual export to a scraper to an API, and a short script that runs on a schedule.

Piyushh · November 13, 2026 · 4 min read

Four routes for collecting social media data ending in one organised drawer.

The best way to collect social media data for a repeat job is an API that returns it as JSON. You send a query and get posts, authors and scores back, and nobody has to maintain a scraper when the page layout changes. This guide covers only Reddit and X, because those are the platforms ChatterAPI serves. Instagram, TikTok and LinkedIn are not covered.

Below are the four ways to get social media data and what each costs you in time, followed by a script that collects from both platforms and saves the result.

Four ways to get social media data

MethodSetup timeUpkeepGood for
Manual export and copy-pasteNoneYour time, every timeOne question, one afternoon
A scraper you writeDaysBreaks when pages changeTeams that want full control
The platform's own APIHours to daysBilling, limits, app approvalA first-party relationship with the platform
A hosted data APIMinutesNone on your sideRepeat jobs, agents, dashboards

The rest of the post covers each row in turn.

Manual social media data extraction

Open the search box on Reddit or X, run the query, and copy what you need into a sheet. That works until you need the same pull every Monday.

Manual is still the right way to check an idea. To see what a query returns before you commit to anything, the free X tweet search tool and the Reddit post search tool show the rows in your browser. Test your keywords there, then automate.

Build a scraper to extract data from social media

A scraper fetches pages and parses the HTML or the JSON hidden inside them. You get full control, and every breakage is yours to fix.

The usual costs are a login wall, page structure that keeps changing, blocked IPs, and rate limits you only learn about once you're blocked. Reddit and X both set rules on automated access, so read their terms before you point a scraper at either one. Our guide on Reddit API vs scraping covers the trade-off for Reddit.

Scraping makes sense when the data is unusual and no API offers it. For plain search, timelines and thread data, it's a lot of work for something you can rent.

Using APIs for data collection on social media

An API turns the job into a request and a response. There are two kinds to choose from.

The first is the platform's own API. You get a direct relationship and, for X, access to first-party features. You also need a developer account and an app, and you pay in the platform's units under its rate limits. For X, the current price list is on X's pricing page, and we summarise it in is the X API free. If you are weighing it up for Twitter API data collection specifically, start there.

The second is a hosted data API: one key, one credit balance, JSON back, and no developer app to set up. ChatterAPI is independent and is not affiliated with Reddit or X. Reddit is read-only. X has reads, plus actions from accounts you connect.

A twitter data collection tool, as code

This call searches X. On a live key it costs 2 credits per page of about 20 tweets:

curl -s "https://api.chatterapi.com/v1/x/tweets/search?q=crm%20recommendation&sort=latest" \
  -H "Authorization: Bearer $CHATTERAPI_KEY"

Every response uses the same envelope: data for the rows, page with nextCursor and hasMore, and credits with what was charged. Below is the first row of a real response recorded with a test key on 2026-10-13, trimmed to the fields that matter:

{
  "data": [
    {
      "id": "2099495333231288346",
      "text": "meta ads: how important is the “connect to your crm” recommendation? ...",
      "createdAt": "2026-09-14T13:47:48.000Z",
      "likes": 2, "retweets": 0, "replies": 2, "views": 138,
      "author": { "handle": "HesomParhizkar", "followers": 584 }
    }
  ],
  "page": { "nextCursor": "AcWNIo1WFWYR7CN…", "hasMore": true }
}

A test key is free and returns the same recorded real response no matter what you send, so you can build against the shape first. The X search endpoint docs list every parameter, including since, until and lang.

Get social media data from Reddit and X in one script

With a hosted API, adding the second platform costs you almost nothing extra. You keep the same key and the same envelope, and only the path changes:

import os, json, requests

BASE = "https://api.chatterapi.com/v1"
H = {"Authorization": f"Bearer {os.environ['CHATTERAPI_KEY']}"}

def rows(path, **params):
    r = requests.get(f"{BASE}{path}", headers=H, params=params, timeout=60)
    r.raise_for_status()
    return r.json()["data"]

query = "crm recommendation"
out = []
for t in rows("/x/tweets/search", q=query, sort="latest"):
    out.append({"platform": "x", "date": t["createdAt"], "text": t["text"], "url": t["url"]})
for p in rows("/reddit/search", q=query, sort="new", time="month"):
    out.append({"platform": "reddit", "date": p["createdAt"], "text": p["title"], "url": p["url"]})

with open("mentions.json", "w") as f:
    json.dump(out, f, indent=2)
print(len(out), "rows")

On a live key that costs 4 credits per run, 2 for each side. To get both in one request, GET /v1/search queries Reddit and X together for the same 4 credits. The free search everywhere tool runs it in the browser.

Run the script on a schedule (cron, GitHub Actions or any job runner) and dedupe on url. Every list paginates the same way: pass page.nextCursor back as cursor until hasMore is false. Cursors are opaque, so don't try to build one by hand.

What an API will not give you

Know the limits before you build on this.

  • It works by polling. You send a request and get an answer back, and nothing is pushed to you as a stream.
  • Search returns only what the platform's own search returns. Deleted and removed content is gone.
  • It covers public data only. Private accounts and private messages are out of scope.
  • It covers two platforms. For a third, you need another source.

Before you store or republish what you collect, check each platform's rules for your use case, especially if it's for model training. For the full picture of what a scraping API does, see our page on the social media scraping API.

Pick a method

For a one-off job, do it by hand. If you want to own every detail, write the scraper and budget for the upkeep. If you need Reddit and X data on a schedule without maintaining anything, call an API.

You can run one Reddit and X search now without signing up, using the search everywhere tool. When the rows look right, sign up with Google for a key. The 7-day free trial includes 1,000 credits, enough for 250 runs of the script above. Plans start at $19 a month for 35,000 credits, as listed on the pricing page.

FAQ

What is the best way to collect social media data?
For repeat jobs, a data API. You send a query, you get JSON back, and someone else keeps the access working. Manual export is fine for one-off questions, and a home-made scraper makes sense only if you want to own the maintenance.
How do I extract data from social media without writing a scraper?
Call an API that already does the extraction. With ChatterAPI one GET request returns Reddit posts or tweets as JSON with author, score and date fields, and a cursor for the next page.
Is there a Twitter data collection tool I can try without signing up?
Yes. The free X tweet search and X timeline scraper tools on ChatterAPI run a query in the browser and show the rows, so you can check the data before you write any code.
How do I get social media data from Reddit and X in one script?
Use one HTTP client and one key. Loop over the Reddit search endpoint and the X search endpoint, tag each row with its platform, and write both to the same file. The cross-platform search endpoint does both in a single call for 4 credits.
Does this cover Instagram, TikTok and LinkedIn?
No. ChatterAPI covers Reddit and X only. This guide is about those two platforms.

More on Monitoring and social listening

Try it free for 7 days.

1,000 credits to make your first calls. One key for Reddit and X, over REST or MCP.