How to scrape Twitter and Reddit in 2026: one script for both

To scrape Twitter and Reddit for the same keyword in 2026, run a tweet search and a Reddit search side by side, map both results into one schema and write a single CSV. Doing the collection yourself means fighting two platforms at once: X puts search behind a login and checks whether your client looks like a real browser, and Reddit has shut its unauthenticated endpoints one after another. This guide is for jobs that need both platforms together, such as social listening, lead finding, research datasets and AI agents.

You'll get a working Python script that pulls tweets and Reddit posts for one keyword, normalises them and writes mentions.csv. Then there's a one-call version, what it costs to run as a daily job, and the legal notes with sources. If you only need X, XAutoDM's guide on how to scrape Twitter (X) data with an API goes deeper.

Why DIY scrapers break on both platforms

Reddit and X block scrapers in different ways, so a scraper that covers both has two separate sets of breakage to maintain.

X uses a login wall and fingerprints clients. On June 30, 2023, Twitter started requiring a login to view tweets (Bleeping Computer). Elon Musk called it a "temporary emergency measure" because "we were getting data pillaged so much that it was degrading service for normal users" (Deccan Herald). Some logged-out viewing returned a few days later (Fox Business), but tools built on logged-out access stayed broken. snscrape, the tool most old tutorials use, has an open issue from that day reporting that every Twitter scrape except trends fails (issue #996), and its last commit is from June 2023. Logging in brings its own problems. The README of the-convocation's open-source twitter-scraper says login requests can hit 403 Forbidden because "standard Node.js TLS fingerprints are detected as non-browser clients," and warns that "any account you log into with this library is subject to being banned at any time" (README, checked 2026-10-14). XAutoDM's post on why generic scrapers can't scrape Twitter goes further into rendering and rate limits.

Reddit has closed its routes one at a time. It shut down unauthenticated .json endpoints in May 2026, restricted Old Reddit to logged-in users, and announced it stops accepting new public API requests on October 31, 2026. Our guide on how to scrape Reddit tests each route and shows what comes back.

Developers asking about this in r/webscraping describe the same thing. One, building a search tool, wrote that reverse engineering X's internal API was doable, but "the problem becomes how to do this without being banned (likely need rotating residential proxies and many accounts etc)" (u/Pr3fix). Another, working on social listening across X, Reddit and review sites, dropped the official routes because of "extreme API costs (Twitter/X), performance issues, or API restrictions on scraping for commercial use (Reddit)" (u/KDavidP1987).

Ways to scrape Twitter and Reddit together

You have three realistic setups, and none of them fits every job.

Separate scrapers suit a small job where you need a field nothing else returns. In a 2024 r/webscraping thread, one reply said Playwright "mostly works" for X search, but "it can take +5 minutes to get 400 posts, so anything real time is ruled out" (u/bestjaegerpilot).

Apify actors fit if you already use Apify or prefer filling in a form to writing code. As listed on Oct 14, 2026, Tweet Scraper V2 starts at $0.40 per 1,000 tweets, and every query has to return at least 50 tweets. Reddit Scraper Lite starts at $3.40 per 1,000 results and recommends residential proxies. You end up with two datasets in two shapes, so the merge step is still yours to write. Our pages on the Apify Twitter scraper and snscrape cover each tool separately.

One API suits recurring jobs. ChatterAPI is one option: one key and one credit balance for Reddit and X, over REST or a hosted MCP server, and you don't need a Reddit or X developer account. It is independent and not affiliated with Reddit or X. Our roundup of the best Reddit scrapers compares other Reddit options, using prices from each provider's own page.

A Python script to scrape tweets and Reddit posts into one CSV

The script calls two endpoints with the same keyword:

• x.search (GET /v1/x/tweets/search, 3 credits a page of about 20 tweets). Parameters include q (keywords and X operators), sort (latest or top), since and until (YYYY-MM-DD), lang and cursor.
• reddit.search (GET /v1/reddit/search, 2 credits a page of about 25 posts). Parameters include q, subreddit, sort (relevance, new, hot, top, comments), time (hour to all) and cursor.

First, look at what each one returns. The samples below come from a test key, which returns one real recorded response per endpoint for free. That is why we use the keyword those samples match: crm recommendation. Here is one tweet, trimmed:

curl -G "https://api.chatterapi.com/v1/x/tweets/search" \
  -H "Authorization: Bearer $CHATTERAPI_KEY" \
  --data-urlencode 'q=crm recommendation'

{
  "data": [
    {
      "id": "2099495333231288346",
      "text": "meta ads: how important is the “connect to your crm” recommendation? ...",
      "url": "https://x.com/HesomParhizkar/status/2099495333231288346",
      "createdAt": "2026-09-14T13:47:48.000Z",
      "lang": "en",
      "likes": 2,
      "retweets": 0,
      "replies": 2,
      "views": 138,
      "author": { "handle": "HesomParhizkar", "followers": 584, "canDm": true }
    }
  ],
  "page": { "nextCursor": "AcWNIo1WFWYR…", "hasMore": true },
  "credits": { "charged": 0, "remaining": 717 },
  "requestId": "req_01M4X4TZAZP9QF4W3NFD",
  "cached": true
}

And one Reddit post from the matching reddit.search sample, trimmed:

{
  "id": "1wdihhj",
  "fullname": "t3_1wdihhj",
  "title": "What CRM should a small business use?",
  "body": "We’re a small B2B business with 6 people and sales is still pretty founder led ...",
  "author": "NotedWhisky",
  "subreddit": "CRM",
  "url": "https://www.reddit.com/r/CRM/comments/1wdihhj/what_crm_should_a_small_business_use/",
  "score": 51,
  "numComments": 186,
  "createdAt": "2026-09-11T14:52:16.000Z"
}

A tweet has likes and a nested author. A Reddit post has title plus body, a net score and a subreddit. The normaliser below maps both shapes into one row format. We ran it against the test key before publishing:

import csv, os, requests

API = "https://api.chatterapi.com/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['CHATTERAPI_KEY']}"}


def get(path, **params):
    r = requests.get(f"{API}{path}", headers=HEADERS, params=params, timeout=60)
    r.raise_for_status()
    return r.json()


def pages(path, max_pages, **params):
    """Yield items from up to max_pages pages, following page.nextCursor."""
    cursor = None
    for _ in range(max_pages):
        body = get(path, **params, **({"cursor": cursor} if cursor else {}))
        yield from body["data"]
        cursor = body["page"].get("nextCursor")
        if not body["page"].get("hasMore") or not cursor:
            break


def from_tweet(t):
    return {
        "platform": "x",
        "id": t["id"],
        "url": t["url"],
        "author": t["author"]["handle"],
        "community": "",
        "text": t["text"],
        "engagement": t["likes"] or 0,      # likes
        "replies": t["replies"] or 0,
        "created_at": t["createdAt"],
    }


def from_post(p):
    return {
        "platform": "reddit",
        "id": p["fullname"],
        "url": p["url"],
        "author": p["author"],
        "community": p["subreddit"],
        "text": f'{p["title"]}\n\n{p["body"] or ""}'.strip(),
        "engagement": p["score"] or 0,      # upvotes minus downvotes
        "replies": p["numComments"] or 0,
        "created_at": p["createdAt"],
    }


def scrape(keyword, max_pages=3):
    rows = [from_tweet(t) for t in pages("/x/tweets/search", max_pages, q=keyword, sort="latest")]
    rows += [from_post(p) for p in pages("/reddit/search", max_pages, q=keyword, sort="new", time="month")]
    seen, unique = set(), []
    for row in rows:
        key = (row["platform"], row["id"])
        if key not in seen:
            seen.add(key)
            unique.append(row)
    return sorted(unique, key=lambda r: r["created_at"] or "", reverse=True)


if __name__ == "__main__":
    rows = scrape("crm recommendation")
    with open("mentions.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)
    print(f"wrote {len(rows)} rows: "
          f"{sum(r['platform'] == 'x' for r in rows)} tweets, "
          f"{sum(r['platform'] == 'reddit' for r in rows)} Reddit posts")

On the test key it printed wrote 45 rows: 20 tweets, 25 Reddit posts. The test key serves the same recorded page every time, so dedupe collapsed three pages per platform into one. On a live key, three pages per platform comes to up to about 135 rows for 15 credits.

Three choices in that script matter:

1. It dedupes on platform plus id. Tweet ids and Reddit fullnames (t3_...) never collide, but the same item can appear on two pages of a fresh search.
2. It keeps engagement per platform. Reddit's score is upvotes minus downvotes and X's is likes. Those are different units, so don't rank the merged CSV by score. Sort by date, or rank within each platform.
3. It drops some fields you may want. The X response also carries views, retweets, author followers and canDm, and the Reddit response carries numComments and nsfw. If you need them, add columns for them.

To see the output before writing any code, try the free X tweet search and Reddit post search tools, which make the same calls from a browser.

The one-call version with everywhere.search

If you only need the shared fields, you can skip the normaliser. everywhere.search (GET /v1/search) runs both searches in parallel and returns one merged list in one shape. A page costs 5 credits (2 for the Reddit side, 3 for the X side), and limit sets the number of rows per platform (1 to 50, default 20).

import csv, os, requests

r = requests.get(
    "https://api.chatterapi.com/v1/search",
    headers={"Authorization": f"Bearer {os.environ['CHATTERAPI_KEY']}"},
    params={"q": "crm recommendation", "sort": "new", "time": "all", "limit": 20},
    timeout=60,
)
r.raise_for_status()
body = r.json()

for w in body.get("warnings", []):
    print("warning:", w)   # one platform didn't answer; you paid only for the other

fields = ["platform", "kind", "id", "url", "author", "community", "score", "createdAt", "text"]
with open("mentions.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
    writer.writeheader()
    writer.writerows(body["data"])

print(len(body["data"]), "rows, charged", body["credits"]["charged"], "credits")

On the test key it printed 40 rows, charged 0 credits: 20 tweets and 20 Reddit posts, newest first, going from September 2026 back to 2019. Each item looks like this:

{
  "platform": "reddit",
  "kind": "post",
  "id": "t3_1wdihhj",
  "url": "https://www.reddit.com/r/CRM/comments/1wdihhj/what_crm_should_a_small_business_use/",
  "text": "What CRM should a small business use?\n\nWe’re a small B2B business with 6 people ...",
  "author": "NotedWhisky",
  "authorUrl": "https://www.reddit.com/user/NotedWhisky/",
  "community": "CRM",
  "score": 51,
  "createdAt": "2026-09-11T14:52:16.000Z"
}

Pick based on the job. The one-call version means less code, and it handles a platform outage for you: if one side fails, you get the other side's results plus a warnings entry, and you pay only for what came back. The two-call script costs the same per keyword (3 + 2 credits) but keeps the fields only one platform has, such as views, follower counts and comment counts. Our post on searching Reddit and X in one call explains the merge and sort rules, and the cross-platform docs list every endpoint. To try it in a browser, use the free search everywhere tool.

What a real scraping job costs

Take a common job: tracking 10 keywords once a day on both platforms, one page each. That works out to up to about 450 rows a day.

ChatterAPI plans start at $19 a month for 25,000 credits, so this job uses about 6% of the smallest plan. The Apify figures come from the per-result prices listed on Oct 14, 2026. You pay them on top of an Apify plan: the Free plan includes $5 of monthly usage and Starter is $19 a month, per Apify pricing. The official X API lists post reads at $0.005 per resource on pay-per-use (X API pricing, checked 2026-10-14). At that rate, the 6,000 tweets this job reads in a month would cost $30 before any Reddit data.

A one-off research pull is different. 50 keywords at five pages per platform is 1,250 credits, paid once. Running the daily job every hour multiplies its cost by 24, so start with daily runs and move to hourly only for the keywords where timing matters.

Is it legal to scrape Twitter and Reddit?

Both platforms' terms prohibit it without permission, and the legal picture is unsettled. Read the primary sources.

• X's Terms of Service (effective October 9, 2026) say "crawling or scraping the Services in any form, for any purpose without our prior written consent is expressly prohibited." They also set liquidated damages of $15,000 USD per 1,000,000 posts for anyone who violates the terms and requests, views or accesses more than 1,000,000 posts in a 24-hour period. X's robots.txt disallows every path for crawlers it doesn't name.
• Reddit's User Agreement, effective July 1, 2026, says "scraping the Services without Reddit's prior written consent is prohibited," and Reddit has sued scrapers.
• As for the law, internet law professor Eric Goldman calls scraping law "an area of doctrinal chaos" (write-up).

Students run into this as well. One r/learnpython poster planning a sentiment project wrote that after reading X's policies, "browser-based scraping/automation of X is not allowed, so I want to find a legitimate approach" (thread).

A hosted API changes who maintains the collection. It doesn't change what you may do with the data. Store only what the job needs, respect deletions, and don't use results for spam or bulk DMs. This isn't legal advice. For a commercial product, talk to a lawyer.

The case for doing it yourself

Experienced scrapers push back on all of this, and they have a point. In the same r/webscraping thread on social listening, u/HLCYSWAP argued the official X API is "insanely expensive" and "everything is against ToS" (comment), then claimed that under every unofficial API "its simply a bot farm underneath" (reply).

Some of that is fair. Nothing on this page makes either platform's terms go away, and a hosted API is a third party you have to trust. If you only need a few hundred posts once, a Playwright session you watch yourself costs nothing but time. Where the argument breaks down is the setup it implies. Spreading requests across a dozen logged-in accounts is what gets accounts banned, and it's the opposite of what X's and Reddit's rules allow. For a job that runs every day, or one that feeds customers or an agent, you want the collection to be someone else's 2 a.m. problem and your own accounts kept out of it.

Try it

ChatterAPI gives your code and your agents Reddit and X data through one key, over REST or MCP. There's no developer account to apply for. Sign up with Google and you get 100 free credits without a card. That covers 20 single-page runs of the two-platform script, or 20 everywhere.search pages. Paid plans start at $19 a month for 25,000 credits.