Skip to main content
Skip to main content
Helix
Book a briefing →

News Feeds

Collect and monitor news content from websites, sitemaps, and index pages with structured data extraction.

Overview​

News feeds aggregate articles from multiple sources into a single, organized stream. Each feed continuously monitors its sources for new content, automatically extracting structured data including titles, descriptions, and metadata.

Key benefits:

  • Automatic discovery: New articles are detected and processed without manual intervention
  • Clean data extraction: Structured content with titles, descriptions, and metadata
  • Flexible retrieval: Access via API polling or webhooks
  • Historical backfill: Automatically includes the past 14 days of articles from each source, once, when you enable a feed or add a source to an enabled feed

Quick Example​

# Create a feed
curl -X POST https://api.feeds.onhelix.ai/feeds/news \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Tech News", "description": "Technology coverage"}'

# Add a website
curl -X POST https://api.feeds.onhelix.ai/feeds/news/{feedId}/sources \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"type": "site", "url": "https://techcrunch.com"}'

# Decide what it carries: the Standard news set of opt-in filters
curl -X POST https://api.feeds.onhelix.ai/feeds/news/{feedId}/filter-sets/standard_news \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

# Enable the feed (a new news feed starts disabled)
curl -X PATCH https://api.feeds.onhelix.ai/feeds/news/{feedId} \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

# Retrieve articles
curl "https://api.feeds.onhelix.ai/feeds/news/{feedId}/items?limit=20" \
  -H "Authorization: Bearer YOUR_API_KEY"

See the Quickstart Guide for a complete walkthrough.

Source Types​

News feeds support seven types of sources: websites, sitemaps, and index pages (web-based), plus Instagram, TikTok, Twitter, and Facebook (social). Through the API you add websites; our team adds the other types on request. You can list, enable, disable and remove every source on your feeds. See Sources.

Website Sources​

Monitor entire domains for new articles.

Use when:

  • You want to track all content from a publisher
  • The site has clear article pages (news sites, blogs)
  • You don't need to filter by specific sections

How it works:

  1. Helix crawls the domain to discover article pages
  2. New pages are detected automatically as they're published
  3. Articles are extracted with clean, structured data

Example:

{
  "type": "site",
  "url": "https://techcrunch.com"
}

Monitors all articles published on techcrunch.com. A URL with a path adds the whole site too, and www.techcrunch.com and techcrunch.com are the same site.

Sitemap Sources​

Track specific XML sitemaps for more targeted content discovery: the site publishes a news sitemap, and you want faster, more reliable article detection. Helix monitors the sitemap for new URLs and extracts the articles they point to. Added by our team on request.

Index Page Sources​

Monitor specific pages that list articles, such as category or section pages, to track one section of a site. Helix monitors the page for article links and processes new ones as they appear. Added by our team on request.

Social Sources​

Monitor posts and reels from an Instagram account, videos from a TikTok account, tweets from a Twitter / X account, or content from a Facebook page, group or profile. Added by our team on request.

How Processing Works​

Adding a Source​

When you add a source to a feed:

  1. Immediate validation: the site's host must be a public domain that resolves
  2. Automatic backfill: The site's articles from the past 14 days are added to your feed, right away on an enabled feed, or when you enable a disabled feed (a new feed starts disabled)
  3. Continuous monitoring: The source is monitored for new content

Feed Items​

Each item in your feed has the same shape, whether the article came from a website, Instagram, TikTok, Twitter or Facebook. Optional fields are omitted (not nulled) when the underlying value is missing.

FieldDescription
idUnique identifier for the feed item
newsPageIdThe article or post. The same value in every feed that carries it
addedToFeedAtWhen the item entered your feed
updatedAtWhen the item last changed in a way your webhooks are told about
titleHeadline (omitted when not available)
descriptionArticle description, or the post text
urlLink to the article or post
publishedAtWhen it was published (omitted when unknown)
publisherWho published it: name, domain or handle, and platform
authorThe byline as the article states it (websites only)
originalWho first published it, for a syndicated article or a retweet
aboutThe places the item is about, each with a name, level, display text and map position
mediaImages that we host, with a signed temporary link, the source and the credit

Example feed item:

{
  "id": "3f6c1d52-7a0e-4b8e-9d1c-5a2b8e4f6a10",
  "newsPageId": "9b2e7c44-1d3a-4f65-8a7b-0c5d2e9f1b33",
  "addedToFeedAt": "2026-10-20T14:03:00.000Z",
  "updatedAt": "2026-10-20T14:03:00.000Z",
  "title": "Sterling Heights approves library expansion",
  "description": "The city council voted 6-1 on Monday to fund a 12,000-square-foot addition to the Sterling Heights Public Library.",
  "url": "https://www.macombdaily.com/2026/10/19/sterling-heights-approves-library-expansion/",
  "publishedAt": "2026-10-20T09:15:00.000Z",
  "publisher": {
    "platform": "web",
    "name": "Macomb Daily",
    "domain": "macombdaily.com"
  },
  "author": "Jane Smith",
  "original": {
    "publisher": {
      "platform": "web",
      "name": "Detroit Free Press",
      "domain": "freep.com"
    },
    "url": "https://www.freep.com/story/news/local/michigan/macomb/2026/10/19/sterling-heights-library-expansion/1234567890/",
    "publishedAt": "2026-10-19T11:30:00.000Z"
  },
  "about": {
    "places": [
      {
        "name": "Sterling Heights",
        "level": "place",
        "displayText": "Sterling Heights, MI",
        "geo": {
          "lat": 42.57,
          "lng": -83.025,
          "bbox": [-83.08, 42.52, -82.97, 42.62]
        },
        "city": "Sterling Heights",
        "district": "Macomb County",
        "region": "Michigan",
        "country": "United States"
      },
      {
        "name": "Macomb County",
        "level": "district",
        "displayText": "Macomb County, MI",
        "geo": {
          "lat": 42.65,
          "lng": -82.9,
          "bbox": [-83.1, 42.5, -82.7, 42.8]
        },
        "district": "Macomb County",
        "region": "Michigan",
        "country": "United States"
      }
    ]
  },
  "media": {
    "cover": {
      "url": "https://example.r2.cloudflarestorage.com/media/2026-10/library-expansion.jpg?X-Amz-Signature=...",
      "expiresAt": "2026-10-28T12:00:00.000Z",
      "usage": "display_with_attribution",
      "kind": "article_image",
      "source": {
        "imageUrl": "https://www.macombdaily.com/wp-content/uploads/2026/10/library-expansion.jpg",
        "pageUrl": "https://www.macombdaily.com/2026/10/19/sterling-heights-approves-library-expansion/",
        "publisher": {
          "platform": "web",
          "name": "Macomb Daily",
          "domain": "macombdaily.com"
        },
        "credit": "Jane Smith / The Macomb Daily"
      },
      "width": 1200,
      "height": 800
    },
    "images": [
      {
        "url": "https://example.r2.cloudflarestorage.com/media/2026-10/library-expansion.jpg?X-Amz-Signature=...",
        "expiresAt": "2026-10-28T12:00:00.000Z",
        "usage": "display_with_attribution",
        "kind": "article_image",
        "source": {
          "imageUrl": "https://www.macombdaily.com/wp-content/uploads/2026/10/library-expansion.jpg",
          "pageUrl": "https://www.macombdaily.com/2026/10/19/sterling-heights-approves-library-expansion/",
          "publisher": {
            "platform": "web",
            "name": "Macomb Daily",
            "domain": "macombdaily.com"
          },
          "credit": "Jane Smith / The Macomb Daily"
        },
        "width": 1200,
        "height": 800
      }
    ]
  }
}

Image links expire (see expiresAt): do not store them. Show each image with its source and credit. Images stay available for 30 days after the item enters the feed, or for the number of days set in the feed's config.media.retentionDaysAfterAdded. Every URL an item carries is an http(s) URL; a link that is not is left out. See the API reference for every field.

Which Articles a Feed Carries​

Each feed decides for itself which articles it carries, and you set it up through the API. Every article is labelled with how much it belongs to each of 14 content types (editorial news, opinion, sponsored content, daily life and so on), and the feed's include and exclude filters are written on those labels and on your own processors. An article is in the feed when no exclude filter matches and at least one include filter does. A safety floor on the article's text and images applies to every feed and cannot be switched off, and a feed can be limited to an area. Our ready-made opt-in filter sets, Standard news and Creator social, set up a feed in one request. See Concepts for the details.

An item is one article or one post. If the publisher edits it, the same item changes in place, and it can leave the feed later if it stops passing the feed's filters.

A new news feed starts disabled and with no filters. Enabling a feed through PATCH needs at least one include filter, and is refused with 422 without one.

Feed Management​

Enabling and Disabling Sources​

Temporarily pause monitoring without deleting the source:

# Disable a source
curl -X PATCH https://api.feeds.onhelix.ai/feeds/news/{feedId}/sources/{sourceId} \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

# Re-enable later
curl -X PATCH https://api.feeds.onhelix.ai/feeds/news/{feedId}/sources/{sourceId} \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

A disabled source stays on the feed but adds no new articles to it, and its articles already in the feed stay. We stop crawling a source once no enabled feed uses it. Re-enabling a source does not add the articles it brought in while it was disabled, unless the feed has never been caught up on it (see What a new feed receives).

Organizing Multiple Feeds​

Create separate feeds for different purposes:

# News monitoring feed
curl -X POST https://api.feeds.onhelix.ai/feeds/news \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Daily News", "description": "General news monitoring"}'

# Competitor analysis feed
curl -X POST https://api.feeds.onhelix.ai/feeds/news \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Competitors", "description": "Competitor announcements"}'

# Research feed
curl -X POST https://api.feeds.onhelix.ai/feeds/news \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Research Topics", "description": "Academic and research content"}'

Each feed has its own sources, filters and items. A website can be on several feeds; it is crawled once.

Event Delivery​

Get notified in real time when new articles are added to your feeds, instead of polling for changes.

Available event types:

  • news.item.added - a new article was added to your feed
  • news.item.updated - an article in your feed changed (title, description, link, publication time, publisher, author, original, places or images)
  • news.item.removed - an article left your feed, or the feed was deleted

Having a news feed does not, by itself, cause anything to be delivered. Event Delivery only sends an event to a destination that has a subscription matching this feed — you register the destination once, then subscribe it to the feeds and event types you care about. See Event Delivery: Two objects, not one for details.

See the news.item.added reference for the full payload schema and an example, and News feed changes for when each of the three is sent.