Data sources

Data sources are how pensieve pulls from sources that don't push to it. The trait is small, the registry is one map, and the runner is the same group-commit loop the OTLP and Kafka frontends use — so a data source's output lands on the same staging buffer, through the same snapshot CAS, into the same pensieve extents.

One framework. One reference engine in tree. Three operational-database engines on the way, each shipping in its own milestone.

The Add data source catalog — code, knowledge, project, and data sources side by side. Available sources are one click to connect; the grid is engine-driven, so a new data source type appears here automatically.

Framework

The DataSource trait, the registry, the periodic scheduler, the admin REST API at POST /v1/data-sources, and the secret-by-reference resolver. Generic to every data source type — read this once and the five pages below are mostly config.

Prometheus

The reference implementation. Scrapes /metrics on a schedule, parses OpenMetrics, lands one row per sample in the configured table. Shipped today; the smallest possible thing that proves the framework.

OAuth data sources

SaaS sources behind a one-click Connect flow — Notion, Google Drive, Gmail, Slack, Jira, Confluence. OAuth2 authorize → encrypted credential → automatic token refresh. Operator-configured apps or bring-your-own.

GitHub

Repositories, branches, pulls, issues, contributors, plus an optional parsed code graph (functions, classes, calls, imports via tree-sitter). One CLI command stands it up — token discovered from $GITHUB_TOKEN, $GH_TOKEN, or gh auth token.

GitLab

Projects, branches, merge requests, issues, members. Self-hosted GitLab supported via --api-url.

Bitbucket

Repositories, branches, pull requests, issues from Bitbucket Cloud. App-password + username (basic) or PAT auth.

Microsoft Fabric

The first federated data source: Lakehouse / Warehouse tables queried live over the SQL analytics endpoint. Schemas cataloged in pensieve, data stays in Fabric, whole subplans (filters, joins, aggregations) pushed down as one T-SQL statement. Shipped today.

Postgres 🚧

Federation for live reads, replication-slot CDC for sync, both at once. Pushdown of filters, projection, LIMIT, single-source aggregation. Lands in DB-M1.

MySQL 🚧

Same shape as Postgres. Binlog row events with GTID checkpoints for CDC; collation-safety rules in the planner so case-insensitive columns never silently corrupt federated equality. Lands in DB-M2.

MongoDB 🚧

Change streams with startAtOperationTime for CDC. BSON type coercion, nested-document flattening up to a configurable depth, polymorphic-field demotion to dynamic. Lands in DB-M3.

Multi-source data 🚧

The marquee surface — live(...) for federated reads, the pushdown_summary on every federated response. Cross-references the conceptual model at Multi-source data.

What's the same across all data source types

• One trait. Implement DataSource::run_once; the framework owns scheduling, retry classification, secret resolution, and the row sink.
• One registry. Data source types register at startup; the admin API validates type against it before persisting any row.
• One write path. Rows produced by a tick go through the same JSON-to-Arrow coercion, the same staging buffer, and the same snapshot CAS as REST/NDJSON ingest. See Extents and snapshots.
• One health surface. Per-tick Prometheus metrics (pensieve_data_source_cursor_age_seconds, pensieve_data_source_rows_ingested_total, etc.) plus last_error, last_rows_ingested, last_success_at on the data source row (GET /v1/data-sources/:id). See Observability.

What you get: a context graph

Graph data sources (GitHub, GitLab, Notion, Slack, Jira, …) don't just land rows — they emit nodes and edges that register as a queryable property graph. A GitHub data source produces Repository, PullRequest, Issue, Branch, and User nodes linked by AUTHORED, HAS_PULL_REQUEST, TARGETS_BRANCH, and more; a Notion data source produces Page/Database/User nodes linked by CONTAINS and RELATES_TO. Every source feeds the same graph, so you can traverse from a Slack message to the Jira issue it mentions to the GitHub PR that closed it.

The Graph view rendering a GitHub data source's output — repositories, pull requests, issues, branches, and contributors as a navigable property graph. See Query → Graph traversals and the graph concepts.