Skip to main content

Astra DB Destination

Availability
Available CoreAvailable StandardAvailable PlusAvailable ProAvailable Enterprise FlexAvailable PyAirbyte
Support Level
Marketplace
Connector Version
0.1.44 (last updated a year ago)
CDK Version
0.81.6
Definition ID
042ce96f-1158-4662-9543-e2ff015be97a

This page contains the setup guide and reference information for the destination-astra connector.

Use this destination to load records into Astra DB as vector documents. For each record, the connector concatenates the text fields you select, splits the text into chunks, generates an embedding for each chunk with the embedding service you configure, and writes each chunk as a document in a single Astra DB collection through the Data API.

Prerequisites​

  • A Serverless (Vector) database in Astra DB, and its API endpoint. The endpoint looks like https://<database-id>-<region>.apps.astra.datastax.com.
  • An Astra DB application token that can create collections and read and write data in the target keyspace. A token generated from the database's Overview tab gets a Database Administrator role scoped to that database, which is sufficient.
  • A keyspace in that database. New Serverless (Vector) databases include default_keyspace.
  • A name for the collection to write to. The connector creates the collection if it doesn't exist.
  • An API key for the embedding service you want to use, unless you pick the Fake embedder. The connector supports OpenAI, Azure OpenAI, Cohere, and any OpenAI-compatible embedding service.

Set up an Astra DB database​

If you don't already have a Serverless (Vector) database, create one:

  1. Create an Astra account at astra.datastax.com/signup.
  2. In the Astra Portal, select Databases, then click Create Database.
  3. Select the Serverless (Vector) deployment type.
  4. Enter a name in the Database name field. You can't change the name later. Names must start and end with an alphanumeric character, and can contain the following special characters: & + - _ ( ) < > . , @.
  5. Select your preferred provider and region. The Free plan offers a limited set of regions. Regions with a lock icon require the Pay As You Go plan.
  6. Click Create Database. The database starts in Pending status, moves to Initializing, and you get a notification when it's ready.

Get the endpoint and token​

  1. Open the Overview tab for your database in the Astra Portal.
  2. Under Database Details, copy the endpoint and enter it in Airbyte as the Astra DB Endpoint.
  3. Click Generate Token, then copy the token and enter it in Airbyte as the Astra DB Application Token. Astra shows the token only once, so store it somewhere safe.
  4. Enter the keyspace you want to write to as the Astra DB Keyspace, and the collection name as the Astra DB collection. To create a keyspace or inspect existing ones, use the Data Explorer tab.

Choose an embedding service​

The embedding service determines the vector dimension of your documents. Astra fixes a collection's dimension when the collection is created, so if you change the embedding model or dimension later, write to a new collection. Astra supports vectors of up to 4,096 dimensions.

If you select OpenAI-compatible, you must supply the dimension of the model yourself, along with the base URL of the service. Airbyte can't detect it from the service.

The Fake embedder generates random vectors. Use it to test the pipeline end to end without paying for embeddings. Don't use it in production, because random vectors make search results meaningless.

How the connector stores data​

Every stream you sync writes into the single collection you configure. The connector creates that collection with the cosine similarity metric and the dimension of your embedding model.

Each document corresponds to one chunk of one record and contains:

  • _id: a generated UUID.
  • $vector: the embedding of the chunk.
  • text: the chunk's text. The connector omits this field if you enable Do not store raw text.
  • _ab_stream: the stream the chunk came from, including its namespace.
  • _ab_record_id: the primary key of the source record. The connector uses this field to delete outdated chunks in deduplicating sync modes.
  • One field for each of the metadata fields you configure.

Because all streams share one collection, filter on _ab_stream when you query documents from a specific stream.

Supported sync modes​

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedYes
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Sync modes behave as follows:

  • Overwrite deletes only the documents whose _ab_stream matches the stream being synced, then writes the new documents. Documents from other streams in the same collection are untouched.
  • Append adds documents without deleting anything. Syncing the same record again creates duplicate documents.
  • Deduped modes delete the existing documents for each incoming record's primary key before writing that record's new chunks, so the stream needs a primary key.

Namespace support​

This destination supports namespaces. A namespace doesn't create a separate Astra keyspace or collection. It becomes part of the _ab_stream value on each document.

Limitations​

Keep the following Astra DB limits in mind when you plan a sync:

  • A collection's vector dimension is fixed when the collection is created. To switch to an embedding model with a different dimension, configure a different collection.
  • A collection can hold no more than 64 distinct fields across all of its documents. Every metadata field you configure, on every stream that shares the collection, counts toward that limit.
  • A database supports approximately 10 collections.
  • Indexed string values are limited to 8,000 bytes. The connector creates collections with default indexing, which indexes every field, including text. If inserts fail because a chunk is too large, reduce the chunk size.

Reference​

Config fields reference

Field
Type
Property name
object
embedding
object
indexing
object
processing
boolean
omit_raw_text

Changelog​

Expand to review
VersionDatePull RequestSubject
0.1.452026-08-1384359Update the CDK to remediate CVE-2025-68664 in the langchain dependency
0.1.442025-03-2956606Update dependencies
0.1.432025-03-2256098Update dependencies
0.1.422025-03-0855394Update dependencies
0.1.412025-03-0154871Update dependencies
0.1.402025-02-2254244Update dependencies
0.1.392025-02-1553883Update dependencies
0.1.382025-02-0853388Update dependencies
0.1.372025-02-0152943Update dependencies
0.1.362025-01-2552179Update dependencies
0.1.352025-01-1151295Update dependencies
0.1.342025-01-0450910Update dependencies
0.1.332024-12-2850446Update dependencies
0.1.322024-12-2150213Update dependencies
0.1.312024-12-1449288Update dependencies
0.1.302024-11-2548674Update dependencies
0.1.292024-10-2947105Update dependencies
0.1.282024-10-1246857Update dependencies
0.1.272024-10-0546402Update dependencies
0.1.262024-09-2846179Update dependencies
0.1.252024-09-2145829Update dependencies
0.1.242024-09-1445498Update dependencies
0.1.232024-09-0745330Update dependencies
0.1.222024-08-3144983Update dependencies
0.1.212024-08-2444700Update dependencies
0.1.202024-08-2244530Update test dependencies
0.1.192024-08-1744319Update dependencies
0.1.182024-08-1243811Update dependencies
0.1.172024-08-1043598Update dependencies
0.1.162024-08-0343075Update dependencies
0.1.152024-07-2742805Update dependencies
0.1.142024-07-2042251Update dependencies
0.1.132024-07-1341698Update dependencies
0.1.122024-07-1041451Update dependencies
0.1.112024-07-0941095Update dependencies
0.1.102024-07-0640779Update dependencies
0.1.92024-06-2940626Update dependencies
0.1.82024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.1.72024-06-2540467Update dependencies
0.1.62024-06-2240162Update dependencies
0.1.52024-06-0639198[autopull] Upgrade base image to v1.2.2
0.1.42024-05-1638181Add explicit projection when reading from Astra DB
0.1.32024-04-1937405Add "airbyte" user-agent in the HTTP requests to Astra DB
0.1.22024-04-15Moved to Poetry; Updated CDK & pytest versions
0.1.12024-01-26DS Branding Update
0.1.02024-01-08Initial Release
Was this page helpful?