mongodb-rag-ingest
Namespaces
Interfaces
Type Aliases
Config
Ƭ Config: Object
The configuration for ingest.
You can provide your own configuration to the ingest tool.
Every property is a function that constructs an instance (synchronously or asynchronously). This allows you to run logic for construction or build async. It also avoids unnecessary construction and cleanup if that field of the config is overridden by a subsequent config.
Type declaration
| Name | Type | Description |
|---|---|---|
chunkOptions? | Constructor<Partial<ChunkOptions>> | Options for the chunker. |
concurrencyOptions? | Constructor<ConcurrencyOptions> | Options for concurrency. |
dataSources | Constructor<DataSource[]> | The data sources that you want ingest to pull content from. |
embeddedContentStore | Constructor<EmbeddedContentStore> | The store that holds the embedded content and vector embeddings for later vector search. |
embedder | Constructor<Embedder> | The embedding function. |
ingestMetaStore | Constructor<IngestMetaStore> | The store that contains the ingest meta document. The ingest meta document stores the date of the last successful run. |
pageStore | Constructor<PageStore> | The store that holds pages downloaded from data sources. |
Defined in
mongodb-rag-ingest/src/Config.ts:22
Constructor
Ƭ Constructor<T>: () => T | () => Promise<T>
Type parameters
| Name |
|---|
T |
Defined in
mongodb-rag-ingest/src/Config.ts:80
IngestMetaEntry
Ƭ IngestMetaEntry: Object
Type declaration
| Name | Type |
|---|---|
_id | string |
lastIngestDate | Date |
Defined in
mongodb-rag-ingest/src/IngestMetaStore.ts:40
IngestMetaStore
Ƭ IngestMetaStore: Object
The ingest meta has information about ingest runs so that the script can resume from a known successful run date.
If the 'since' date given to the embed command is too late, pages that were updated during a failed run will not be picked up.
If too early, more pages and embeddings will be checked than necessary. The embed command will not unnecessarily create new embeddings for page updates that it has already created embeddings for, but it would still be wasteful to have to check potentially all pages and embeddings when the date is early enough.
Type declaration
| Name | Type | Description |
|---|---|---|
entryId | string | The ID of the specific metadata document this store is associated with. Generally there should be only one document per ingest_meta collection per database. |
close | () => Promise<void> | Closes the connection. Must be called when done. |
loadLastSuccessfulRunDate | () => Promise<null | Date> | Returns the last successful run date for the store's entry. |
updateLastSuccessfulRunDate | () => Promise<void> | Sets the store's entry to the current date. |
Defined in
mongodb-rag-ingest/src/IngestMetaStore.ts:16
LoadConfigArgs
Ƭ LoadConfigArgs: Object
Type declaration
| Name | Type |
|---|---|
config? | string |
Defined in
mongodb-rag-ingest/src/withConfig.ts:6
ResolvedConfig
Ƭ ResolvedConfig: { [K in keyof Config]: Constructed<Config[K]> }
Config with promises resolved.
Defined in
mongodb-rag-ingest/src/withConfig.ts:98
Variables
INGEST_ENV_VARS
• Const INGEST_ENV_VARS: Object
Type declaration
| Name | Type |
|---|---|
MONGODB_CONNECTION_URI | string |
Defined in
mongodb-rag-ingest/src/IngestEnvVars.ts:1
Functions
loadConfig
▸ loadConfig(«destructured»): Promise<Config>
Parameters
| Name | Type |
|---|---|
«destructured» | LoadConfigArgs |
Returns
Promise<Config>
Defined in
mongodb-rag-ingest/src/withConfig.ts:10
makeIngestMetaStore
▸ makeIngestMetaStore(«destructured»): IngestMetaStore
Creates a connection to ingest meta collection.
Parameters
| Name | Type |
|---|---|
«destructured» | Object |
› connectionUri | string |
› databaseName | string |
› entryId | string |
Returns
Defined in
mongodb-rag-ingest/src/IngestMetaStore.ts:48
withConfig
▸ withConfig<T>(action, args): Promise<void>
Type parameters
| Name |
|---|
T |
Parameters
| Name | Type |
|---|---|
action | (config: ResolvedConfig, args: T) => Promise<void> |
args | LoadConfigArgs & T |
Returns
Promise<void>
Defined in
mongodb-rag-ingest/src/withConfig.ts:62
withConfigOptions
▸ withConfigOptions<T>(args): Argv<T & LoadConfigArgs>
Apply config options to CLI command.
Type parameters
| Name |
|---|
T |
Parameters
| Name | Type |
|---|---|
args | Argv<T> |
Returns
Argv<T & LoadConfigArgs>