The unidirectional file synchronization interface allows you to implement 1-way sync in your integration to import files from an external service to Botpress.
## Filesystem abstraction
The file synchronization interface provides a filesystem-like abstraction that works with any kind of data source. The external service doesn't need to provide an actual filesystem - your integration just needs to represent the external data as files and folders.
For example:
* If you are building a website crawler, individual pages could be folders and HTML contents and assets like images or stylesheets could be files.
* For a note-taking platform, notebooks could be folders with individual notes being files.
* For an email provider, mailboxes or labels could be folders and individual emails could be files.
This abstraction allows the interface to work consistently regardless of what type of data is being synchronized from your external service.
## Terminology
Throughout this document, we will use the following terms:
The code that connects Botpress to an external service.
The service from which you want to import files. This could be a cloud storage service, a file server, or any other type of external service that stores files.
The interface that defines the contract for implementing unidirectional file synchronization in your integration. This interface specifies the actions and events that your integration must implement to support file synchronization.
The Botpress plugin that orchestrates file synchronization. This plugin is responsible for managing the synchronization process, including scheduling, error handling, and reporting.
A file is a single unit of data that can be synchronized from the external service to Botpress. Files can contain any type of data, such as text, images, or binary data. Files can't contain other files or folders.
A folder is a container for files. Folders can contain other folders and files, allowing for a hierarchical organization of data.
A synchronization mode where changes in the external service are immediately reflected in Botpress. This is typically achieved through webhooks or other push mechanisms. Integrations aren't required to support this mode, but it's recommended for better user experience.
## External service requirements
The providing file synchronization functionality **must** support the following:
* An API that allows listing all files and folders in a folder.
* Must support pagination. This means that the API should return a limited number of items at a time, along with a token that can be used to retrieve the next set of items.
* An API that allows downloading files.
The **may** also support the following in order to provide :
* Webhooks that can notify your of the following events:
* A was created.
* A was updated.
* A was deleted.
* A was deleted.
## Updating your `package.json` file
### Finding the current interface version
The current version of the `files-readonly` interface is:
You will need this version number for the next steps.
### Adding the interface as a dependency
Once you have the version, you can add it as a dependency to your :
Open your 's `package.json` file.
If there is no `bpDependencies` section in your 's `package.json` file, create one:
```json package.json {2} theme={null}
{
"bpDependencies": {}
}
```
In the `bpDependencies` section, add the as a dependency. For example, for version `0.2.0`, you would add the following:
```json package.json {3} theme={null}
{
"bpDependencies": {
"files-readonly": "interface:files-readonly@0.2.0"
}
}
```
It's very important to follow this syntax:
`"": "interface:@"`.
Save the `package.json` file.
Now that you have added the as a dependency, you can run the [`bp add`](/integrations/sdk/cli-reference#add) command to install it. This command will:
* Download the interface from Botpress.
* Install it in a directory named `bp_modules` in your 's root directory.
### Adding a helper build script
To keep your up to date, we recommend adding a helper build script to your `package.json` file:
Open your 's `package.json` file.
In the `scripts` section, add the following script:
```json package.json {3} theme={null}
{
"scripts": {
"build": "bp add -y && bp build"
}
}
```
If the `build` script already exists in your `package.json` file, please replace it.
Save the `package.json` file.
Now, whenever you run `npm run build`, it will automatically install the and build your .
## Editing your integration definition file
### Adding the interface to your integration definition file
Now that the is installed, you must add it your integration definition file in order to implement it.
Open your 's `integration.definition.ts` file.
At the top of the file, import the :
```typescript integration.definition.ts theme={null}
import filesReadonly from './bp_modules/files-readonly'
```
Use the `.extend()` function at the end of your `new IntegrationDefinition()` statement:
```typescript integration.definition.ts {4-6} theme={null}
export default new sdk.IntegrationDefinition({
...
})
.extend(files-readonly, () => ({
entities: {},
}))
```
The exact syntax of `.extend()` will be explained in the next section.
### Configuring the interface
The `.extend()` function takes two arguments:
* The first argument is a reference to the interface you want to implement. In this case, it's `filesReadonly`.
* The second argument is a configuration object. Using this object, you can override interface defaults with custom names, titles, and descriptions.
Whilst renaming actions, events and channels is optional, it's highly recommended to rename these to match the terminology of the . This will help you avoid confusion and make your easier to understand.
#### Renaming actions
The defines two actions that are used to interact with the :
* `listItemsInFolder` - Used by the to request a list of all files and folders in a folder.
* `transferFileToBotpress` - Used by the to request that a file be downloaded from the and uploaded to Botpress.
If you want to rename these actions, you can do so in the configuration object. For example, if you want to rename `listItemsInFolder` to `crawlFolder`, you can do it like this:
```typescript integration.definition.ts {4} theme={null}
.extend(filesReadonly, () => ({
actions: {
listItemsInFolder: {
name: 'crawlFolder',
},
},
}))
```
For example, if you're using a note-taking platform such as Microsoft OneNote, you might rename `listItemsInFolder` to `listNotebooksAndPages` and `transferFileToBotpress` to `downloadPage`. This way, the action names reflect the specific context of the note-taking platform, making your clearer and easier to understand.
#### Renaming events
The interface defines these events to notify the plugin of changes in the :
* `fileCreated` - Emitted by your to notify the that a new has been created in the .
* `fileUpdated` - Emitted by your to notify the that a has been updated in the .
* `fileDeleted` - Emitted by your to notify the that a has been deleted in the .
* `folderDeletedRecursive` - Emitted by your to notify the that a and all of its contents have been deleted in the .
If the emits several filesystem changes at once, it's also possible for your integration to emit a `aggregateFileChanges` event, which contains all the changes in a single event.
If you want to rename these events, you can do so in the configuration object. For example, if you want to rename `fileCreated` to `pageCreated`, you can do it like this:
```typescript integration.definition.ts {4} theme={null}
.extend(filesReadonly, () => ({
events: {
fileCreated: {
name: 'pageCreated',
},
},
}))
```
## Implementing the interface
### Implementing the actions
#### Implementing `listItemsInFolder`
The `listItemsInFolder` action is used by the to request a list of all files and folders in a folder.
If you opted to rename the action to something else to `listItemsInFolder` in the "Configuring the interface" section, please use the new name instead of `listItemsInFolder`.
Please refer to the expected input and output schemas for the action:
[interface.definition.ts line 52](https://github.com/botpress/botpress/blob/master/interfaces/files-readonly/interface.definition.ts#L52).
This action should implement the following logic:
Get the folder identifier from `input.folderId`. When this value is `undefined`, it means the is requesting a list of all items in the root directory of the . For root directory requests, please refer to the documentation of the to determine the correct root identifier - this is typically an empty string, a slash character (`/`), or a special value defined by the service.
Use the 's API to get the list of items in the folder. If the supports filtering by item type (file or folder), by maximum file size, or by modification date, please use these filters to limit the number of items returned. This will help reduce the amount of data transferred and improve performance.
```typescript {3-7} theme={null}
type Input = {
folderId?: string;
filters?: {
itemType?: "file" | "folder";
maxSizeInBytes?: number;
modifiedAfter?: string; // <= ISO 8601 date string
};
nextToken?: string; // <= pagination token
};
```
If a pagination token is provided (`input.nextToken`), use it to get the next page of items. The should return a new pagination token in the response, which you should return with the action's response.
**Don't** list items recursively. The is responsible for handling recursion. Your should only return the items in the specified folder.
Map each item to the expected schema. The expects the following schemas:
```typescript {2-4} theme={null}
type Folder = {
id: string;
type: "folder";
name: string;
parentId?: string;
absolutePath?: string;
}
```
This could be a unique identifier from the , or a relative or absolute path, so long as it's unique. Your must be able to resolve this identifier to the actual folder in the . The will use this identifier to enumerate children of this folder.
This should always be `folder`.
The name of the folder. This should be the name of the folder as it appears in the .
The identifier of the parent folder. This should be the same identifier as the one used in the `folderId` field of the input.
The absolute path of the folder in the , if available.
```typescript {2-4} theme={null}
type File = {
id: string;
type: "file";
name: string;
parentId?: string;
absolutePath?: string;
sizeInBytes?: number;
lastModifiedDate?: string;
contentHash?: string;
}
```
This could be a unique identifier from the , or a relative or absolute path, so long as it's unique. Your must be able to resolve this identifier to the actual file in the . The will use this identifier to download the file.
This should always be `file`.
The name of the file. This should be the name of the file as it appears in the , including its file extension. This same name will be used to display the file in Botpress.
The identifier of the parent folder. This should be the same identifier as the one used in the `folderId` field of the input.
The absolute path of the file in the , if available.
The size of the file in bytes, if available.
The last modified date of the file, if available. This should be an ISO 8601 date string.
The content hash of the file, if available. This should be a unique identifier for the file's content, such as an MD5 or SHA-1 hash.
This is used to detect changes in the file's content. If the content hash isn't available but the provides a version or revision number, you can use that instead.
Yield control back to the by returning the list of items. The will then handle the rest of the synchronization process.
```typescript {2-3} theme={null}
return {
items: [...mappedFolders, ...mappedFiles],
meta: { nextToken: hasMoreItems ? nextToken : undefined },
}
```
If the indicates it has more items, return the pagination token in the `nextToken` field. The will use this token to request the next page of items. Otherwise, return `undefined`.
As reference, here's how this logic is implemented in the Dropbox integration:
```typescript src/index.ts {4,7,11,14,15,20,27} theme={null}
export default new bp.Integration({
actions: {
async listItemsInFolder({ ctx, input, client }) {
// Extract input parameters:
const { folderId, filters, nextToken: prevToken } = input
// Get the folder ID:
// (Dropbox expects an empty string for the root directory)
const parentId = folderId ?? ''
// Get the list of items in the folder
const { items, nextToken, hasMore } = await dropboxClient.listItemsInFolder({
path: parentId,
recursive: false, // <= The integration shouldn't list recursively
nextToken: prevToken, // <= Use the pagination token if provided
})
const mappedItems = items
.filter((item) => item.itemType !== 'deleted')
// Call utility functions to handle the mapping:
.map((item) =>
item.itemType === 'file' ?
filesReadonlyMapping.mapFile(item) :
filesReadonlyMapping.mapFolder(item)
)
// Yield control back to the plugin and return the items:
return {
items: mappedItems,
meta: { nextToken: hasMore ? nextToken : undefined },
}
},
},
})
```
#### Implementing `transferFileToBotpress`
The `transferFileToBotpress` action is used by the to request that a file be downloaded from the and uploaded to Botpress.
If you opted to rename the action to something else to `transferFileToBotpress` in the "Configuring the interface" section, please use the new name instead of `transferFileToBotpress`.
Please refer to the expected input and output schemas for the action:
[interface.definition.ts line 88](https://github.com/botpress/botpress/blob/master/interfaces/files-readonly/interface.definition.ts#L88).
This action should implement the following logic:
Get the file identifier from `input.file.id`. This is the identifier of the file to be downloaded from the .
Use the 's API to download the file's content.
Upload the file to Botpress using the `client.uploadFile` method. This method expects both the file's content and a file key, which is provided by the as `input.fileKey`.
Yield control back to the by returning the ID of the file that was uploaded to Botpress.
As reference, here's how this logic is implemented in the Dropbox integration:
```typescript src/index.ts {4,7,10,16} theme={null}
export default new bp.Integration({
actions: {
async transferFileToBotpress({ ctx, input, client }) {
// Extract input parameters:
const { file: fileToTransfer, fileKey } = input
// Use Dropbox's SDK to download the file:
const fileBuffer = await dropboxClient.getFileContents({ path: fileToTransfer.id })
// Upload the file to Botpress:
const { file: uploadedFile } = await client.uploadFile({
key: fileKey,
content: fileBuffer,
})
// Yield control back to the plugin:
return {
botpressFileId: uploadedFile.id
}
},
},
})
```
### Implementing real-time sync
The can be configured to use real-time synchronization. This means that changes in the are immediately reflected in Botpress. To enable this functionality, the must support webhooks that can notify your of changes in the filesystem.
#### Implementing `fileCreated`
In your , add a webhook handler that can receive file change notifications from the .
In your handler, map the file to the expected schema. The expects the following schema:
```typescript {2-5} theme={null}
type File = {
id: string;
type: "file";
name: string;
absolutePath: string;
parentId?: string;
sizeInBytes?: number;
lastModifiedDate?: string;
contentHash?: string;
}
```
This could be a unique identifier from the , or a relative or absolute path, so long as it's unique. Your must be able to resolve this identifier to the actual file in the . The will use this identifier to download the file.
This should always be `file`.
The name of the file. This should be the name of the file as it appears in the , including its file extension. This same name will be used to display the file in Botpress.
The absolute path of the file in the . Unlike with the `listItemsInFolder` action, this field is required for the `fileCreated` event.
The identifier of the parent folder.
The size of the file in bytes, if available.
The last modified date of the file, if available. This should be an ISO 8601 date string.
The content hash of the file, if available. This should be a unique identifier for the file's content, such as an MD5 or SHA-1 hash.
This is used to detect changes in the file's content. If the content hash isn't available but the provides a version or revision number, you can use that instead.
Emit the `fileCreated` event with the mapped file as the payload. The will then handle the rest of the synchronization process.
```typescript {2-3} theme={null}
await client.createEvent({
name: 'fileCreated',
payload: { file: mappedFile },
})
```
#### Implementing `fileUpdated`
The logic is identical to the `fileCreated` event, but you should emit the `fileUpdated` event instead.
#### Implementing `fileDeleted`
The logic is identical to the `fileCreated` event, but you should emit the `fileDeleted` event instead.
#### Implementing `folderDeletedRecursive`
In your , add a webhook handler that can receive file change notifications from the .
In your handler, map the folder to the expected schema. The expects the following schema:
```typescript {2-5} theme={null}
type Folder = {
id: string;
type: "folder";
name: string;
absolutePath: string;
parentId?: string;
}
```
This could be a unique identifier from the , or a relative or absolute path, so long as it's unique.
This should always be `folder`.
The name of the folder. This should be the name of the folder as it appears in the
The absolute path of the folder in the . Unlike with the `listItemsInFolder` action, this field is required for the `folderDeletedRecursive` event.
The identifier of the parent folder.
Emit the `folderDeletedRecursive` event with the mapped folder as the payload. The will then handle the rest of the synchronization process.
```typescript {2-3} theme={null}
await client.createEvent({
name: 'folderDeletedRecursive',
payload: { folder: mappedFolder },
})
```
#### Implementing `aggregateFileChanges`
The logic is identical to the `fileCreated`, `fileUpdated`, `fileDeleted`, or `folderDeletedRecursive` events, but you should emit the `aggregateFileChanges` event instead:
```typescript {4-8} theme={null}
await client.createEvent({
name: 'aggregateFileChanges',
payload: {
modifiedItems: {
created: [...mappedCreatedFiles],
updated: [...mappedUpdatedFiles],
deleted: [...mappedDeletedFilesOrFolders],
},
},
})
```
If your needs to emit more than one filesystem change event, you should combine them into a single `aggregateFileChanges` event. This is more efficient and faster to process for the .