Current state: Accepted
ISSUE: https://github.com/milvus-io/milvus/issues/15604
PRs:
Keywords: bulk load, import
Released: with Milvus 2.1
Authors:
Import data by a shortcut to get better performance compared with insert().
Typically, it cost several hours to insert one billion entities with 128-dimensional vectors. We need a new interface to do bulk load for the following purposes:
Some points to consider:
A row-based example:
{ "table": { "rows": [ {"id": 1, "year": 2021, "vector": [1.0, 1.1, 1.2]}, {"id": 2, "year": 2022, "vector": [2.0, 2.1, 2.2]}, {"id": 3, "year": 2023, "vector": [3.0, 3.1, 3.2]} ] } } |
A column-based example:
{ "table": { "columns": [ "id": [1, 2, 3], "year": [2021, 2022, 2023], "vector": [ [1.0, 1.1, 1.2], [2.0, 2.1, 2.2], [3.0, 3.1, 3.2] ] ] } } |
Based on the several points, we choose a JSON object as a parameter of python import() API, the API declaration will be like this:
def import(options)
The "options" is a JSON object which has the following format:
{ "data_source": { // required "type": "minio", // required, currently only support "minio"/"s3" "address": "localhost:9000", // optional, milvus server will use its minio setting if without this value "accesskey_id": "minioadmin", // optional, milvus server will use its minio setting if without this value "accesskey_secret": "minioadmin", // optional, milvus server will use its minio setting if without this value "use_ssl": false, // optional, milvus server will use its minio setting if without this value "bucket_name": "mybucket" // optional, milvus server will use its minio setting if without this value }, "internal_data": { // optional, external_data or internal_data. (external files include json, npy, etc. internal files are exported by milvus) "path": "xxx/xxx/xx", // relative path to the source storage where store the exported data "collections_mapping": { // optional, give a new name to collection during importing. "aaa": "bbb", "ccc": "ddd" } }, "external_data": { // optional, external_data or internal_data. (external files include json, npy, etc. internal files are exported by milvus) "target_collection": "xxx", // target collection name "files": [ // required { "file": xxxx/xx.json, // required, relative path under the storage source defined by DataSource, currently support json/npy "type": "row_based", // required, row_based or column_based "fields_mapping": { // optional, specify the target fields which should be imported. Milvus will import all fields if this list is empty. "table.rows.id": "uid", "table.rows.year": "year", "table.rows.vector": "vector", } }, { "file": xxxx/xx.json, // required, relative path under the storage source defined by DataSource, currently support json/npy "type": "column_based", // required, row_based or column_based "fields_mapping": { // optional, specify the target fields which should be imported. Milvus will import all fields if this list is empty. "table.columns.id": "uid", "table.columns.year": "year", "table.columns.vector": "vector", } }, { "file": xxxx/xx.npy, // required, relative path under the storage source defined by DataSource, currently support json/npy "type": "column_based", // required, row_based or column_based "fields_mapping": { // optional, specify the target fields which should be imported. Milvus will import all fields if this list is empty. "vector": "vector", } } ], "default_fields": { // optional, use default value to fill some fields "age": 0, "weight": 0.0 }, } } |
Key fields of the JSON object:
How to pass this parameter in different situations:
Assume there is a collection named "TEST" with these fields:
{"uid":INT64, "year":INT32, "age":INT8, "embedding":FLOAT_VECTOR} |
For the following situations:
User has some JSON files store data with the row-based format: file_1.json, file_2.json.
{ "data": { "rows": [ {"id": 1, "year": 2021, "vector": [1.0, 1.1, 1.2]}, {"id": 2, "year": 2022, "vector": [2.0, 2.1, 2.2]}, {"id": 3, "year": 2023, "vector": [3.0, 3.1, 3.2]} ] } } |
The "options" will be like this:
{ "data_source": { "type": "Minio", "address": "localhost:9000", "accesskey_id": "minioadmin", "accesskey_secret": "minioadmin", "use_ssl": false, "bucket_name": "mybucket" }, "external_data": { "target_collection": "TEST", "files": [ { "file": file_1.json, "type": "row_based", "fields_mapping": { "data.rows.id": "uid", "data.rows.year": "year", "data.rows.vector": "embedding", } }, { "file": file_2.json, "type": "row_based", "fields_mapping": { "data.rows.id": "uid", "data.rows.year": "year", "data.rows.vector": "embedding", } } ] "default_fields": { "age": 0 }, } } |
User has some JSON files store data with the column-based format: file_1.json, file_2.json.
{ "table": { "columns": [ "id": [1, 2, 3], "year": [2021, 2022, 2023], "vector": [ [1.0, 1.1, 1.2], [2.0, 2.1, 2.2], [3.0, 3.1, 3.2] ] ] } } |
The "options" will be like this:
{ "data_source": { "type": "Minio", "address": "localhost:9000", "accesskey_id": "minioadmin", "accesskey_secret": "minioadmin", "use_ssl": false, "bucket_name": "mybucket" }, "external_data": { "target_collection": "TEST", "files": [ { "file": file_1.json, "type": "column_based", "fields_mapping": { "data.columns.id": "uid", "data.columns.year": "year", "data.columns.vector": "embedding", } }, { "file": file_2.json, "type": "column_based", "fields_mapping": { "data.columns.id": "uid", "data.columns.year": "year", "data.columns.vector": "embedding", } } ] "default_fields": { "age": 0 }, } } |
{ "table": { "columns": [ "id": [1, 2, 3], "year": [2021, 2022, 2023], "age": [23, 34, 21] ] ] } } |
The "options" will be like this:
{ "data_source": { "type": "Minio", "address": "localhost:9000", "accesskey_id": "minioadmin", "accesskey_secret": "minioadmin", "use_ssl": false, "bucket_name": "mybucket" }, "external_data": { "target_collection": "TEST", "files": [ { "file": file_1.json, "type": "column_based", "fields_mapping": { "data.columns.id": "uid", "data.columns.year": "year", "data.columns.age": "age", } }, { "file": file_2.npy, "type": "column_based", "fields_mapping": { "embedding": "embedding", } } ] } } |