...
We introduce embedded Milvus in this MEP.
With embedded Milvus, you just need a clean environment with Python installed. You can then just do:
| Code Block | ||||
|---|---|---|---|---|
| ||||
$ pip install pymilvus # Install pymilvus. |
...
$ python3 |
...
>>> From ... import milvus # Milvus is brought up here. |
...
>>> hello_milvus = Collection(…) |
...
>>> insert_result = hello_milvus.insert(...) |
...
>>> hello_milvus.create_index(…) |
...
>>> hello_milvus.load(...) |
...
>>> search_result = hello_milvus.search(…) |
...
hit: (distance: 0.0, id: 2998), random field: -11.0 |
...
hit: (distance: 0.11455299705266953, id: 1581), random field: -18.0 |
...
hit: (distance: 0.1232629269361496, id: 2647), random field: -13.0 |
...
hit: (distance: 0.0, id: 2999), random field: -11.0 |
...
hit: (distance: 0.10560893267393112, id: 2430), random field: -18.0 |
...
hit: (distance: 0.13938161730766296, id: 377), random field: -14.0 |
...
search latency = 0.2796s |
...
>>> exit() # Milvus is shut down, but all data lives on. |
The code piece above is pretty self-explanatory.
You don't need any Milvus server pre-installed. And of course you don't need to keep any Milvus process running in the meantime.
The embedded Milvus starts and exits whenever you wish it to, but all data and logs persist.
We believe this embedded Milvus version makes Milvus a real "DB for AI" as it would make Milvus extremely easy to use for data scientists and AI engineers, etc.
...
Up until version 2.0, Milvus is a typical go project published as a go binary. Milvus also has a typical go code structure, as shown below:milvus
├── cmd // Entrance of Milvus binary. (the main() function)
├── internal // Milvus code.
├── configs
├── deployments
├── docs
├── scripts
├──
| Code Block | ||||
|---|---|---|---|---|
| ||||
milvus ├── cmd // Entrance of Milvus binary. (the main() function) ├── internal // Milvus code. ├── configs ├── deployments ├── docs ├── scripts ├── ... |
The first thing an embedded Milvus needs is to publish Milvus as a library.
Following the go code structure convention, we just need to add another layer to Milvus named pkg where we can export Milvus as a library:
| Code Block |
|---|
...
| ||||
milvus ├── cmd // Entrance of Milvus binary. (the main() function) |
...
├── internal // Milvus code. |
...
├── pkg |
...
│ ├── embedded // Where embedded Milvus code lies. |
...
│ ├── ... |
...
├── configs |
...
├── deployments |
...
├── docs |
...
├── scripts |
...
├── ... |
Running Go From Python
The goal is to use go code in Python. One option is to use CGO. We could export (//export)go code as a C library libmilvus.so and have Python to import this library:
| Code Block | ||||
|---|---|---|---|---|
| ||||
milvus: build-cpp print-build-info
@echo "Building Milvus ..."
@mkdir -p $(INSTALL_PATH) && go env -w CGO_ENABLED="1" && GO111MODULE=on $(GO) build \
-ldflags="-X 'main.BuildTags=$(BUILD_TAGS)' -X 'main.BuildTime=$(BUILD_TIME)' -X 'main.GitCommit=$(GIT_COMMIT)' -X 'main.GoVersion=$(GO_VERSION)'" \
-buildmode=c-shared-o $(INSTALL_PATH)/libmilvus.so $(PWD)/pkg/embedded/embeddedmilvus.go 1>/dev/null |
milvus: build-cpp print-build-info
@echo "Building Milvus ..."
@mkdir -p $(INSTALL_PATH) && go env -w CGO_ENABLED="1" && GO111MODULE=on $(GO) build \
-ldflags="-X 'main.BuildTags=$(BUILD_TAGS)' -X 'main.BuildTime=$(BUILD_TIME)' -X 'main.GitCommit=$(GIT_COMMIT)' -X 'main.GoVersion=$(GO_VERSION)'" \
-o $(INSTALL_PATH)/milvus $(PWD)/cmd/main.go 1>/dev/null
embedded-milvus: build-cpp print-build-info
@echo "Building Embedded Milvus …" @go env -w CGO_ENABLED="1" && GO111MODULE=on $(GO) build \
-ldflags="$(LD_FLAGS)" \
-buildmode=c-shared -o $(INSTALL_PATH)/libmilvus.so \
$(PWD)/pkg/embedded/embeddedmilvus.go 1>/dev/null
...