Advanced Containerization#

time expected: 12 minutes

This guide describes advanced containerization options provided by BentoML:

Using base image
Using dockerfile template

This is an advanced feature for user to customize container environment that are not directly supported in BentoML. For basic containerizing options, see Docker Options.

Why you may need this?#

If you want to customize the containerization process of your Bento.
If you need a certain tools, configs, prebuilt binaries that is available across all your Bento generated container images.
A big difference with base image features is that you don’t have to setup a custom base image and then push it to a remote registry.

Custom Base Image#

If none of the provided distros work for your use case, e.g. if your infrastructure requires all docker images to be derived from the same base image with certain security fixes and libraries, you can config BentoML to use your base image instead:

docker:
    base_image: "my_custom_image:latest"

When a base_image is provided, all other docker options will be ignored, (distro, cuda_version, system_packages, python_version). bentoml containerize will build a new image on top of the base_image with the following steps:

setup env vars
run the setup_script if provided
install the required Python packages
copy over the Bento file
setup the entrypoint command for serving.

Note

Warning: user must ensure that the provided base image has desired Python version installed. If the base image you have doesn’t have Python, you may install python via a setup_script. The implementation of the script depends on the base image distro or the package manager available.

docker:
    base_image: "my_custom_image:latest"
    setup_script: "./setup.sh"

Warning

By default, BentoML supports multi-platform docker image build out-of-the-box. However, when a custom base_image is provided, the generated Dockerfile can only be used for building linux/amd64 platform docker images.

If you are running BentoML from an Apple M1 device or an ARM based computer, make sure to pass the --opt platform=linux/amd64 parameter when containerizing a Bento. e.g.:

bentoml containerize iris_classifier:latest --opt platform=linux/amd64

Dockerfile Template#

The dockerfile_template field gives the user full control over how the Dockerfile is generated for a Bento by extending the template used by BentoML.

First, create a Dockerfile.template file next to your bentofile.yaml build file. This file should follow the Jinja2 template language, and extend BentoML’s base template and blocks. The template should render a valid Dockerfile. For example:

{% extends bento_base_template %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
RUN echo "We are running this during bentoml containerize!"
{% endblock %}

Then add the path to your template file to the dockerfile_template field in your :code: bentofile.yaml:

docker:
    dockerfile_template: "./Dockerfile.template"

Now run bentoml build to build a new Bento. It will contain a Dockerfile generated with the custom template. To confirm the generated Dockerfile works as expected, run bentoml containerize <bento> to build a docker image with it.

Examples#

Building TensorFlow custom op
Access AWS credentials during image build

Building TensorFlow custom op#

Let’s start with an example that builds a custom TensorFlow op binary into a Bento, which is based on zero_out.cc implementation details:

Define the following Dockerfile.template:

Dockerfile.template#

{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}

{{ super() }}

WORKDIR /tmp

COPY ./src/tfops/zero_out.cc .

RUN pip3 install tensorflow
RUN set -ex && \
	TF_CFLAGS=( $(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') ) && \
	TF_LFLAGS=( $(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') ) && \
	g++ --std=c++14 -shared zero_out.cc -o zero_out.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -I$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_include());') -D_GLIBCXX_USE_CXX11_ABI=0 -O2

{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
RUN stat /usr/lib/zero_out.so
{% endblock %}

Then add the following to your bentofile.yaml:

include:
  - "zero_out.cc"
python:
  packages:
  - tensorflow
docker:
  dockerfile_template: ./Dockerfile.template

Proceed to build your Bento with bentoml build and containerize with bentoml containerize:

bentoml build

bentoml containerize <bento>:<tag>

Tip

You can also provide --progress plain to see the progress from buildkit in plain text

bentoml containerize --progress plain <bento>:<tag>

Access AWS credentials during image build#

We will now demonstrate how to provide AWS credentials to a Bento via two approaches:

Using environment variables.
Mount credentials from host.

Note

Remarks: We recommend for most cases to use the second option (Mount credentials from host) as it prevents any securities leak.

By default BentoML uses the latest dockerfile frontend which allows mounting secrets to container.

For both examples, you will need to add the following to your bentofile.yaml:

python:
  packages:
  - awscli
docker:
  dockerfile_template: ./Dockerfile.template

Using environment variables#

Define the following Dockerfile.template:

{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_ACCESS_KEY_ID
{{ super() }}

ARG AWS_SECRET_ACCESS_KEY
ARG AWS_ACCESS_KEY_ID

ENV AWS_SECRET_ACCESS_KEY=$ARG AWS_SECRET_ACCESS_KEY
ENV AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}

RUN aws s3 cp s3://path/to/file {{ bento__path }}

{% endblock %}

After building the bento with bentoml build, you can then pass AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID as arguments to bentoml containerize:

bentoml containerize --build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
                     --build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
                     <bento>:<tag>

Mount credentials from host#

Define the following Dockerfile.template:

{% extends bento_base_template %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}

RUN --mount=type=secret,id=aws,target=/root/.aws/credentials \
     aws s3 cp s3://path/to/file {{ bento__path }}

{% endblock %}

Follow the above addition to bentofile.yaml to include awscli and the custom dockerfile template.

To pass in secrets to the Bento, pass it via --secret to bentoml containerize:

bentoml containerize --secret id=aws,src=$HOME/.aws/credentials <bento>:<tag>

See also

Mounting Secrets

Writing `dockerfile_template`#

BentoML utilize Jinja2 to structure a Dockerfile.template.

The Dockerfile template is a mix between Jinja2 syntax and Dockerfile syntax. BentoML set both trim_blocks and lstrip_blocks in Jinja templates environment to True.

Note

Make sure that your Dockerfile instruction is unindented as if you are writting a normal Dockerfile.

See also

Jinja Whitespace Control.

An example of a Dockerfile template takes advantage of multi-stage build to isolate the installation of a local library mypackage:

{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}
FROM --platform=$BUILDPLATFORM python:3.7-slim as buildstage
RUN mkdir /tmp/mypackage

WORKDIR /tmp/mypackage/
COPY mypackage .
RUN python setup.py sdist && mv dist/mypackage-0.0.1.tar.gz mypackage.tar.gz

{{ super() }}
{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
COPY --from=buildstage mypackage.tar.gz /tmp/wheels/
RUN --network=none pip install --find-links /tmp/wheels mypackage
{% endblock %}

Note

Notice how for all Dockerfile instruction, we consider as if the Jinja logics aren’t there 🚀.

Jinja templates#

One of the powerful features Jinja offers is its template inheritance. This allows BentoML to enable users to fully customize how to structure a Bento’s Dockerfile.

Note

To use a custom Dockerfile template, users have to provide a file with a format that follows the Jinja2 template syntax. The template file should have extensions of .j2, .template, .jinja.

Note

This section is not meant to be a complete reference on Jinja2. For any advanced features from on Jinja2, please refers to their Templates Design Documentation.

To construct a custom Dockerfile template, users have to provide an extends block at the beginning of the Dockerfile template Dockerfile.template followed by the given base template name bento_base_template:

{% extends bento_base_template %}

Tip

Warning: If you pass in a generic Dockerfile file, and then run bentoml build to build a Bento and it doesn’t throw any errors.

However, when you try to run bentoml containerize, this won’t work.

This is an expected behaviour from Jinja2, where Jinja2 accepts any file as a template.

We decided not to put any restrictions to validate the template file, simply because we want to enable users to customize to their own needs.

`{{ super() }}`#

As you can notice throughout this guides, we use a special function {{ super() }}. This is a Jinja features that allow users to call content of parent block. This enables users to fully extend base templates provided by BentoML to ensure that the result Bentos can be containerized.

See also

{{ super() }} Syntax for more information on template inheritance.

Blocks#

BentoML defines a sets of Blocks under the object bento_base_template.

All exported blocks that users can use to extend are as follow:

Blocks	Definition
`SETUP_BENTO_BASE_IMAGE`	Instructions to set up multi architecture supports, base images as well as installing system packages that is defined by users.
`SETUP_BENTO_USER`	Setup bento users with correct UID, GID and directory for a 🍱.
`SETUP_BENTO_ENVARS`	Add users environment variables (if specified) and other required variables from BentoML.
`SETUP_BENTO_COMPONENTS`	Setup components for a 🍱 , including installing pip packages, running setup scripts, installing bentoml, etc.
`SETUP_BENTO_ENTRYPOINT`	Finalize ports and set `ENTRYPOINT` and `CMD` for the 🍱.

Note

All the defined blocks are prefixed with SETUP_BENTO_*. This is to ensure that users can extend blocks defined by BentoML without sacrificing the flexibility of a Jinja template.

To extend any given block, users can do so by adding {{ super() }} at any point inside block.

Dockerfile instruction#

See also

Dockerfile reference for writing a Dockerfile.

We recommend that users should use the following Dockerfile instructions in their custom Dockerfile templates: ENV, RUN, ARG. These instructions are mostly used and often times will get the jobs done.

The use of the following instructions can be potentially harmful. They should be reserved for specialized advanced use cases.

Instruction	Reasons not to use
`FROM`	Since the containerized Bento is a multi-stage builds container, adding `FROM` statement will result in failure to containerize the given Bento.
`SHELL`	BentoML uses heredoc syntax and using `bash` in our containerization process. Hence changing `SHELL` will result in failure.
`CMD`	Changing `CMD` will inherently modify the behaviour of the bento container where docker won’t be able to run the bento inside the container. More below

The following instructions should be used with caution:

`WORKDIR`#

See also

WORKDIR reference

Since WORKDIR determines the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile, make sure that your instructions define the correct path to any working files.

Note

By default, all paths for Bento-related files will be generated to its fspath, which ensures that Bento will work regardless of WORKDIR

`ENTRYPOINT`#

See also

ENTRYPOINT reference

The flexibility of a Jinja template also brings up the flexibility of setting up ENTRYPOINT and CMD.

From Dockerfile documentation:

Only the last ENTRYPOINT instruction in the Dockerfile will have an effect.

By default, a Bento sets:

ENTRYPOINT [ "{{ bento__entrypoint }}" ]

CMD ["bentoml", "serve", "{{ bento__path }}"]

This aboved instructions ensure that whenever docker run is invoked on the 🍱 container, bentoml is called correctly.

In scenarios where one needs to setup a custom ENTRYPOINT, make sure to use the ENTRYPOINT instruction under the SETUP_BENTO_ENTRYPOINT block as follows:

{% extends bento_base_template %}
{% block SETUP_BENTO_ENTRYPOINT %}
{{ super() }}

...
ENTRYPOINT [ "{{ bento__entrypoint }}", "python", "-m", "awslambdaric" ]
{% endblock %}

Tip

{{ bento__entrypoint }} is the path the BentoML entrypoint, nothinig special here 😏.

Read more about CMD and ENTRYPOINT interaction here.

Advanced Options#

The next part goes into advanced options. Skip this part if you are not comfortable with using it.

Dockerfile variables#

BentoML does expose some variables that user can modify to fit their needs.

The following are the variables that users can set in their custom Dockerfile template:

Variables	Description
`bento__home`	Setup bento home, default to `/home/{{ bento__user }}`
`bento__user`	Setup bento user, default to `bentoml`
`bento__uid_gid`	Setup UID and GID for the user, default to `1034:1034`
`bento__path`	Setup bento path, default to `/home/{{ bento__user }}/bento`

If any of the aforementioned fields are set with {% set ... %}, then we will use your value instead, otherwise a default value will be used.

Adding `conda` to CUDA-enabled Bento#

Tip

Warning: miniconda install scripts provided by ContinuumIO (the parent company of Anaconda) supports Python 3.7 to 3.9. Make sure that you are using the correct python version under docker.python_version.

If you need to use conda for CUDA images, use the following template ( partially extracted from ContinuumIO/docker-images ):

Expands me

Dockerfile.template#

{% import '_macros.j2' as common %}
{% extends bento_base_template %}
{# Make sure to change the correct python_version and conda version accordingly. #}
{# example: py38_4.10.3 #}
{# refers to https://repo.anaconda.com/miniconda/ for miniconda3 base #}
{% set conda_version="py39_4.11.0" %}
{% set conda_path="/opt/conda" %}
{% set conda_exec=[conda_path, "bin", "conda"] | join("/") %}
{% block SETUP_BENTO_BASE_IMAGE %}
FROM debian:bullseye-slim as conda-build

RUN --mount=type=cache,from=cached,sharing=shared,target=/var/cache/apt \
	--mount=type=cache,from=cached,sharing=shared,target=/var/lib/apt \
	apt-get update -y && \
	apt-get install -y --no-install-recommends --allow-remove-essential \
	software-properties-common \
	bzip2 \
	ca-certificates \
	git \
	libglib2.0-0 \
	libsm6 \
	libxext6 \
	libxrender1 \
	mercurial \
	openssh-client \
	procps \
	subversion \
	wget && \
	apt-get clean

ENV PATH {{ conda_path }}/bin:$PATH

ARG CONDA_VERSION={{ conda_version }}

RUN set -ex && \
	UNAME_M=$(uname -m) && \
	if [ "${UNAME_M}" = "x86_64" ]; then \
	MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-x86_64.sh"; \
	SHA256SUM="4ee9c3aa53329cd7a63b49877c0babb49b19b7e5af29807b793a76bdb1d362b4"; \
	elif [ "${UNAME_M}" = "s390x" ]; then \
	MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-s390x.sh"; \
	SHA256SUM="e5e5e89cdcef9332fe632cd25d318cf71f681eef029a24495c713b18e66a8018"; \
	elif [ "${UNAME_M}" = "aarch64" ]; then \
	MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-aarch64.sh"; \
	SHA256SUM="00c7127a8a8d3f4b9c2ab3391c661239d5b9a88eafe895fd0f3f2a8d9c0f4556"; \
	elif [ "${UNAME_M}" = "ppc64le" ]; then \
	MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-ppc64le.sh"; \
	SHA256SUM="8ee1f8d17ef7c8cb08a85f7d858b1cb55866c06fcf7545b98c3b82e4d0277e66"; \
	fi && \
	wget "${MINICONDA_URL}" -O miniconda.sh -q && echo "${SHA256SUM} miniconda.sh" > shasum && \
	if [ "${CONDA_VERSION}" != "latest" ]; then \
	sha256sum --check --status shasum; \
	fi && \
	mkdir -p /opt && \
	sh miniconda.sh -b -p {{ conda_path }} && rm miniconda.sh shasum && \
	find {{ conda_path }}/ -follow -type f -name '*.a' -delete && \
	find {{ conda_path }}/ -follow -type f -name '*.js.map' -delete && \
	{{ conda_exec }} clean -afy

{{ super() }}

ENV PATH {{ conda_path }}/bin:$PATH

COPY --from=conda-build {{ conda_path }} {{ conda_path }}

RUN set -ex && \
	ln -s {{ conda_path }}/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
	echo ". {{ conda_path }}/etc/profile.d/conda.sh" >> ~/.bashrc && \
	echo "{{ conda_exec }} activate base" >> ~/.bashrc

{% endblock %}
{% block SETUP_BENTO_ENVARS %}

SHELL [ "/bin/bash", "-eo", "pipefail", "-c" ]
{{ super() }}
{{ common.setup_conda(__python_version__, bento__path, conda_path=conda_path) }}
{% endblock %}

Containerization with different container engines#

In BentoML version 1.0.11 [1], we support different container engines aside from docker.

BentoML-generated Dockerfiles from version 1.0.11 onward will be OCI-compliant and can be built with:

To use any of the aforementioned backends, they must be installed on your system. Refer to their documentation for installation and setup.

Note

By default, BentoML will use Docker as the container backend. To use other container engines, please set the environment variable BENTOML_CONTAINERIZE_BACKEND or pass in --backend to bentoml containerize:

# set environment variable
BENTOML_CONTAINERIZE_BACKEND=buildah bentoml containerize pytorch-mnist

# or pass in --backend
bentoml containerize pytorch-mnist:latest --backend buildah

To build a BentoContainer in Python, you can use the Container SDK method bentoml.container.build():

import bentoml

bentoml.container.build(
   "pytorch-mnist:latest",
   backend="podman",
   features=["grpc","grpc-reflection"],
   cache_from="registry.com/my_cache:v1",
)

Register custom backend#

To register a new backend, there are two functions that need to be implemented:

arg_parser_func: a function that takes in keyword arguments that represents the builder commandline arguments and returns a list[str]:

def arg_parser_func(
    *,
    context_path: str = ".",
    cache_from: Optional[str] = None,
    **kwargs,
) -> list[str]:
    if cache_from:
        args.extend(["--cache-from", cache_from])
    args.append(context_path)
    return args

health_func: a function that returns a bool to indicate if the backend is available:

import shutil

def health_func() -> bool:
    return shutil.which("limactl") is not None

To register a new backend, use bentoml.container.register_backend():

from bentoml.container import register_backend

register_backend(
   "lima",
   binary="/usr/bin/limactl",
   buildkit_support=True,
   health=health_func,
   construct_build_args=arg_parser_func,
   env={"DOCKER_BUILDKIT": "1"},
)

BuildKit interop#

BentoML leverages BuildKit for a more extensive feature set. However, we recognise that BuildKit has come with a lot of friction for migration purposes as well as restrictions to use with other build tools (such as podman, buildah, kaniko).

Therefore, since BentoML version 1.0.11, BuildKit will be an opt-out. To disable BuildKit, pass DOCKER_BUILDKIT=0 to bentoml containerize, which aligns with the behaviour of docker build:

$ DOCKER_BUILDKIT=0 bentoml containerize ...

Note

All Bento container will now be following OCI spec instead of Docker spec. The difference is that in OCI spec, there is no SHELL argument.

Note

The generated Dockerfile included inside the Bento will be a minimal Dockerfile, which ensures compatibility among build tools. We encourage users to always use bentoml containerize.

If you wish to use the generated Dockerfile, make sure that you know what you are doing!

CLI enhancement#

To better support different backends, bentoml containerize will be more agnostic when it comes to parsing options.

One can pass in options for specific backend with --opt:

$ bentoml containerize pytorch-mnist:latest --backend buildx --opt platform=linux/arm64

--opt also accepts parsing :

$ bentoml containerize pytorch-mnist:latest --backend buildx --opt platform:linux/arm64

Note

If you are seeing a warning message like:

'--platform=linux/arm64' is now deprecated, use the equivalent '--opt platform=linux/arm64' instead.

BentoML used to depends on Docker buildx. These options are now backward compatible with --opt. You can safely ignore this warning and use --opt to pass options for --backend=buildx.

Notes

Advanced Containerization#

Why you may need this?#

Custom Base Image#

Dockerfile Template#

Examples#

Building TensorFlow custom op#

Access AWS credentials during image build#

Using environment variables#

Mount credentials from host#

Writing dockerfile_template#

Jinja templates#

{{ super() }}#

Blocks#

Dockerfile instruction#

WORKDIR#

ENTRYPOINT#

Advanced Options#

Dockerfile variables#

Adding conda to CUDA-enabled Bento#

Containerization with different container engines#

Register custom backend#

BuildKit interop#

CLI enhancement#

Writing `dockerfile_template`#

`{{ super() }}`#

`WORKDIR`#

`ENTRYPOINT`#

Adding `conda` to CUDA-enabled Bento#