Slow build/export time

Habitat Help
#1

Hello, I’m trying to improve the building time of of a docker image. Currently I have a plan that looks likes:

pkg_name=Application
pkg_origin=company
pkg_version=“0.0.3”
pkg_license=(‘Apache-2.0’)
pkg_deps=(core/jre8 core/glibc core/gcc-libs core/curl)
pkg_build_deps=(core/git)
pkg_exports=(
[port]=server.port
)

pkg_exposes=(port)

pkg_svc_user=“root”

do_download() {
build_line " do_download() ================================================== "

if [ ! -d depedency-repository ]; then
git clone --depth 1 https://private.gitlab.repository.git
else
git -C depedency-repository pull origin master
fi
pushd depedency-repository
tar -cf dependency.tar.gz dependency
popd

export VERSION=${pkg_version}
jar_filename=Application-$VERSION-exec.jar;
if [ ! -e proprietary.jar ]; then
wget --header “Authorization:Basic some-token” https://private-maven-repositor/artifactory/local/br/com/company/Application/VERSION/{jar_filename} -O proprietary.jar
fi

}

do_verify() {
return 0;
}

do_build() {
return 0;
}

do_install() {
build_line " do_install() ================================================== "
mkdir {pkg_prefix}/shipyard_files mkdir {pkg_prefix}/tmp
mv /src/habitat/dependency/dependency.tar.gz {pkg_prefix} mv /src/habitat/proprietary.jar {pkg_prefix}
}

But when i try to export the .hart to docker it take almost 4 minutes to export.

real 3m40.905s
user 3m5.497s
sys 0m4.787s

The image is currently being builded in a machine with 24 cores and 128GB of ram, I’m guessing this is not a hardware problem. Can anyone help me identifying where is the problem with my plan?

how can i achieve faster build times?

#2

@GustavooPaiva I’m assuming this might be just some editing in the forum but {pkg_prefix} should be ${pkg_prefix}.

How big is the resulting hart file? Do you have the build times for exporting this jar to a container with just a regular dockerfile? The exporter doesn’t currently leverage docker layering so our build times are generally a little slower but they shouldn’t be significantly slower

#3

Yes, It removed some of the ‘$’, but that part is okay.

The hart file is 219MB. I do not have tried exporting with a regular docker file, habitat was my first try to dockerize my application.

Is there anyway to create a ‘base’ docker and increment with only the jar, for example?

Sorry if I’m not getting the right terms, I’m really new to the habitat/docker world :slight_smile:

#4

No worries, everyone starts at the beginning :smiley:. You can’t currently layer docker images in Habitat. a 219M file isn’t huge and shouldn’t be taking 4 minutes to export. Can you run the export with RUST_LOG=debug set and paste in the output?

#5

I tried to upload a text file, but that was not possible. I’m posting a link to pastebin here, if there are a better way to do it, I can repost, but here is the output: https://pastebin.com/eNhfW2gd

#6

Well looking at the output it looks like the extracted container size is just over a gig which would explain the long export time:

Sending build context to Docker daemon 1.072GB

#7

Indeed I have a very large dependency tar, it is 400MB. But even without adding this dependency, it is still slow (2 minutes) or is this time acceptable to building an image?

Here is the pastebin: https://pastebin.com/18scVLbM

Some remarks, I do need to have this dependency.

I’m starting to guess the way I’m building an image. Currently, we’re trying to build a docker image for every single version of my application.

Maybe, Should I do an image that is capable of downloading any version of it and running? So I only have to create an image once and with the configs it downloads the latest or an specific version?

#8

I don’t think there is a good way of getting the build time down without layering the image. It’s a pretty constant cost of moving bits into the containers mounted space

#9

Oh, I see :frowning: Is it on the roadmap? Anyway I can follow the development?

Anyway, thank you very much for your time.

#10

It was actually something we supported previously and removed. I think @christophermaier may have some context there.

1 Like
#11

Here is the PR that was submitted to introduce layers. https://github.com/habitat-sh/habitat/pull/1207. It was created to help java devs as its not the application layer that costed the most, it was the damn JRE layer. :stuck_out_tongue:

You can read along but the gist is the docker exporter got rewritten and if there is a need for layers, we should submit a request for a new feature. :slight_smile:

2 Likes