Comparative Analysis of Default Buildpack Behavior Across Modern Web Frameworks

Abstract

Automatic source-to-container systems reduce the configuration required to deploy web applications, but their default behavior differs across programming languages, framework conventions, package managers, runtime metadata, and repository layouts. This paper presents an empirical comparison of five buildpack-style systems: Paketo Cloud Native Buildpacks, Heroku Cloud Native Buildpacks, Nixpacks, Railpack, and Zbpack. The benchmark covers 13 web framework families, 39 public repositories, 195 buildpack/repository cells, and 390 build runs. Each cell was evaluated with one cold build and one warm build. The primary metric is reliable build success, defined as both runs producing a container image. Runtime startup was intentionally excluded to isolate image-build behavior.

Railpack achieved the strongest result, with 32 reliable successes out of 39 cells (82.1%). Heroku Cloud Native Buildpacks followed with 25/39 (64.1%), Zbpack reached 24/39 (61.5%), Nixpacks reached 22/39 (56.4%), and Paketo Cloud Native Buildpacks reached 19/39 (48.7%). The results show that buildpack choice is not interchangeable: each system has a distinct profile across framework families and repository features. The most important failure causes were runtime version selection, unavailable runtime patch versions, package manager detection, missing start command inference, native dependency gaps, Dockerfile context assumptions, and production build scripts. These findings can inform repository-aware buildpack selection, where deployment platforms select a default builder based on framework family and repository contents rather than applying one universal build strategy.

Keywords

buildpacks; containerization; Cloud Native Buildpacks; Nixpacks; Railpack; Zbpack; deployment automation; repository analysis; web frameworks

1. Introduction

Containerized deployment is now a standard target for web applications, but producing a correct image from source code is still complex. A deployment system must infer the programming language, framework, runtime version, package manager, dependency installation steps, build command, filesystem layout, and sometimes the start command. Buildpack-style systems address this problem by inspecting repository contents and producing an image without requiring a hand-written Dockerfile.

This study compares default build behavior across five systems:

•          Paketo Cloud Native Buildpacks (CNB)

•          Heroku Cloud Native Buildpacks

•          Nixpacks

•          Railpack

•          Zbpack

The central research question is:

How reliably do different buildpack-style systems produce container images for different web framework families under default, no-intervention conditions?

The practical motivation is repository-aware buildpack selection. If different buildpacks have slight but repeatable advantages across framework families and repository contents, a deployment platform can choose a better default builder before attempting deployment. For example, the presence of a Ruby patch pin, pnpm-lock.yaml, packageManager, composer.lock, .ruby-version, go.mod, a framework-specific subdirectory, or a Dockerfile can change which buildpack is most likely to build successfully.

This paper measures image build success, not runtime correctness. A produced image can still require environment variables, external services, health checks, or a start command. However, successful image creation is a necessary deployment step and a useful basis for comparing buildpack behavior.

2. Related Background

Cloud Native Buildpacks define a lifecycle for transforming application source into container images. Detection is a core phase: buildpacks inspect application files and decide whether they apply to the project [1]. The lifecycle also includes build, export, and related phases that together produce the final image [2]. Paketo and Heroku both provide CNB builders, but they differ in buildpack sets, runtime support, and framework policies.

Nixpacks uses source inspection to generate a build plan and container image using Nix-based packages and generated build phases [4]. Railpack detects project metadata and builds application images through BuildKit [5, 6]. Zbpack is used in Zeabur’s deployment workflow and can infer build behavior or use repository Dockerfile paths depending on project structure [7].

This work is also related to deployment automation and reproducible build research. Deployment automation technologies attempt to make the transition from source code to running software more declarative and repeatable [9]. Reproducible build research shows that build outputs can be affected by toolchain versions, dependency repositories, and environment state [10]. Containerization improves portability, but containers alone do not guarantee reproducible builds because base images, package repositories, local daemon behavior, and build context can still vary [11].

3. Materials and Methods

3.1 Dataset

The benchmark covers 13 framework families with three public repositories per family where possible. The selected repositories were chosen to represent variation in size and structure: smaller templates or examples, medium applications, and larger real-world applications.

The framework families are Vite, Vue, Next.js, Nuxt, SvelteKit, Jekyll, Express, Django, FastAPI, Gin, Laravel, Symfony, and Rails. Earlier exploratory runs were excluded where local Docker failures or unsuitable repository choices would have distorted the benchmark. The final Rails set uses deployable Rails repositories because this paper compares buildpack behavior across framework families rather than stressing a single framework with unusually difficult applications.

3.2 Build Systems

All systems were evaluated in default mode. No project-specific tuning files, runtime overrides, or manual source edits were added to make a build pass. Paketo CNB and Heroku CNB were executed through the pack lifecycle with their respective builders. Nixpacks, Railpack, and Zbpack were executed through their default command-line interfaces. Railpack and Zbpack used BuildKit as part of their build process.

3.3 Metrics

Each buildpack/repository cell was run twice:

•          Run 1: cold build

•          Run 2: warm build

The primary metric is reliable success:

reliable success = cold build succeeded and warm build succeeded

Supporting metrics are any success, partial success, median cold build time, median warm build time, and median image size. Failed rows are excluded from build-time and image-size medians. Runtime startup probing was disabled, so a success means that an image was produced, not that the application started correctly.

3.4 Execution Environment

The benchmark ran locally under Docker Desktop with WSL2. Environment metadata was captured during report preparation on 2026-05-15.

Observed build tool versions were pack 0.40.2, Nixpacks 1.41.0, Railpack 0.23.0, and BuildKit 0.28.1. The local Zbpack binary did not expose a version flag in this setup.

4. Results

4.1 Overall Buildpack Reliability

Table 1. Reliable success and performance summary by build system.

Railpack had the highest reliable success rate and the smallest median successful image size. Heroku CNB was the second-most reliable system in this final dataset, while Zbpack was close behind. Nixpacks had very fast warm builds where it succeeded, but more failures on metadata-sensitive cases. Paketo CNB was limited by strict runtime availability and detection behavior in several ecosystems.

Reliable success rate by buildpack

Figure 1. Reliable success rate by build system.

4.2 Framework-Level Results

Table 2. Reliable success by framework family.

Go and conventional Node.js repositories were the easiest default-build targets. PHP, Python, and Ruby applications were more sensitive to runtime versions, native dependencies, lockfiles, production build scripts, and service assumptions.

Reliable success matrix by framework and buildpack

Figure 2. Reliable success matrix by framework family and build system.

4.3 Build Time and Image Size

Warm build times were usually much shorter than cold build times because successful first runs populated reusable layers, package caches, or previous images. Railpack and Nixpacks had the fastest median warm builds among successful cells. Railpack also produced the smallest median successful image size.

Median cold and warm build time

Figure 3. Median cold and warm build times for successful builds.

Median image size

Figure 4. Median image size for successful builds.

5. Buildpack Profiles by Framework Signal

The aggregate results show the overall ranking, but the more useful finding is that each buildpack has different advantages depending on repository contents.

Table 3. Framework-level buildpack selection signals from the benchmark.

These results suggest that buildpack selection should not be a fixed global default. A repository-aware selector should inspect both framework family and repository contents.

For example, a Nuxt repository with clear Node metadata is a strong Railpack or Heroku candidate, while a Go/Gin repository can be handled reliably by several buildpacks. A Ruby repository with an exact patch-level .ruby-version should be checked against the builder’s available Ruby versions before selecting Paketo CNB. A PHP repository should be checked for Composer lockfiles and required platform extensions before choosing a PHP-oriented build path.

6. Framework-by-Framework Reliability Diagnostics

The aggregate success rates hide the most important practical detail: each framework failed for different reasons. Table 4 summarizes the dominant causes behind the reliable-success results. These explanations come from the archived build logs and run summaries, and they show which repository signals should be inspected before selecting a buildpack.

Table 4. Framework-level reliability diagnostics by build system.

This table is intentionally repository-aware rather than framework-only. For example, “Next.js” alone was not enough to choose a buildpack: the difference between pnpm-lock.yaml alone and an explicit packageManager field changed Heroku’s outcome. Similarly, “Rails” alone was not enough: exact Ruby patch availability, Node engine constraints, Redis-dependent asset compilation, and PostgreSQL headers changed which buildpack succeeded. The relevant selection unit is therefore a framework plus its repository metadata.

7. Failure Analysis

The most common failure categories in the benchmark were metadata and environment inference failures rather than broad framework incompatibilities.

Table 5. Main failure categories observed in the benchmark.

Examples from the benchmark illustrate these patterns. Paketo CNB failed several Ruby applications because the builder did not provide the exact Ruby patch version required by repository metadata. Nixpacks failed several Nuxt and SvelteKit cases because it required or inferred a start command even though startup was outside this benchmark. Railpack failed one Rails case by selecting a Node patch version that was close but incompatible with the repository’s strict engine constraint. Zbpack failed a Rails case because the generated image lacked PostgreSQL development tooling required by the pg gem. Heroku CNB reached a Rails asset-precompile stage but failed when the application expected a production environment variable.

8. Discussion

8.1 Comparative Buildpack Behavior

Railpack was the strongest general default builder in this dataset. It combined broad framework detection, high reliable success, fast warm builds, and small median image size. Its advantage was clearest in frontend frameworks, Go/Gin, Jekyll, and deployable Rails.

Heroku CNB was strong on conventional Node, Nuxt, SvelteKit, Jekyll, Laravel, and deployable Rails repositories. Its failures tended to appear when production framework scripts required environment variables or when dependency metadata interacted poorly with selected runtime versions.

Nixpacks was strong on Express, Gin, Next.js, and Vite and had fast warm builds. Its weaknesses were missing start command requirements, generated build-context assumptions, and some native dependency cases.

Paketo CNB performed well on several clean paths, including Express, SvelteKit, Gin, FastAPI, Vite, and Vue. Its main weakness was strict detection and exact runtime availability, especially in Ruby and PHP cases.

Zbpack performed well on Gin, deployable Rails, Laravel, Symfony, and several frontend cases. Its risk was that generated or Dockerfile-aware behavior sometimes missed native dependencies or inherited repository assumptions that were not suitable for generic source deployment.

8.2 Repository-Aware Selection Strategy

The results support a two-stage selection strategy. First, identify the framework family from repository contents. Second, inspect high-impact metadata that changes buildpack suitability:

•          language runtime files such as .ruby-version, .python-version, go.mod, and PHP Composer constraints;

•          JavaScript package-manager files such as package-lock.json, pnpm-lock.yaml, yarn.lock, and packageManager;

•          framework-specific directories such as SvelteKit, Nuxt, Rails, Django, Laravel, or Symfony project structure;

•          native dependency signals such as pg, sqlite3, image libraries, or Python/Ruby native extensions;

•          Dockerfiles and whether they are source-buildable from the selected repository subpath;

•          production build scripts that load framework settings or require environment variables.

A selector using these signals would not merely ask “what framework is this?” It would ask “which buildpack is most compatible with this framework and this repository’s metadata?” This is the main practical finding of the study.

8.3 Interpreting Build Success

Partial success should be interpreted cautiously. If a cold build fails and a warm build succeeds, the buildpack has shown some ability to build the repository, but not reliable reproducibility from a clean state. The strict reliable success metric is therefore the appropriate primary metric for default deployment.

The framework-level results should also not be read as universal framework rankings. They describe how selected repositories behaved under default build systems. Repository selection and metadata quality materially affect the result.

9. Threats to Validity

9.1 Construct Validity

This study measures image build success, not runtime success. A successful image build does not prove that the application starts, serves traffic, passes health checks, or is production ready. The study also does not measure security, vulnerability posture, software bill of materials quality, memory use, or runtime performance.

9.2 Internal Validity

The benchmark was executed in a local container build environment. Earlier exploratory runs exposed disk pressure, daemon instability, and local export failures. Corrected results were selected where such issues were identified. Nevertheless, local infrastructure can affect cold build time, image export behavior, and cache behavior.

Warm builds are affected by Docker cache, BuildKit cache, CNB restore behavior, and previous images. The harness reused the same image name across cold and warm runs, but warm-build timing should still be interpreted as tool-plus-environment behavior rather than pure algorithmic performance.

9.3 External Validity

Each framework family is represented by three repositories. This captures variation across repository size and shape but cannot represent the full diversity of each ecosystem. Results are directional evidence about buildpack behavior across framework families, not universal statements about all applications in a framework.

9.4 Conclusion Validity

Build-time and image-size medians are computed only from successful builds. This prevents failed rows from distorting performance data but means that performance comparisons are conditional on success. A buildpack with fewer successful cells may appear fast or small because it succeeded on an easier subset.

Failure categories were manually classified from archived logs and run summaries. The categories explain dominant root causes but should not be treated as a complete formal taxonomy of all possible build failures.

10. Artifact Availability

The benchmark results, aggregate tables, figures, and supporting logs are retained by the authors. A cleaned artifact package can be prepared for review or publication. It should include the repository list, raw result tables, aggregate summaries, generated figures, and enough environment metadata to reproduce the analysis without exposing unrelated working-directory details.

11. Conclusion

This study shows that buildpack choice significantly affects default image-build reliability across web frameworks. In the final benchmark dataset, Railpack achieved the highest reliable success rate at 82.1%, followed by Heroku CNB at 64.1%, Zbpack at 61.5%, Nixpacks at 56.4%, and Paketo CNB at 48.7%.

The main finding is that buildpack advantages are framework- and repository-dependent. Railpack was the strongest general default builder, but other systems had clear strengths in particular contexts. Heroku CNB performed well for conventional Node, Jekyll, Laravel, and deployable Rails repositories. Nixpacks was strong for Express, Gin, Next.js, and Vite. Zbpack was competitive for Gin, PHP frameworks, Django, and deployable Rails, but could miss native dependency requirements. Paketo CNB was effective on several clean paths but more sensitive to exact runtime availability and detection constraints.

Most failed builds were systematic inference failures rather than broad framework incompatibilities. Runtime metadata, package manager files, native dependencies, Dockerfile context, and production build scripts were decisive. Therefore, an effective deployment platform should select buildpacks based on repository contents, not only on framework labels or a single global default.

References

[1] Cloud Native Buildpacks. “Detect.” Available: https://buildpacks.io/docs/for-platform-operators/concepts/lifecycle/detect/

[2] Cloud Native Buildpacks. “What is the lifecycle?” Available: https://buildpacks.io/docs/for-buildpack-authors/concepts/lifecycle-phases/

[3] Heroku Dev Center. “Managing Buildpacks.” Available: https://devcenter.heroku.com/articles/managing-buildpacks

[4] Nixpacks. “Getting Started.” Available: https://nixpacks.com/docs/getting-started

[5] Railpack. “Getting Started.” Available: https://railpack.com/getting-started

[6] Railway Docs. “Railpack.” Available: https://docs.railway.com/builds/railpack

[7] Zeabur Docs. “How Deploys Work: Builds.” Available: https://zeabur.com/docs/en-US/deploy/how-deploys-work

[8] ACM. “Artifact Review and Badging - Current.” Available: https://www.acm.org/publications/policies/artifact-review-and-badging-current

[9] C. Quinton, R. Rabiser, M. Vierhauser, P. Grünbacher, and L. Baresi. “The Essential Deployment Metamodel: A Systematic Review of Deployment Automation Technologies.” Available: https://arxiv.org/abs/1905.07314

[10] C. Lamb and S. Zacchiroli. “Reproducible Builds: Increasing the Integrity of Software Supply Chains.” Available: https://arxiv.org/abs/2104.06020

[11] D. Chondros, G. Psomakis, M. Anagnostou, and D. Spinellis. “Docker Does Not Guarantee Reproducibility.” Available: https://arxiv.org/abs/2601.12811