Troubleshooting Pods Restart Loop On Bitwarden 2025.6.2 With MSSQL On RDS And EKS Deployment

Introduction

We encountered a critical issue after upgrading to version 2025.6.2 of the Bitwarden Helm chart. When deploying this version on our production infrastructure, all pods enter a restart loop with no logs returned, and the containers exit with code 137. This article delves into the intricacies of this problem, outlining the steps taken to reproduce it, the environment in which it occurs, and the troubleshooting efforts made to resolve it. The main objective is to provide a comprehensive overview of the issue, aiding in its diagnosis and resolution for both the Bitwarden team and other users who might encounter similar challenges. The issue's impact on our production environment necessitates a swift and effective solution to ensure the continued availability of our Bitwarden service. The details provided in this article aim to facilitate a deeper understanding of the underlying cause and potential remedies.

The Bitwarden platform is a critical component of our infrastructure, providing secure password management for our organization. Any disruption to its service can significantly impact productivity and security. Therefore, understanding and resolving this restart loop issue is of paramount importance. The following sections will explore the environment, steps to reproduce, and attempted solutions in detail. By documenting our experience, we hope to contribute to a collective knowledge base that benefits the broader Bitwarden community. This detailed account also serves as a reference for future troubleshooting efforts, ensuring that similar issues can be addressed more efficiently. The persistence of the issue across fresh installs and upgrades underscores its significance and the need for a thorough investigation. This article aims to be a valuable resource for anyone facing similar challenges with Bitwarden deployments in Kubernetes environments.

Problem Description

The pods on our Bitwarden deployment are stuck in a restart loop, rendering the service unavailable. This issue arose immediately after upgrading to version 2025.6.2 of the Bitwarden Helm chart. The containers exit with code 137, indicating a potential out-of-memory issue or a signal termination. However, initial attempts to address memory constraints by increasing limits to 2Gi did not resolve the problem. The absence of logs further complicates the diagnosis, making it challenging to pinpoint the exact cause of the restarts. This restart loop is a critical issue that prevents the Bitwarden service from functioning correctly. Understanding the root cause is essential to restoring the service and preventing future occurrences. The consistency of the issue across different environments (pre-production and production) suggests a fundamental incompatibility or bug in the new version.

The behavior observed is that all pods enter a continuous cycle of restarting, with no apparent progress towards a stable state. This pattern indicates a systemic problem rather than an isolated incident within a single pod. The containers' exit code 137 typically signifies that a process was terminated due to an out-of-memory (OOM) condition or a signal such as SIGKILL. However, our attempts to mitigate potential memory issues by increasing the memory limits have not yielded any positive results. The lack of logs from the pods is particularly concerning, as it deprives us of valuable diagnostic information that could help identify the source of the problem. Without logs, it is difficult to determine what processes are running, what errors are occurring, or what resources are being consumed. The reproduction of the issue in both pre-production and production environments underscores its severity and the need for a comprehensive solution. This article aims to provide a detailed account of the problem, the environment, and the troubleshooting steps taken, in the hope of facilitating a swift resolution.

Steps to Reproduce

To reproduce the issue, deploy Bitwarden version 2025.6.2 using the Helm chart on a Kubernetes cluster (AWS EKS 1.32) with MSSQL on RDS (version 16.00.4185.3.v1) as the database and EFS for storage. The issue manifests as a restart loop in all pods, with no logs generated and containers exiting with code 137. This reproducibility across multiple environments highlights the systematic nature of the problem. By following these steps, others can verify the issue and potentially contribute to its resolution. The specific combination of Bitwarden version, database type, and storage solution appears to be a key factor in triggering the restart loop. The ability to consistently reproduce the issue is crucial for effective debugging and testing of potential fixes. This section provides a clear and concise set of instructions for recreating the problem, ensuring that others can participate in the troubleshooting process.

The steps outlined above are designed to replicate the exact conditions under which we encountered the restart loop. This includes the specific versions of Kubernetes, MSSQL, and the Bitwarden Helm chart. The use of EFS for storage is also a critical component of our environment, and it may play a role in the issue. By adhering to these steps, other users can accurately assess whether they are experiencing the same problem and potentially share their findings. The importance of precise replication cannot be overstated in troubleshooting complex issues like this. Any deviation from the specified environment or steps could lead to different results and hinder the diagnostic process. The goal is to create a consistent and reliable method for triggering the restart loop, thereby facilitating a more efficient and collaborative approach to finding a solution. This detailed procedure ensures that anyone attempting to reproduce the issue has a clear and unambiguous guide to follow.

Detailed Steps

1. Set up an AWS EKS cluster version 1.32.
2. Provision an MSSQL instance on RDS, version 16.00.4185.3.v1.
3. Configure EFS for persistent storage.
4. Deploy Bitwarden version 2025.6.2 using the official Helm chart.
5. Monitor the pods for restart loops. They will typically start immediately after deployment.
6. Check the pod logs; they will likely be empty or contain minimal information.
7. Observe the container exit codes, which should be 137.

These detailed steps provide a granular breakdown of the process required to reproduce the issue. Each step is essential and contributes to replicating the environment in which the problem occurs. The use of specific versions of Kubernetes, MSSQL, and the Bitwarden Helm chart is crucial for consistency. The inclusion of EFS for persistent storage is another key factor, as it may be related to the root cause of the restart loop. By following these steps meticulously, users can ensure that they are accurately recreating the conditions under which the issue manifests. The emphasis on monitoring pods and checking logs and exit codes is important for verifying that the issue has been successfully reproduced. Empty logs and exit code 137 are strong indicators that the restart loop is occurring. This comprehensive guide serves as a valuable resource for anyone attempting to diagnose and resolve this problem.

Environment Details

Our environment consists of an AWS EKS 1.32 Kubernetes cluster, MSSQL on RDS (version 16.00.4185.3.v1) as the database, and EFS for storage. We are using Helm chart version 2025.6.2, having previously used 2025.5.3 (web 5.1) successfully. SSO is enabled with trusted device functionality, and we operate under an enterprise license in a production-grade setup. This specific combination of technologies and configurations may be a contributing factor to the issue. Understanding the environment in detail is crucial for identifying potential incompatibilities or conflicts. The use of MSSQL on RDS and EFS, in particular, may be relevant, as they represent key components of the data storage and persistence layers. The previous working version, 2025.5.3, serves as a valuable baseline for comparison, helping to narrow down the scope of the problem to changes introduced in 2025.6.2. The enterprise license and production-grade setup underscore the importance of resolving this issue promptly to maintain service availability and security.

The choice of AWS EKS as the Kubernetes platform is significant, as it provides a managed environment for deploying and scaling containerized applications. However, the specific version, 1.32, may have its own nuances and potential compatibility issues. Similarly, MSSQL on RDS offers a managed database service, but its interaction with Bitwarden in this particular configuration needs careful consideration. EFS, as a network file system, provides persistent storage for the Bitwarden pods, but its performance characteristics and potential bottlenecks should be evaluated. The fact that SSO is enabled with trusted device functionality adds another layer of complexity to the system. Any changes in the authentication or authorization mechanisms in Bitwarden 2025.6.2 could potentially interact with the SSO setup and contribute to the problem. The detailed environment description provided here is intended to be as comprehensive as possible, allowing for a thorough analysis of potential causes and solutions. This information is crucial for both the Bitwarden team and other users who may be experiencing similar issues.

Troubleshooting Steps

We attempted a fresh install of 2025.6.2, which resulted in the same issue. Upgrading from 2025.5.3 also led to the restart loop. Increasing memory limits to 2Gi (previously 512Mi) did not resolve the problem. Enabling development mode for more logs yielded no additional output. The issue was reproduced in both pre-production and production environments. These troubleshooting steps highlight our efforts to isolate and address the problem. The failure of multiple approaches suggests that the root cause is not immediately apparent and may require deeper investigation. The initial focus on memory limits was based on the container exit code 137, which often indicates an out-of-memory condition. However, the lack of improvement after increasing memory suggests that this is not the primary driver of the restart loop. The attempt to enable development mode was aimed at increasing the verbosity of logging, but the absence of additional output indicates that the issue may be occurring very early in the startup process, before logging is initialized. The consistent reproduction of the issue across different environments strengthens the case for a systematic problem related to Bitwarden 2025.6.2 and its interaction with our infrastructure.

The strategy employed in our troubleshooting efforts was to systematically eliminate potential causes, starting with the most likely suspects. The fresh install and upgrade attempts were intended to rule out any issues related to the existing deployment state. Increasing memory limits was a direct response to the container exit code, but its failure to resolve the problem led us to consider other factors. The attempt to enable development mode reflects our desire to gain more insight into the startup process and identify potential error points. The reproduction of the issue in both pre-production and production environments was a critical step in validating the problem and ruling out environment-specific configurations or anomalies. This also underscores the urgency of finding a solution, as the issue is not isolated to a test environment. The combination of these troubleshooting steps provides a comprehensive overview of our efforts to diagnose and address the restart loop. While these attempts have not yet yielded a solution, they have helped to narrow down the scope of the problem and provide valuable information for further investigation. This detailed account serves as a foundation for future troubleshooting efforts and collaboration with the Bitwarden team.

Suspicions and Hypotheses

We suspect a change in the startup script in version 2025.6.2 may be incompatible with our infrastructure, specifically the combination of Bitwarden, MSSQL on RDS, and EFS. This suspicion is based on the fact that the previous version, 2025.5.3, worked without issue. The specific interaction between Bitwarden's startup process and the database and storage layers is a potential area of concern. The hypothesis is that a change in how Bitwarden initializes or interacts with these components is triggering the restart loop. This could involve database connection issues, file system access problems, or other initialization errors. The absence of logs makes it difficult to pinpoint the exact nature of the incompatibility, but the consistent reproduction of the issue suggests a systematic problem rather than a transient error.

The focus on the startup script is driven by the observation that the pods are failing to initialize correctly. This suggests that the problem is occurring early in the lifecycle of the containers, before the main application logic is fully loaded. Changes in the startup script could introduce new dependencies, modify initialization sequences, or alter resource requirements in ways that are incompatible with our environment. The specific combination of MSSQL on RDS and EFS is also a key factor in our hypothesis. These components represent critical infrastructure elements that Bitwarden relies on for data storage and persistence. Any issues in their interaction with Bitwarden could lead to the observed restart loop. The absence of logs further reinforces the suspicion that the problem is occurring very early in the startup process, as the logging mechanisms may not be fully initialized before the containers fail. This hypothesis provides a framework for further investigation and testing, focusing on the startup script and its interaction with the database and storage layers.

Conclusion

The pods restart loop issue on Bitwarden 2025.6.2 with MSSQL on RDS (EKS Deployment) represents a critical challenge that requires immediate attention. Our systematic approach to troubleshooting, including fresh installs, upgrades, memory limit adjustments, and development mode activation, has not yet yielded a solution. We suspect an incompatibility between a change in the startup script in version 2025.6.2 and our infrastructure, particularly the combination of Bitwarden, MSSQL on RDS, and EFS. Further investigation and collaboration with the Bitwarden team are essential to identify the root cause and implement a fix. This issue's impact on our production environment underscores the importance of finding a resolution promptly to ensure the continued availability and security of our Bitwarden service. The detailed account provided in this article serves as a valuable resource for both the Bitwarden team and other users who may encounter similar challenges. We remain committed to working towards a solution and will continue to update this article with any new findings or developments.

The persistence of the issue across multiple environments and troubleshooting attempts highlights its complexity and the need for a comprehensive solution. The absence of logs further complicates the diagnostic process, making it challenging to pinpoint the exact cause of the restart loop. Our hypothesis regarding the startup script provides a starting point for further investigation, but additional testing and analysis may be required to confirm its validity. The collaboration with the Bitwarden team is crucial for leveraging their expertise and resources in addressing this issue. We believe that a collaborative approach, combining our understanding of our infrastructure with Bitwarden's knowledge of their software, will be the most effective way to find a resolution. This article serves as a living document, and we will continue to update it with any new information or progress made in resolving the restart loop. Our ultimate goal is to restore the Bitwarden service to full functionality and prevent future occurrences of this issue. This requires a thorough understanding of the root cause and the implementation of robust preventative measures.