Dear Xen Community,
We regret to inform you that the Xen Project is currently experiencing unexpected changes due to the sudden shutdown of our colocated (colo) data center facility by Synoptek. This incident is beyond our control and will impact the continuity of OSSTest (the gating Xen Project CI loop), which we heavily rely on as a project. We understand the inconvenience this may cause to our users and stakeholders, and we are working diligently to consider different solutions as swiftly as possible.
What Happened?
We received a notification from our colo facility provider Synoptek indicating the shutdown of their services in Marlborough, where our current facility is located. As a result, the OSSTest instance hosted at this facility will likely be disrupted with an anticipated shutdown date of October 31st, 2024.
Potential Impact
The potential consequence of this shutdown may mean availability of OSSTest will be impacted, therefore affecting the number of bugs being caught. Contributors may experience quality issues with code, or face slower response times as our technical community spends more time resolving issues. We are aware of the critical nature of the services we provide and the significant impact this downtime may have.
Our Response
We are in constant communication with the colo facility provider, our IT contractors, and our Board Members to understand the scope of the problem and the estimated time for resolution. Additionally, we are exploring alternative solutions to restore our services as quickly as possible. These measures include:
Engaging with the Colo Facility Provider (Synoptek): We are working with the provider to negotiate costs and the likelihood of us moving to another facility as this was an unforeseen change.
Engaging with third parties: We have initiated discussions with our IT contractors and third-party providers to find potential alternatives, including other location facilities and donations of funds to support this move.
Options we are considering:
- Move our current hardware to a different co-location.
- This could be another provider, a member of the Advisory Board, a University, or a similar organization that offers services for Open Source projects.
- This could include moving all the infrastructure, or just part of it, depending on costs and space available at the destination.
- Abandon our hardware, and either re-create OSSTest from scratch at a different location or switch to a different test system (Gitlab CI).
All options have risks associated.
Further Considerations:
Moving the current hardware is expensive, even to a location provided for free. It is unknown whether our hardware will survive the move, or if we will be able to recreate the setup at the new location. Furthermore, our current IT contractors who help us manage the facility can only serve the Boston area which means we will need to factor this in when choosing a new location.
Our technical leadership team has considered abandoning our infrastructure and creating a new OSSTest instance, however, as it involves acquiring new hardware there is no guarantee we will be able to set up the new instance successfully.
An option to mitigate this risk could be to deploy on a metal-as-a-service provider, but that would require changes to OSSTest.
As a last resort, abandoning OSSTest has been discussed. The project has no replacement for a test coverage similar to what OSSTest provides. Even if Gitlab CI expanded test coverage similar to what OSSTest currently covers, some features are unlikely to be replicated: automatic bisections on regressions and testing of external projects (Linux kernel, QEMU, OVMF…).
Input from the community
While we work on identifying different options, there are a few steps you can take to stay informed.
Stay Updated: Keep an eye on our official communication channels for real-time updates and information about the status of our services.
Get involved: We appreciate you may have feedback and questions, which is why you are welcome to join the next monthly community call (Thursday 11th July 2024) to provide your input.
Assistance: We welcome members of our community to provide resources or suggest ideas that may help our project.
Looking Ahead
Our goal is to identify and implement measures that will minimize the impact on our project. We will carefully consider the feasibility of solutions, budget constraints, and the long-term needs of the Xen Project.
We apologize for the inconvenience caused by this unexpected announcement and appreciate your understanding and patience as we navigate this challenging situation. Your trust and continued support are invaluable to us, and we remain committed to developing a powerful and reliable open-source hypervisor.
Sincerely,
Kelly
Xen Project Community Manager
Update (12th July): The colo houses OSSTest which is a complex custom tool in Xen Project. After many years, it could be said that some machines are slightly outdated, but at the time of creation, it was very useful and has helped developers work on the code. As a community, we are now looking to shift CI and test efforts to options such as GitLab.