ETH-Services Status

Upstream confirmed the issue and is preparing a hotfix.

Upstream is checking for logic errors on backup creation.

Enduser capability to create new backups has been restored, but may still be impacted from inconsistency. The recommendation to create backups from within VPS itself remains in place.

Enduser capability to restore backups remain disabled. Please check with support in case a restore is required.

We are currently observing a faulty patch from virtualization upstreams on the backup logic which falsely interprets an exit code from a sanity check, leading to potential inconsistency of created backups. From current log validation, only a small subset of VMs are impacted by this.

A ticket with the upstream has been opened and we are awaiting feedback.

For the moment, end-user capability to create and restore backups have been disabled. Scheduled weekly backups continue to run normal, but may be impacted from the same bug. We suggest users create their own backup from within the systems to be on the safe side.

In case a restore of such automatic backup is needed, please open a support ticket and we will validate the consistency before initiating a restore.

The incident is resolved.

Despite RETNs assuring that the issue should be resolved, today some POPs experienced issues again.

We will keep RETN drained for now and check in a few days if their network has stabilised to such degree that we can consider them a stable network partner again.

No customer impact is expected.

Situation seem stable for now and RETN has ben re-enabled. We are still waiting for RFO.

According to RETN the problem should be permantetly fixed. However, we will not re-enable the upstream for the next few days and instead monitor the situation.

Carrier RETN is facing another outage. Sessions have been drained. No customer impact is expected.

Confirmation from the carrier has been received that situation is stable again and the upstream has been re-enabled. We are waiting on further information regarding RFO.

The situation seems currently stable again, however, since RETN is not of critical importance for our network, the upstream remains disabled for now.

We are currently observing an outage of upstream RETN/AS9002 in DRT.

Traffic was automatically shifted towards other carriers and sessions have been administratively disabled until feedback from carrier is received. No customer impacts is expected.

XC switchover went without any problems. AS33891 has been decommissioned and AS5405 is now live.

We are going to decommission upstream Core-Backbone in February 2025. They will be replaced with Inter.Link / AS5405, another premium T2 Carrier with similar performance.

Due to precabling situation, the new carrier will not be deployed before CBB has been fully decommissioned. During this time frame customers at AS3320 might experience degraded performance.

The issue has been resolved.

Update 1: a first hotfix was deployed which enables supports ability to create backups again. However, enduser functionality is not yet recovered. In case a new backup is needed, please open a ticket and a backup will be created accordingly.

Due to an faulty patch from virtualization upstream, new backups can currently not be created. We are in contact with the vendor and a fix is already completed but not yet considered stable for production. We will apply the fix ASAP once confirmation is received.