All systems are operational.

The incident has been resolved.

Upstream confirmed the issue and is preparing a hotfix.

Upstream is checking for logic errors on backup creation.

Enduser capability to create new backups has been restored, but may still be impacted from inconsistency. The recommendation to create backups from within VPS itself remains in place.

Enduser capability to restore backups remain disabled. Please check with support in case a restore is required.

We are currently observing a faulty patch from virtualization upstreams on the backup logic which falsely interprets an exit code from a sanity check, leading to potential inconsistency of created backups. From current log validation, only a small subset of VMs are impacted by this.

A ticket with the upstream has been opened and we are awaiting feedback.

For the moment, end-user capability to create and restore backups have been disabled. Scheduled weekly backups continue to run normal, but may be impacted from the same bug. We suggest users create their own backup from within the systems to be on the safe side.

In case a restore of such automatic backup is needed, please open a support ticket and we will validate the consistency before initiating a restore.

The incident is resolved.

Despite RETNs assuring that the issue should be resolved, today some POPs experienced issues again.

We will keep RETN drained for now and check in a few days if their network has stabilised to such degree that we can consider them a stable network partner again.

No customer impact is expected.

Situation seem stable for now and RETN has ben re-enabled. We are still waiting for RFO.

According to RETN the problem should be permantetly fixed. However, we will not re-enable the upstream for the next few days and instead monitor the situation.

Carrier RETN is facing another outage. Sessions have been drained. No customer impact is expected.

Confirmation from the carrier has been received that situation is stable again and the upstream has been re-enabled. We are waiting on further information regarding RFO.

The situation seems currently stable again, however, since RETN is not of critical importance for our network, the upstream remains disabled for now.

We are currently observing an outage of upstream RETN/AS9002 in DRT.

Traffic was automatically shifted towards other carriers and sessions have been administratively disabled until feedback from carrier is received. No customer impacts is expected.