NORDUnet status page

All back to normal, delay caused by an organisation doing a bulk upload af a lot of material.

We are working on resolving the problem.

Problem has been resolved.

Problem has been reported to Kaltura on-prem support.

The transcoding que has been processed and normal operations has been restored.

Problem has been resolved, and transcoding has resumed. There is a backlog in que, that will be transcoded over the next couple of hours.

A permanent fix will be implemented late June.

A work a round has been implemented. We are working with Panopto to establish a fix for the root cause.

Problem is being investigated with Panopto support.

Functionality is restored. Waiting for further information and updates from Zoom.

Progress updates available on https://status.zoom.us

Problem has been reported to Zoom and is being handled.

Problem has been located and resolved.

The re-sync of the indexing servers has been completed. We continue to montor the situation.

At 10 UTC an optimisation of the indexing servers were implemented. Since then, the service has been stable and no issues have been seen. We continue to monitor the situation closely.

The work a round has worked and the service is performing as should be expected. The re-sync is running again and is being monitored closely. While the re-sync process is running there may be sporadic erros encountered regarding activities that causes changes to the service content i.e. uploads, trimming, renaming, publishing and deletion of videos.

The initail re-sync has failed. There continues to be problems. We will be implementing a work around that should of load the process and improve the performance by tomorrow (Thursday) morning.

The Kaltura indexing servers will have to be resynced. During the process, the Kaltura service will be degraded regarding all activities that causes changes to the service i.e. uploads, trimming, renaming, publishing and deletion of videos, will all take longer time to process and will occasionally fail, especially during peak hours. The resync process is expected to take at least another 48 hours to complete.

There are still some issues seen with upload and deletion of videos.

A work-around has been implemented that appears to have improved and stabilised the situation for now.

There continues to be instability regarding the API performance affecting the service.

API servers have been restarted and normal operation has been restored.

Stopping the scripts triggered a problem with the API's that is currently being investigated.

Degraded performance is caused by some additional synchronisation scripts. These will be stopped before 06:00 UTC, if not completed by then, to restore performance

Service has been restored. The cause is being investigated.

Issue has been reported to Zoom and is being investigated.

Work on site #1 completed

Work on site #1 has commenced.

Issue was caused by routing problems between the Zones.

To finally resolve the issue and restore redundancy there will be an EMERGENCY maintenance window tonight Friday 19'th between 20:00 and 21:00 UTC. During the window there will be a short interruption of all services.

Services restored. Currently without firewall redundancy.

Services are in the process of fully restoring.

Manually forcing backup firewall online.

The problem was resolved at 07:44 UTC.

Problem was relating to DNS, and has been resolved.

As a work-a-round you can connect using the ZIP and H.323 gateway load balancer IP's directly; IP's: 109.105.112.235/.236.

The interconnect with TDC has been upgraded to 40Gig.

As a workaround there have been implemented some traffic engineering solutions that will i.e. route traffic to TDC via our IP Transit provider.

Issue also experienced between 12:00 and 12:45 UTC.

We have been working with TDC to establish a 100G interconnect since November 2020, but this is still not in place. Currently we are working with TDC to put some emergency meassures in place, to resolve the issue.

The problem should be fixed.

The issue has be escalated.

The issue has been reported to Zoom.

The changes implemented looks to have resolved the problem.

Various meassures are being deployed to resolve the problem and improve user experience.

Zoom has resolved the problem. https://status.zoom.us/incidents/x51v9gpg8xb6

The issue has been reported to Zoom. The problem is a generic issue not related to on-prem nor to SSO.

A work-a-round for the Epiphan Pearl devices is to downgrade the firmware version to 4.11.0h.

The issue has been escalated with Kaltura support.

The problem has been reported to Kaltura support and is being investigated.

Awaiting upgrade of KMS.

As a work-a-round, affected users can reinstall the old version 4.2.83.

Windows: https://cdnapisec.kaltura.com/content/static/classroom/v4.2.83/KalturaCapture_4.2.83.msi
Mac: https://cdnapisec.kaltura.com/content/static/classroom/v4.2.83/KalturaCapture_4.2.83.dmg

Maintenance sucessfully completed. Total down time 20 minutes.

Problem should be fixed.

This is a global Zoom problem https://status.zoom.us/incidents/wvbq5y0wy3wf

We are investigating and have raised the issue with Zoom.

Que is back to normal.

The +700 files are now processed and the regular backlog from this morning is being processed.

The problem has been resolved we are investigating the cause of the problem.

Following some changes to the setup, things have become stable.

We are investigating with Panopto support.

full backend capacity restored.

No user impact as system load is low,

Updates completed.

Power restored and traffic back on normal route.

Issue was caused by a power outage at a main communication site. Service was affected as trafic between our two datacenters were rerouted. The site is still without power.

Service is restored.

Service window to be extended, till 02:00 UTC.

Fix applied.

No cause was indentified.

Problem has been resolved.

The issue is caused by a problem with the backend storage.

Problem has been resolved.

Service is functioning again. We are monitoring the situation.

The problem has been resolved.

The database completed restoring at 12:55 UTC. Service is operational and functioning again.

The job is now 92% completed, but has slowed down, new extimate for completion is 13 UTC.

New expected restore time is aproximately 11 UTC.

Service should be back up approximately 10:30 UTC.

The Database is still recovering, it is expected to take a few more hours for the recovery to complete.

Database recovery is still ongoing....

Kaltura is down again. Database issue.

All services fully restored.

Problem has been resolved and services are restoring.

Issue is caused by database problems.

As load is building up the system and response times remain low and stable.

Corrections have been implemented, but will have to be verified during real load.

Issue is stille being investigated.

The Moodle integration issue has caused the platform to be unresponsive, and considered to be down. We are working on the isuues.

We are working with Panopto to resolve the problem.

Problem has bee resolved

The migration of the Panopto front-end serveres has been sucessfully completed.

The changes in the Datacenter has improved the the situation. We will continue our action plan and will as a next step migrate all Panopto Front-End servers from the virtualised environment to physical servers.

There continues to be load related issues affecting Panopto. Over this week we will be optimising things in the datacentre infrastructure. In addition later in the week we will deploy new load balancers and add cashing capacity.

After the upgrades done last night, we will monitor the situation.

the load related issues have been resolved.

To night September 30, there will be additional work done that should further improve the situation for both MediaSite and Panopto.

There have been several updates and performance optimisations during the weekend. MediaSite on-demand playback should be much improved even though loading the player itself, could take some seconds during peak hours.

There are still load related issues during peak hours. We are working to resolve the issue.

We are still monitoring the situation but the capacity upgrade and rebalance of all systems seems to have resolved the issue.

Resources allocated to Mediasite and Panopto has been rebalanced. We continue monitoring and will decided on further actions if the issues persist.

More system capacity was added again last night (15/9-2020). We continue to monitor the situation.

More capacity has been added. We continue to monitor the situation, and will be adding more system capacity if the problem persist.

A fix has been implemented and so far the metrics are looking good.

service is back up. We are investigating the cause.

We are working with Zoom to find the root cause.

Maintenance sucessfully completed

Things remain stable.

The issue was caused by a problem in the Datacenter and the service has been restored.

Maintenance completed.

Capacity was added as planed.

Zoom AWS is back to normal operations.

We are monitoring, and waiting for feedback from Zoom on the cause of the outage.

Zoom AWS is back up.

Organisations that migrated to on-prem, can migrate back to licensed.

The service is fully restored.

The Zoom AWS cluster is back up, and working, but some users will still get various error messages.

Users with an urgent need to host meetings can migrate to on-prem by having an administrator change setting from from licensed to on-prem.

Maintenance activities sucessfullt completed

The resource rebalancing was completed and the systems are back in operation.

Analytics has stopped working again.

We will continue to monitor the situation over the coming week.

A fix has been found and implemented.

We will closely monitor the situation during the day.

No cause for the slow response experinced has been identifyed.

Problem has been resolved for all accounts.

The data is starting to become available again.

The problem relating to the storage infrastructure, has been identified and fixed. Normal operations will be fully restored within 1 hour.

We are currently investigating.

Problem has been resolved. The issue was caused by a packaged deployed with inconsistant module versions.

Problem is being investigated.

Backlog is still being processed. Normal operations are expected at 14:30 UTC.

Back to normal opertions.

Problem has been identified and rectified. Backlog is still being processed.

Issue is being investigated.

We will keep monitoring the situation.

The issue has been resolved. Waiting for more information regarding the cause of the outage.

Zoom have identified the issue causing users to be unable to authenticate to the Zoom website (zoom.us) and unable to start and join Zoom Meetings and Webinars, and are working on a fix for this issue.

The issues is being investigated with the highest priority.

Maintenance sucessfully completed.

This issue has been resolved

The problem is currently being investigated by Kaltura.

The issus have been resolved

The problem with thumbnails affects all Kaltura deployments, not only the NORDUnet instance.

The Zoom AWS cluster has been stable for 24 hours, and users can migrate back to the AWS cluster by changing setting to licensed.

During the incident more than 50% of accounts have been moved to run on-prem. We will be monitoring the situation for the next 24 hours before taking any further action. We are waiting for a more detailed RFO from Zoom.

The issue has been resolved and service for the customers running licensed has been restored.

We are working with Zoom to find the cause of the problem with the AWS cluster.

As a temporare meassure, to get things to work, it is required to change the setting from cloud to on-prem.

We are investigating and looking at mitigation options.

Kaltura is currently investigating possible solutions to this problem.

Service restored investigating, failover issue.

Migration was completed as planned without any issues.

The Netwok problem has been resolved.

The cause of the problem is being invetigated.

We are working with Panopto to find a solution.

The problem has been resolved.

Work is still ongoing to establish a solution.

The Kaltura database re-sync has completed and everything is back to normal operation.

The Kaltura database re-sync will not be completed until aproximately 14 UTC.

The problem has been resolved. The Kaltura database is currently re-syncing and Kaltura can therefore not accept new video material. When the re-sync is completed transcoding will resume. The Kaltura service is expected to be fully operational at 11 UTC.

The problem is Intermediate packet loss. This is specially affecting Kaltura due to recovery times.

We have received information from Zoom that even if the Datacentre is labled as AM the meetings are actually hosted in the AWS cluster in Stockholm. The Datacentre will be relabled to show SK.

Still being worked on

We are working with Zoom to migrate licensed accounts back to the Stockholm datacentre.

No further issues have been seen or reported.

The problem was caused by a backend storage job moving large amount of data and causing poor performance for the Kaltura API servers communicating with the storage backend. The backend job has completed, but we continue to monitor the situation.

The issue is caused by communication problems between the Kaltura servers and the storage system.

We have received a confirmation that Zoom will not be deploying a EU based connector and not be providing the local virtual connector for Skype for business. The reasoning is that SFB is coming to the end of life for the product and Zoom will not be providing any more updates to the virtual connector if something were to break and Zoom want to prevent new customers from jumping on the software that is not going to be supported. Zoom is working with the documentation team to remove the option from the admin profile.

The recommended workaround is to install the Zoom client and use the plugin https://support.zoom.us/hc/en-us/articles/204481169-Zoom-Skype-for-Business-Lync-Plugin.

If it is not possible to use the Zoom client, please refer to the other workaround options.

As a temporare meassure it is possible to use either SIP when connecting from Skype for business to a Zoom meeting, or use the clientless direct browser access.

The root case was that due to the network problems causing firwall flaps, the Kaltura master and slave datatbase got out of sync. The system was unable to accept new data until the databases were re-sync'd and database consistancy was restored.

The issue has been resolved, but there is a large backlog of videos that need to be transcoded.

The issue is still being worked on and is also affecting the performance of other parts of the Kaltura service.

The issue was resolved by reloading the software on a core router component.

The problem has been resolved and all users previously running with Copenhagen as the primary location have been migrated back.

As a workaround users are being migrated to Stockholm.

upgrade completed.

The problem has been resolved. A Reason For Outage (RF) report will be made available.

The problem has been located to a routing problem for the Storage network.

The backlog has been processed and upload times are back to normal.

Problem was fixed on April 5, at 04:00.

Waiting for information from Zoom. Next update Monday April 6, at 08:00 UTC

We are investigating the issue to identify the cause. Next update at 10:00 UTC

The issue has been fixed.

Waiting for Kaltura support. Next update, March 30, at 09 UTC.

There is currently a problem with livestreaming on the Kaltura platform. The issue is causing some live streams not being able to start. The issue has been escalated to Kaltura support.

There is currently a problem with livestreaming on the Kaltura platform. The issue is being investigated. Next update at 08:00 UTC.

The problem has been resolved.

The issue is being investigated with Kaltura on-premise engineering. Next update at 12:00 CET.

Additional API server added.

The issue is being investigated with Kaltura on-premise engineering.

Additional API servers were added at 03:00 UTC

The issue is being investigated with Kaltura on-premise engineering. Work is ongoing to add more system resources

The issue has been resolved.

The issue is being investigated with Kaltura on-premise engineering, to establish what can be done to improve system performance.

The issue has been resolved but users will experience a degraded performance as the upload backlog is being processed.

All systems are comming back online, and returning to normal.

During maintenance we experienced a large outage on ndn services running on our virtualisation setup. The cause was human error, but we are looking into how/why it ended up affecting most of our systems.

We have found the cause and are working with Zoom to see if we can find a work around.
For now please change users to Licensed instead of On-Premise as the system hits low logical limits
and then block new meeting creation.

The issue has now been mitigated, we continue to monitor the situation. Official zoom status

We are currently having errors creating new meetings on the Zoom service for both Cloud and On-Premise. We are working with Zoom to resolve this issue.

The issue has now been mitigated, we continue to monitor the situation.

We are currently having errors creating new meetings on the Zoom service. We are working with Zoom to resolve this issue.