NORDUnet status page

Incident feeds: RSS JSON Maintenance RSS

All Systems Operational













Upcomming maintenance

MediaSite version upgrade

2020-10-21 11:07 UTC

On Wednesday October 28, between 19-21 UTC, we will be upgrading MediaSite to the latest version. There wil be downtime during the service window.


Kaltura transcoding flavours

It is now again possible to choose which transcoding flavours to enable. But before you go ahead and enable all flavours please read this post about transcoding settings , as the number of flavours will impact your storage consumption, and thereby the cost for the service.

KMS go for iOS

Due to a recent update to the KMS Go for iOS, it is not possible to login to the NORDUnet hosted MediaSpaces. Kaltura is currently investigating possible solutions to this problem.

2020-07-24 06:41 UTC

Zoom Info

Zoom concurrent meeting participants
More Zoom statistics

Zoom has located all information about the latest client versions and Release notes on a single page

NORDUnet Zoom: GDPR and Privacy Facts

A helpful article from zoom about error codes

Zoom for education FAQ

2020-10-28 10:43 UTC

Kaltura service issue (Operational)

Closed -
2020-10-28 11:29 UTC
Resolved -

Fix applied.

2020-10-28 10:52 UTC

2020-10-26 11:17 UTC

Panopto issue (Operational)

Resolved -
2020-10-27 11:12 UTC
Monitoring -

No cause was indentified.

2020-10-26 13:34 UTC

2020-10-25 13:06 UTC

Kaltura service issue (Outage)

Closing -
2020-10-26 09:02 UTC
Resolved -

Problem has been resolved.

2020-10-25 16:22 UTC
Monitoring -
2020-10-25 16:22 UTC
Update -

The issue is caused by a problem with the backend storage.

2020-10-25 14:51 UTC

2020-10-23 07:56 UTC

Kaltura play list problems (Operational)

Resolved -

Problem has been resolved.

2020-10-23 09:47 UTC

2020-10-23 05:29 UTC

Kaltura service issue (Degraded)

Reolved -
2020-10-23 07:58 UTC
Update -

Service is functioning again. We are monitoring the situation.

2020-10-23 07:24 UTC

2020-10-22 14:50 UTC

Problem with Kaltura live streams (Degraded)

Resolved -

The problem has been resolved.

2020-10-22 19:24 UTC

2020-10-21 18:40 UTC

Kaltura service issue (Outage)

Update -

The database completed restoring at 12:55 UTC. Service is operational and functioning again.

2020-10-22 12:57 UTC
Update -

The job is now 92% completed, but has slowed down, new extimate for completion is 13 UTC.

2020-10-22 11:00 UTC
Update -

New expected restore time is aproximately 11 UTC.

2020-10-22 10:20 UTC
Update -

Service should be back up approximately 10:30 UTC.

2020-10-22 08:47 UTC
Update -

The Database is still recovering, it is expected to take a few more hours for the recovery to complete.

2020-10-22 06:10 UTC
Update -

Database recovery is still ongoing....

2020-10-22 04:40 UTC
Update -

Kaltura is down again. Database issue.

2020-10-22 00:35 UTC
Resolved -

All services fully restored.

2020-10-21 20:09 UTC
Update -

Problem has been resolved and services are restoring.

2020-10-21 19:57 UTC
Update -

Issue is caused by database problems.

2020-10-21 18:58 UTC

2020-10-19 08:25 UTC

Panopto issue (Outage)

Resolved -
2020-10-20 13:39 UTC
Monitoring -

As load is building up the system and response times remain low and stable.

2020-10-20 07:10 UTC
Update -

Corrections have been implemented, but will have to be verified during real load.

2020-10-19 20:04 UTC
Update -

Issue is stille being investigated.

2020-10-19 13:27 UTC
update -

The Moodle integration issue has caused the platform to be unresponsive, and considered to be down. We are working on the isuues.

2020-10-19 11:11 UTC
Update -

We are working with Panopto to resolve the problem.

2020-10-19 09:13 UTC

2020-10-20 06:43 UTC

Kaltura ingest problem (Operational)

Resloved -

Problem has bee resolved

2020-10-20 10:38 UTC

2020-09-14 11:12 UTC

Slow response on the Panopto and MediaSite services (Operational)

Monitoring -

The migration of the Panopto front-end serveres has been sucessfully completed.

2020-10-14 11:18 UTC
Update -

The changes in the Datacenter has improved the the situation. We will continue our action plan and will as a next step migrate all Panopto Front-End servers from the virtualised environment to physical servers.

2020-10-07 13:09 UTC
update -

There continues to be load related issues affecting Panopto. Over this week we will be optimising things in the datacentre infrastructure. In addition later in the week we will deploy new load balancers and add cashing capacity.

2020-10-06 06:26 UTC
Monitoring -

After the upgrades done last night, we will monitor the situation.

2020-10-01 10:06 UTC
resolved -

the load related issues have been resolved.

2020-09-30 10:19 UTC
update -

To night September 30, there will be additional work done that should further improve the situation for both MediaSite and Panopto.

2020-09-30 10:19 UTC
Update -

There have been several updates and performance optimisations during the weekend. MediaSite on-demand playback should be much improved even though loading the player itself, could take some seconds during peak hours.

2020-09-28 06:43 UTC
Update -

There are still load related issues during peak hours. We are working to resolve the issue.

2020-09-21 09:19 UTC
Monitoring -

We are still monitoring the situation but the capacity upgrade and rebalance of all systems seems to have resolved the issue.

2020-09-18 15:00 UTC
Monitoring -

Resources allocated to Mediasite and Panopto has been rebalanced. We continue monitoring and will decided on further actions if the issues persist.

2020-09-17 05:14 UTC
Monitoring -

More system capacity was added again last night (15/9-2020). We continue to monitor the situation.

2020-09-16 06:16 UTC
Monitoring -

More capacity has been added. We continue to monitor the situation, and will be adding more system capacity if the problem persist.

2020-09-15 07:04 UTC

2020-10-13 08:20 UTC

MeiaSite problem (Operational)

Monitoring -

A fix has been implemented and so far the metrics are looking good.

2020-10-14 18:58 UTC

2020-10-18 18:00 UTC

The NORDUnet Kaltura service is unavailable (Operational)

Update -

service is back up. We are investigating the cause.

2020-10-18 18:38 UTC

2020-10-15 06:39 UTC

Zoom on-prem issue (Operational)

Closed -
2020-10-18 18:25 UTC
Resolved -

We are working with Zoom to find the root cause.

2020-10-15 07:28 UTC

2020-10-14 13:34 UTC

Maintenance on backend storage system (Degraded)

Closed -

Maintenance sucessfully completed

2020-10-14 18:57 UTC

2020-10-12 20:46 UTC

Kaltura service issues (Outage)

Closing -

Things remain stable.

2020-10-13 07:43 UTC
Monitoring -
2020-10-12 22:07 UTC
Resloved -

The issue was caused by a problem in the Datacenter and the service has been restored.

2020-10-12 22:05 UTC

2020-10-05 11:25 UTC

Changes to Datacenter infrastuture (Operational)

Closed -

Maintenance completed.

2020-10-07 06:26 UTC

2020-10-05 06:45 UTC

Kaltura - Live streaming capacity upgrade (Operational)

Closed -

Capacity was added as planed.

2020-10-06 10:39 UTC

2020-09-29 17:03 UTC

MedisSite planned work (Operational)

2020-09-28 06:49 UTC

Zoom AWS Stockholm down (Outage)

Resloved -

Zoom AWS is back to normal operations.

2020-09-28 17:15 UTC
Monitoring -

We are monitoring, and waiting for feedback from Zoom on the cause of the outage.

2020-09-28 08:18 UTC
update -

Zoom AWS is back up.

2020-09-28 08:17 UTC

2020-09-26 09:08 UTC

Problem with Zoom AWS (Operational)

Monitoring -

Organisations that migrated to on-prem, can migrate back to licensed.

2020-09-28 06:41 UTC
Monitoring -

The service is fully restored.

2020-09-27 18:50 UTC
Update -

The Zoom AWS cluster is back up, and working, but some users will still get various error messages.

2020-09-27 06:14 UTC
Update -

Users with an urgent need to host meetings can migrate to on-prem by having an administrator change setting from from licensed to on-prem.

2020-09-26 09:11 UTC

2020-09-24 10:00 UTC

Site and Datacenter maintenance (Degraded)

completed -

Maintenance activities sucessfullt completed

2020-09-25 05:03 UTC

2020-09-16 21:00 UTC

Downtime of Mediasite and Panopto (Operational)

Maintenance completed -

The resource rebalancing was completed and the systems are back in operation.

2020-09-16 21:46 UTC

2020-08-11 09:28 UTC

Kaltura KMC Analytics (Degraded)

Investigating -

Analytics has stopped working again.

2020-09-14 12:21 UTC
Monitoring -

We will continue to monitor the situation over the coming week.

2020-09-11 12:06 UTC
Resolved -

A fix has been found and implemented.

2020-09-11 12:05 UTC

2020-09-08 10:04 UTC

Slow response in the Panoto service (Degraded)

Closed -
2020-09-09 10:01 UTC
Monitoring -

We will closely monitor the situation during the day.

2020-09-09 06:17 UTC
Update -

No cause for the slow response experinced has been identifyed.

2020-09-09 06:16 UTC

2020-08-28 05:15 UTC

Zoom on-prem dashboard issues (Operational)

Closed -
2020-09-09 10:02 UTC
Resolved -

Problem has been resolved for all accounts.

2020-09-04 07:33 UTC
Update -

The data is starting to become available again.

2020-09-01 13:39 UTC

2020-08-28 10:44 UTC

Performance problems with the Kaltura and Panopto service (Degraded)

Resolved -

The problem relating to the storage infrastructure, has been identified and fixed. Normal operations will be fully restored within 1 hour.

2020-08-28 11:25 UTC
Investigating -

We are currently investigating.

2020-08-28 10:46 UTC

2020-08-17 08:02 UTC

Problem with Kaltura live streams (Degraded)

Resolved -

Problem has been resolved. The issue was caused by a packaged deployed with inconsistant module versions.

2020-08-27 10:45 UTC
Investigating -

Problem is being investigated.

2020-08-27 08:04 UTC

2020-08-26 07:30 UTC

Kaltura - Issues with transcoding (Degraded)

Update -

Backlog is still being processed. Normal operations are expected at 14:30 UTC.

2020-09-26 13:40 UTC
Resolved -

Back to normal opertions.

2020-08-26 14:34 UTC
Update -

Problem has been identified and rectified. Backlog is still being processed.

2020-08-26 09:38 UTC
Investigating -

Issue is being investigated.

2020-08-26 07:31 UTC

2020-08-24 12:55 UTC

Problems logging in to the website and joing meetings from the web browser (Operational)

Monitoring -

We will keep monitoring the situation.

2020-08-24 18:42 UTC
Resolved -

The issue has been resolved. Waiting for more information regarding the cause of the outage.

2020-08-24 18:20 UTC
Update -

Zoom have identified the issue causing users to be unable to authenticate to the Zoom website ( and unable to start and join Zoom Meetings and Webinars, and are working on a fix for this issue.

2020-08-24 14:13 UTC
Investigating -

The issues is being investigated with the highest priority.

2020-08-24 13:07 UTC

2020-08-20 21:00 UTC

Planned maintenance affecting Zoom on-prem users (Outage)

Closed -

Maintenance sucessfully completed.

2020-08-21 06:44 UTC

2020-08-11 10:15 UTC

Embed codes in Kaltura (Operational)

Resolved -

This issue has been resolved

2020-08-12 09:01 UTC
Investigating -

The problem is currently being investigated by Kaltura.

2020-08-11 10:15 UTC

2020-08-11 09:27 UTC

Thumbnails in Kaltura (Operational)

Resolved -

The issus have been resolved

2020-08-11 15:59 UTC
Investigating -

The problem with thumbnails affects all Kaltura deployments, not only the NORDUnet instance.

2020-08-11 11:01 UTC

2020-07-22 07:04 UTC

Zoom users running licensed are experiencing problems (Degraded)

Resolved -

The Zoom AWS cluster has been stable for 24 hours, and users can migrate back to the AWS cluster by changing setting to licensed.

2020-07-23 13:12 UTC
Monitoring -

During the incident more than 50% of accounts have been moved to run on-prem. We will be monitoring the situation for the next 24 hours before taking any further action. We are waiting for a more detailed RFO from Zoom.

2020-07-22 14:12 UTC
Resolved -

The issue has been resolved and service for the customers running licensed has been restored.

2020-07-22 14:09 UTC
Investigating -

We are working with Zoom to find the cause of the problem with the AWS cluster.

2020-07-22 08:24 UTC
Work-a-round -

As a temporare meassure, to get things to work, it is required to change the setting from cloud to on-prem.

2020-07-22 07:13 UTC
Investigating -

We are investigating and looking at mitigation options.

2020-07-22 07:06 UTC

2020-06-12 11:19 UTC

Kaltura - KMS Go for iOS is not able to login to the NORDUnet hosted MediaSpaces (Degraded)

Investigating -

Kaltura is currently investigating possible solutions to this problem.

2020-06-12 11:21 UTC

2020-07-23 20:00 UTC

Planned maintenance affecting Zoom on-prem users (Degraded)

2020-07-16 19:00 UTC

The Zoom service is affected by maintenance on a core router (Degraded)

Monitoring -

Service restored investigating, failover issue.

2020-07-16 19:24 UTC

2020-06-24 15:40 UTC

Panopto backend storage migration (Maintenance)

Closed -

Migration was completed as planned without any issues.

2020-06-30 06:55 UTC

2020-06-16 09:59 UTC

Kaltura - there is currently a network problem cauing longer video load and upload times (Degraded)

Resloved -

The Netwok problem has been resolved.

2020-06-17 10:12 UTC
Investigating -

The cause of the problem is being invetigated.

2020-06-16 12:15 UTC

2020-06-09 06:35 UTC

Panopto index search not working (Degraded)

Update -

We are working with Panopto to find a solution.

2020-09-09 13:37 UTC
Resolved -

The problem has been resolved.

2020-06-11 13:27 UTC
Update -

Work is still ongoing to establish a solution.

2020-06-10 13:38 UTC

2020-06-08 15:00 UTC

Planned maintenance - Panopto service upgrade - June 8, 15-22 UTC (Maintenance)

2020-05-27 10:58 UTC

Zoom Datacentre labeling (Operational)

2020-06-04 14:15 UTC

Issues with infrastructure affecting multiple services (Degraded)

Resolved -

The Kaltura database re-sync has completed and everything is back to normal operation.

2020-06-05 14:29 UTC
Update -

The Kaltura database re-sync will not be completed until aproximately 14 UTC.

2020-06-05 09:56 UTC
Monitoring -

The problem has been resolved. The Kaltura database is currently re-syncing and Kaltura can therefore not accept new video material. When the re-sync is completed transcoding will resume. The Kaltura service is expected to be fully operational at 11 UTC.

2020-06-05 09:02 UTC
Investigating -

The problem is Intermediate packet loss. This is specially affecting Kaltura due to recovery times.

2020-06-05 07:10 UTC

2020-05-17 07:24 UTC

NORDUnet Zoom platform - Licensed meetings hosted in AM datacentre (Operational)

Update -

We have received information from Zoom that even if the Datacentre is labled as AM the meetings are actually hosted in the AWS cluster in Stockholm. The Datacentre will be relabled to show SK.

2020-05-19 08:38 UTC
Update -

Still being worked on

2020-05-18 07:28 UTC
Uodate -

We are working with Zoom to migrate licensed accounts back to the Stockholm datacentre.

2020-05-17 18:00 UTC

2020-05-20 16:00 UTC

Planned maintenance - Kaltura service - Wednesday 20/5, 16-19 UTC. (Maintenance)

2020-05-18 06:11 UTC

Issues with Kaltura (Degraded)

Resolved -

No further issues have been seen or reported.

2020-05-20 09:04 UTC
Monitor -

The problem was caused by a backend storage job moving large amount of data and causing poor performance for the Kaltura API servers communicating with the storage backend. The backend job has completed, but we continue to monitor the situation.

2020-05-19 21:44 UTC
Update -

The issue is caused by communication problems between the Kaltura servers and the storage system.

2020-05-18 10:39 UTC

2020-04-22 14:46 UTC

Zoom skype for business connector traffic routed via US (Operational)

Update -

We have received a confirmation that Zoom will not be deploying a EU based connector and not be providing the local virtual connector for Skype for business. The reasoning is that SFB is coming to the end of life for the product and Zoom will not be providing any more updates to the virtual connector if something were to break and Zoom want to prevent new customers from jumping on the software that is not going to be supported. Zoom is working with the documentation team to remove the option from the admin profile.

2020-05-18 16:41 UTC
Recommended workaround -

The recommended workaround is to install the Zoom client and use the plugin

If it is not possible to use the Zoom client, please refer to the other workaround options.

2020-05-12 15:10 UTC
Workaround -

As a temporare meassure it is possible to use either SIP when connecting from Skype for business to a Zoom meeting, or use the clientless direct browser access.

2020-04-22 14:52 UTC

2020-05-15 09:03 UTC

Issues with Kaltura transcoding (Degraded)

Root cause -

The root case was that due to the network problems causing firwall flaps, the Kaltura master and slave datatbase got out of sync. The system was unable to accept new data until the databases were re-sync'd and database consistancy was restored.

2020-05-16 09:24 UTC
Resolved -

The issue has been resolved, but there is a large backlog of videos that need to be transcoded.

2020-05-15 14:46 UTC
Update -

The issue is still being worked on and is also affecting the performance of other parts of the Kaltura service.

2020-05-15 11:25 UTC

2020-05-15 01:03 UTC

Problems on one NORDUnet core router in Denmark is causing packet loss affecting all services (Degraded)

Resolved -

The issue was resolved by reloading the software on a core router component.

2020-05-15 10:08 UTC

2020-05-14 05:42 UTC

Zoom instability (Degraded)

2020-05-13 19:39 UTC

There is currently an issue with NORDUnet Zoom on-prem (Degraded)

2020-05-02 05:00 UTC

Zoom on-premise in Copenhagen down due to failed software update (Outage)

Fixed -

The problem has been resolved and all users previously running with Copenhagen as the primary location have been migrated back.

2020-05-02 18:46 UTC
Workaround in place -
2020-05-02 16:25 UTC
Workaround -

As a workaround users are being migrated to Stockholm.

2020-05-02 16:00 UTC

2020-04-21 12:01 UTC

Firewall software version upgrade (Maintenance)

completed -

upgrade completed.

2020-04-22 04:54 UTC

2020-04-18 10:00 UTC

Firewall capacity upgrade (Maintenance)

2020-04-20 05:00 UTC

Kaltura Outage (Outage)

Resolved -

The problem has been resolved. A Reason For Outage (RF) report will be made available.

2020-04-20 06:32 UTC
Update -

The problem has been located to a routing problem for the Storage network.

2020-04-20 05:58 UTC

2020-04-14 07:46 UTC

Kaltura upload (Degraded)

Resolved -

The backlog has been processed and upload times are back to normal.

2020-04-14 17:05 UTC

2020-03-31 09:37 UTC

Kaltura Storage system (Maintenance)

Maintenance successfully completed. -
2020-04-11 21:44 UTC

2020-04-03 08:00 UTC


Resolved -

Problem was fixed on April 5, at 04:00.

2020-04-06 06:55 UTC
Latest Update -

Waiting for information from Zoom. Next update Monday April 6, at 08:00 UTC

2020-04-03 10:00 UTC
Investigating -

We are investigating the issue to identify the cause. Next update at 10:00 UTC

2020-04-03 08:14 UTC

2020-04-05 11:00 UTC

Datacentre re-cabling (Maintenance)

2020-03-31 14:16 UTC

Datacenter cabling (Maintenance)

2020-03-26 06:10 UTC

NORDUnet Kaltura Service (Degraded)

Closed -
2020-03-30 08:41 UTC
Resolved -

The issue has been fixed.

2020-03-30 05:49 UTC
Update -

Waiting for Kaltura support. Next update, March 30, at 09 UTC.

2020-03-27 14:58 UTC
Update -

There is currently a problem with livestreaming on the Kaltura platform. The issue is causing some live streams not being able to start. The issue has been escalated to Kaltura support.

2020-03-26 07:55 UTC
Investigating -

There is currently a problem with livestreaming on the Kaltura platform. The issue is being investigated. Next update at 08:00 UTC.

2020-03-26 07:12 UTC

2020-03-23 09:15 UTC

NORDUnet Kaltura Service. (Degraded)

Resolved -

The problem has been resolved.

2020-03-20 11:45 UTC
Investigating -

The issue is being investigated with Kaltura on-premise engineering. Next update at 12:00 CET.

2020-03-20 09:25 UTC

2020-03-20 09:25 UTC

NORDUnet Kaltura Service. (Degraded)

Resolved -

Additional API server added.

2020-03-20 15:00 UTC
Investigating -

The issue is being investigated with Kaltura on-premise engineering.

2020-03-20 09:25 UTC

2020-03-19 08:25 UTC

NORDUnet Kaltura Service. (Degraded)

Update -

Additional API servers were added at 03:00 UTC

2020-03-20 03:25 UTC
Investigating -

The issue is being investigated with Kaltura on-premise engineering. Work is ongoing to add more system resources

2020-03-19 08:25 UTC

2020-03-18 14:00 UTC

NORDUnet Kaltura Service. (Degraded)

Resolved -

The issue has been resolved.

2020-03-18 16:00 UTC
Investigating -

The issue is being investigated with Kaltura on-premise engineering, to establish what can be done to improve system performance.

2020-03-18 14:00 UTC

2020-03-17 20:20 UTC

The NORDUnet Panopto on-premise service is currently experiencing problems that prevent videos from being uploaded. (Critical)

Resolved -

The issue has been resolved but users will experience a degraded performance as the upload backlog is being processed.

2020-03-18 09:02 UTC

2020-03-16 19:20 UTC

Most services on our virtualization platform (Outage)

Resolved -

All systems are comming back online, and returning to normal.

2020-03-16 20:08 UTC
Investigating -

During maintenance we experienced a large outage on ndn services running on our virtualisation setup. The cause was human error, but we are looking into how/why it ended up affecting most of our systems.

2020-03-16 19:20 UTC

2020-03-16 08:30 UTC

Creating zoom meetings (Degraded)

Update -

We have found the cause and are working with Zoom to see if we can find a work around.
For now please change users to Licensed instead of On-Premise as the system hits low logical limits
and then block new meeting creation.

2020-03-16 09:50 UTC

2020-03-13 09:30 UTC

Creating zoom meetings (Degraded)

Monitoring -

The issue has now been mitigated, we continue to monitor the situation. Official zoom status

2020-03-11 15:10 UTC
Creating zoom meetings -

We are currently having errors creating new meetings on the Zoom service for both Cloud and On-Premise. We are working with Zoom to resolve this issue.

2020-03-11 9:30 UTC

2020-03-12 10:00 UTC

Creating zoom meetings (Degraded)

Monitoring -

The issue has now been mitigated, we continue to monitor the situation.

2020-03-12 14:30 UTC
Investigating -

We are currently having errors creating new meetings on the Zoom service. We are working with Zoom to resolve this issue.

2020-03-12 10:00 UTC