2021-02-15 09:11 UTC
During the weekends 20/21 and 27/28 of February and 13/14 of March 2021, the storage backend network will be upgraded. There will be loos of redundancy during the upgrades, but no service impact is expected.
Incident feeds: RSS JSON Maintenance RSS
Network
Operational
Zoom
Operational
Kaltura
Operational
MediaSite
Operational
Panopto
Operational
Metadata
Operational
2021-02-15 09:11 UTC
During the weekends 20/21 and 27/28 of February and 13/14 of March 2021, the storage backend network will be upgraded. There will be loos of redundancy during the upgrades, but no service impact is expected.2021-02-01 10:50 UTC
In the period starting March 26'th at 18:00 UTC till March 28'th at 16:00 UTC, the Kaltura service will be upgraded to the latest software version. (Regional Cloud Release 2021 Q1)Additional statistics for external monitoring of services web reachability and response times can be seen on Pingdom
It is now again possible to choose which transcoding flavours to enable. But before you go ahead and enable all flavours please read this post about transcoding settings , as the number of flavours will impact your storage consumption, and thereby the cost for the service.
Due to a recent update to the KMS Go for iOS, it is not possible to login to the NORDUnet hosted MediaSpaces. Kaltura is currently investigating possible solutions to this problem.
Important info regarding: LTI Pro Data Storage and Processing
Zoom has located all information about the latest client versions and Release notes on a single page
NORDUnet Zoom: GDPR and Privacy Facts
A helpful article from zoom about error codes
Issue was caused by routing problems between the Zones.
To finally resolve the issue and restore redundancy there will be an EMERGENCY maintenance window tonight Friday 19'th between 20:00 and 21:00 UTC. During the window there will be a short interruption of all services.
Services restored. Currently without firewall redundancy.
Services are in the process of fully restoring.
Manually forcing backup firewall online.
The problem was resolved at 07:44 UTC.
Problem was relating to DNS, and has been resolved.
As a work-a-round you can connect using the ZIP and H.323 gateway load balancer IP's directly; IP's: 109.105.112.235/.236.
The interconnect with TDC has been upgraded to 40Gig.
As a workaround there have been implemented some traffic engineering solutions that will i.e. route traffic to TDC via our IP Transit provider.
Issue also experienced between 12:00 and 12:45 UTC.
We have been working with TDC to establish a 100G interconnect since November 2020, but this is still not in place.
The problem should be fixed.
The issue has be escalated.
The issue has been reported to Zoom.
The changes implemented looks to have resolved the problem.
Various meassures are being deployed to resolve the problem and improve user experience.
Zoom has resolved the problem. https://status.zoom.us/incidents/x51v9gpg8xb6
The issue has been reported to Zoom. The problem is a generic issue not related to on-prem nor to SSO.
A work-a-round for the Epiphan Pearl devices is to downgrade the firmware version to 4.11.0h.
The issue has been escalated with Kaltura support.
The problem has been reported to Kaltura support and is being investigated.
Awaiting upgrade of KMS.
As a work-a-round, affected users can reinstall the old version 4.2.83.
Windows: https://cdnapisec.kaltura.com/content/static/classroom/v4.2.83/KalturaCapture_4.2.83.msi
Mac: https://cdnapisec.kaltura.com/content/static/classroom/v4.2.83/KalturaCapture_4.2.83.dmg
Maintenance sucessfully completed. Total down time 20 minutes.
Problem should be fixed.
This is a global Zoom problem https://status.zoom.us/incidents/wvbq5y0wy3wf
We are investigating and have raised the issue with Zoom.
Que is back to normal.
The +700 files are now processed and the regular backlog from this morning is being processed.
The problem has been resolved we are investigating the cause of the problem.
Following some changes to the setup, things have become stable.
We are investigating with Panopto support.
full backend capacity restored.
No user impact as system load is low,
Updates completed.
Power restored and traffic back on normal route.
Issue was caused by a power outage at a main communication site. Service was affected as trafic between our two datacenters were rerouted. The site is still without power.
Service is restored.
Service window to be extended, till 02:00 UTC.
Fix applied.
No cause was indentified.
Problem has been resolved.
The issue is caused by a problem with the backend storage.
Problem has been resolved.
Service is functioning again. We are monitoring the situation.
The problem has been resolved.
The database completed restoring at 12:55 UTC. Service is operational and functioning again.
The job is now 92% completed, but has slowed down, new extimate for completion is 13 UTC.
New expected restore time is aproximately 11 UTC.
Service should be back up approximately 10:30 UTC.
The Database is still recovering, it is expected to take a few more hours for the recovery to complete.
Database recovery is still ongoing....
Kaltura is down again. Database issue.
All services fully restored.
Problem has been resolved and services are restoring.
Issue is caused by database problems.
As load is building up the system and response times remain low and stable.
Corrections have been implemented, but will have to be verified during real load.
Issue is stille being investigated.
The Moodle integration issue has caused the platform to be unresponsive, and considered to be down. We are working on the isuues.
We are working with Panopto to resolve the problem.
Problem has bee resolved
The migration of the Panopto front-end serveres has been sucessfully completed.
The changes in the Datacenter has improved the the situation. We will continue our action plan and will as a next step migrate all Panopto Front-End servers from the virtualised environment to physical servers.
There continues to be load related issues affecting Panopto. Over this week we will be optimising things in the datacentre infrastructure. In addition later in the week we will deploy new load balancers and add cashing capacity.
After the upgrades done last night, we will monitor the situation.
the load related issues have been resolved.
To night September 30, there will be additional work done that should further improve the situation for both MediaSite and Panopto.
There have been several updates and performance optimisations during the weekend. MediaSite on-demand playback should be much improved even though loading the player itself, could take some seconds during peak hours.
There are still load related issues during peak hours. We are working to resolve the issue.
We are still monitoring the situation but the capacity upgrade and rebalance of all systems seems to have resolved the issue.
Resources allocated to Mediasite and Panopto has been rebalanced. We continue monitoring and will decided on further actions if the issues persist.
More system capacity was added again last night (15/9-2020). We continue to monitor the situation.
More capacity has been added. We continue to monitor the situation, and will be adding more system capacity if the problem persist.
A fix has been implemented and so far the metrics are looking good.
service is back up. We are investigating the cause.
We are working with Zoom to find the root cause.
Maintenance sucessfully completed
Things remain stable.
The issue was caused by a problem in the Datacenter and the service has been restored.
Maintenance completed.
Capacity was added as planed.
Zoom AWS is back to normal operations.
We are monitoring, and waiting for feedback from Zoom on the cause of the outage.
Zoom AWS is back up.
Organisations that migrated to on-prem, can migrate back to licensed.
The service is fully restored.
The Zoom AWS cluster is back up, and working, but some users will still get various error messages.
Users with an urgent need to host meetings can migrate to on-prem by having an administrator change setting from from licensed to on-prem.
Maintenance activities sucessfullt completed
The resource rebalancing was completed and the systems are back in operation.
Analytics has stopped working again.
We will continue to monitor the situation over the coming week.
A fix has been found and implemented.
We will closely monitor the situation during the day.
No cause for the slow response experinced has been identifyed.
Problem has been resolved for all accounts.
The data is starting to become available again.
The problem relating to the storage infrastructure, has been identified and fixed. Normal operations will be fully restored within 1 hour.
We are currently investigating.
Problem has been resolved. The issue was caused by a packaged deployed with inconsistant module versions.
Problem is being investigated.
Backlog is still being processed. Normal operations are expected at 14:30 UTC.
Back to normal opertions.
Problem has been identified and rectified. Backlog is still being processed.
Issue is being investigated.
We will keep monitoring the situation.
The issue has been resolved. Waiting for more information regarding the cause of the outage.
Zoom have identified the issue causing users to be unable to authenticate to the Zoom website (zoom.us) and unable to start and join Zoom Meetings and Webinars, and are working on a fix for this issue.
The issues is being investigated with the highest priority.
Maintenance sucessfully completed.
This issue has been resolved
The problem is currently being investigated by Kaltura.
The issus have been resolved
The problem with thumbnails affects all Kaltura deployments, not only the NORDUnet instance.
The Zoom AWS cluster has been stable for 24 hours, and users can migrate back to the AWS cluster by changing setting to licensed.
During the incident more than 50% of accounts have been moved to run on-prem. We will be monitoring the situation for the next 24 hours before taking any further action. We are waiting for a more detailed RFO from Zoom.
The issue has been resolved and service for the customers running licensed has been restored.
We are working with Zoom to find the cause of the problem with the AWS cluster.
As a temporare meassure, to get things to work, it is required to change the setting from cloud to on-prem.
We are investigating and looking at mitigation options.
Kaltura is currently investigating possible solutions to this problem.
Service restored investigating, failover issue.
Migration was completed as planned without any issues.
The Netwok problem has been resolved.
The cause of the problem is being invetigated.
We are working with Panopto to find a solution.
The problem has been resolved.
Work is still ongoing to establish a solution.
The Kaltura database re-sync has completed and everything is back to normal operation.
The Kaltura database re-sync will not be completed until aproximately 14 UTC.
The problem has been resolved. The Kaltura database is currently re-syncing and Kaltura can therefore not accept new video material. When the re-sync is completed transcoding will resume. The Kaltura service is expected to be fully operational at 11 UTC.
The problem is Intermediate packet loss. This is specially affecting Kaltura due to recovery times.
We have received information from Zoom that even if the Datacentre is labled as AM the meetings are actually hosted in the AWS cluster in Stockholm. The Datacentre will be relabled to show SK.
Still being worked on
We are working with Zoom to migrate licensed accounts back to the Stockholm datacentre.
No further issues have been seen or reported.
The problem was caused by a backend storage job moving large amount of data and causing poor performance for the Kaltura API servers communicating with the storage backend. The backend job has completed, but we continue to monitor the situation.
The issue is caused by communication problems between the Kaltura servers and the storage system.
We have received a confirmation that Zoom will not be deploying a EU based connector and not be providing the local virtual connector for Skype for business. The reasoning is that SFB is coming to the end of life for the product and Zoom will not be providing any more updates to the virtual connector if something were to break and Zoom want to prevent new customers from jumping on the software that is not going to be supported. Zoom is working with the documentation team to remove the option from the admin profile.
The recommended workaround is to install the Zoom client and use the plugin https://support.zoom.us/hc/en-us/articles/204481169-Zoom-Skype-for-Business-Lync-Plugin.
If it is not possible to use the Zoom client, please refer to the other workaround options.
As a temporare meassure it is possible to use either SIP when connecting from Skype for business to a Zoom meeting, or use the clientless direct browser access.
The root case was that due to the network problems causing firwall flaps, the Kaltura master and slave datatbase got out of sync. The system was unable to accept new data until the databases were re-sync'd and database consistancy was restored.
The issue has been resolved, but there is a large backlog of videos that need to be transcoded.
The issue is still being worked on and is also affecting the performance of other parts of the Kaltura service.
The issue was resolved by reloading the software on a core router component.
The problem has been resolved and all users previously running with Copenhagen as the primary location have been migrated back.
As a workaround users are being migrated to Stockholm.
upgrade completed.
The problem has been resolved. A Reason For Outage (RF) report will be made available.
The problem has been located to a routing problem for the Storage network.
The backlog has been processed and upload times are back to normal.
Problem was fixed on April 5, at 04:00.
Waiting for information from Zoom. Next update Monday April 6, at 08:00 UTC
We are investigating the issue to identify the cause. Next update at 10:00 UTC
The issue has been fixed.
Waiting for Kaltura support. Next update, March 30, at 09 UTC.
There is currently a problem with livestreaming on the Kaltura platform. The issue is causing some live streams not being able to start. The issue has been escalated to Kaltura support.
There is currently a problem with livestreaming on the Kaltura platform. The issue is being investigated. Next update at 08:00 UTC.
The problem has been resolved.
The issue is being investigated with Kaltura on-premise engineering. Next update at 12:00 CET.
Additional API server added.
The issue is being investigated with Kaltura on-premise engineering.
Additional API servers were added at 03:00 UTC
The issue is being investigated with Kaltura on-premise engineering. Work is ongoing to add more system resources
The issue has been resolved.
The issue is being investigated with Kaltura on-premise engineering, to establish what can be done to improve system performance.
The issue has been resolved but users will experience a degraded performance as the upload backlog is being processed.
All systems are comming back online, and returning to normal.
During maintenance we experienced a large outage on ndn services running on our virtualisation setup. The cause was human error, but we are looking into how/why it ended up affecting most of our systems.
We have found the cause and are working with Zoom to see if we can find a work around. For now please change users to Licensed instead of On-Premise as the system hits low logical limits and then block new meeting creation.
The issue has now been mitigated, we continue to monitor the situation. Official zoom status
We are currently having errors creating new meetings on the Zoom service for both Cloud and On-Premise. We are working with Zoom to resolve this issue.
The issue has now been mitigated, we continue to monitor the situation.
We are currently having errors creating new meetings on the Zoom service. We are working with Zoom to resolve this issue.