Active Incident

Updated a few seconds ago

For detailed Network status please visit: https://nordu.net/tickets/

Incident Status

Operational

Components

Zoom

Locations

Recording



September 2, 2024 12:22 UTC
[Investigating] There's an issue with zoom Recording. You may notice delays and the recordings take longer than normal to appear. This has been escalated to Zoom but please send in a ticket via your NREN support channel anyway so they know how many that is affected.

September 16, 2024 13:20 UTC
[Investigating] When utilisation increased after the summer holidays, we have encountered an intermediate issue with the Zoom on-premise recording service. This has resulted in the delay of the processing and making the recordings available and has unfortunately led to the loss of recordings that cannot be recovered. During the period from August 27 to September 5, the average loss of recordings was around 5%, in total around 250 recordings were lost. The night between September 5 and 6 we implemented a workaround changing the disc configuration on the recording server VM’s. After that the number of lost recordings has dropped to less than 0,5% and a total of 5 recordings have been lost since. The root cause for the problem has so far not been identified. There are currently two different lines of investigations. We are in constant contact with Zoom and provide them with all requested information. If there are critical recording needs, it is recommended to use local recording until the issue has been fully resolved.

September 26, 2024 14:26 UTC
[Investigating] The root cause for the problem has still not been identified. Zoom has deployed a new version of the recorder with added debugging, to help with the investigations. The level of lost recordings is still in the 0,5% range. Some of the lost recordings can be partly recovered manually, others are just very late (+48Hours) If there are very critical recording needs, it is recommended to use local recording until the issue has been fully resolved.

October 14, 2024 11:33 UTC
[Identified] Zoom believe that they have identified the root cause of the problem. It has to do with that the recording connector needs to migrate files during processing. (If the disk is busy at the time the file movement might fail and this is resulting in a “dirty” file, and this also affects subsequent files and is the reasoning for the generic slowness.) Zoom is working on a new release of the recording connector, that will be deployed as soon as available.

October 29, 2024 11:30 UTC
[Monitoring] A new version of the Recording Connector has been deployed. The new version includes a permanent fix for the recording problem. During the next weeks we will monitor closely to verify that no recordings are being lost and that the recording flow is stable.

November 4, 2024 09:58 UTC
[Monitoring] Unfortunately the new version of the Recording Connector did not resolve the problem. Zoom is continuing to work on a solution at their end.

November 18, 2024 08:53 UTC
[Monitoring] Things are continuing to improve. The changes made last week has uncovered an NFS permission problem on the recording servers that Zoom has been able to reproduce. A fix for this was deployed over the weekend.

November 25, 2024 10:11 UTC
[Monitoring] The final problem, ref. last weeks update, has been resolved and everything is back to normal operation, with 0 recordings being lost this week. It has also been possible to recover most of the old missing recordings and these have been processed as well. For now we consider the issue to be resolved. A permanent solution will be deployed later.

Network




Operational

Zoom




Operational

Kaltura




Operational

Panopto




Operational

Box




Operational

0

Upcoming Maintenances

0

Incidents Last 30 Days

1

Maintenances Last 30 Days

External Services

History (Last 7 days)

Description

There will be an infrastructure maintenance in Denmark. This was previously planned over two weekends doing one datacenter per weekend. Now both datacenters will be done on one weekend. The operation requires all nodes of the clusters to be in maintenance mode at the same time, which in practical terms means that the cluster will be "shut down". And in order for this to happen, all VMs in the clusters will have to be powered off for the duration. Downtime is expected during the maintenance window on Saturday. Sunday is reserved. Impact: - Kaltura will be completely down during the operation. - Zoom Cloud Recordings might have longer delivery time. - Zoom Virtual Room Connector Load Balancers will be down. Users can either use the Public Cloud CRC or use the Hybrid CRC servers directly: dk-bal-zp2-hcrc-001 A 185.174.119.220 dk-bal-zp2-hcrc-002 A 185.174.119.221 dk-bal-zp2-hcrc-003 A 185.174.119.222 se-tug-zp3-hcrc-001 A 89.46.17.220 se-tug-zp3-hcrc-002 A 89.46.17.221 se-tug-zp3-hcrc-003 A 89.46.17.222 dk-ore2-zp5-hcrc-001 A 185.174.118.220 dk-ore2-zp5-hcrc-002 A 185.174.118.221 dk-ore2-zp5-hcrc-003 A 185.174.118.222


Components

Zoom, Kaltura


Locations

Mediaspace, Transcoding, Live streaming, On-Premise, Recording, REACH, Analytics


Schedule

November 30, 2024 07:00 - December 1, 2024 20:00 UTC



November 30, 2024 07:00 UTC
[Update] Scheduled maintenance is starting.

November 30, 2024 18:56 UTC
[Update] The maintenance is complete.

December 2, 2024 06:40 UTC
[Update] Maintenance is finished.