On Monday 5 October we were one of many organisations and communities that were affected by the Europe-wide disruption to Panopto, our lecture recording service. The Panopto Chief Executive, Eric Burns has sent us a letter (please see the full text below) apologising for the inconvenience caused and saying, “This is absolutely not acceptable, and the problem and its resolution are entirely our responsibility.”
The University’s Executive Director of iSolutions, Mal Allerton said: “We would also like to apologise for the inconvenience caused by yesterday’s outage on the first day of teaching and we will continue to work with Panopto to avoid anything similar happening in the future.”
On a positive note, despite three hours of downtime, the University of Southampton streamed 5,590 hours of video on Monday. Our previous peak was when 3,084 hours were viewed on 28 April.
Dear University of Southampton,
As CEO and co-founder of Panopto, please accept my heartfelt apology for how we have let you down over the past few days. Our European Panopto cloud had multiple service outages of greater than one hour, culminating in three hours of unplanned downtime on October 5th during a day in which Panopto did not deliver expected performance, reliability, and responsiveness. This is absolutely not acceptable, and the problem and its resolution are entirely our responsibility. We also recognize that these disruptions could not occur at a worse time, as students return and our fragile new models of technology-supported instruction are being put to the test. Understanding this, our engineering teams have been working continuously since the first incident last Thursday to stabilize our European cloud. We have immediately begun rolling out a significant infrastructure expansion that we are confident will accommodate the enormous increase in universities’ use of Panopto this term.
Throughout the pandemic, Panopto has worked to support our customers’ rapidly shifting technology needs while ensuring that we are continually investing in the scalability and reliability of our technology platform. Despite the significantly elevated level of use of Panopto in Europe and the UK to date, we anticipated an even greater increase during the back-to-school period. Last year saw recording and streaming climb about 230% from the first week of September to the first week of October; this year, despite an already much higher baseline, it increased by more than 450% in a single month. Even so, all of our cloud monitors and analytics indicated that Panopto’s European cloud continued to operate well below the utilization levels we had already observed in our North American cloud with a comparable configuration. Our engineering team was confident that we were prepared for the back-to-school increase in Panopto use.
This is an incredibly serious situation for us and for you, and I will continue to communicate directly with you until it has been resolved. We will do our best to restore your trust in Panopto and your confidence in our service. I am very sorry that we’ve disappointed you, but as we work to recover, I would like to thank you again for your patience and your business. We will follow up soon with stats updates and a full root cause analysis once the immediate problems have been addressed.
Eric Burns CEO
If you have any concerns or issues following the Panopto outage yesterday please email [email protected].