As students tried to get into Banner, Canvas and other online John Carroll services on Monday, April 23, they were met with error messages. This system interruption lasted roughly eight hours, beginning before noon and being resolved by 8:06 p.m. While many students believe the Wi-Fi simply stopped working, Jim Burke, JCU’s chief information officer, and Marc Barnett, director of IT infrastructure and senior network engineer, explained that was not the case.
“The network was totally fine, but one of the services that allows you to use it was what got attacked,” Burke said. The domain name service, or DNS, was not able to find the IP addresses for the websites and single-sign on (SSO) to allow users to access these resources. A DNS is “a lookup service that says ‘this name belongs to this [IP] address,’” according to Barnett, which meant that when students tried to sign in or reach a website, the service could not identify it and created a lack of access.
“In the beginning, it almost looked like we were being attacked,” Barnett said. Essentially, there was an update with the software that is “running on hundreds of other machines on campus and every other main server we own without any issues… When it updated on this machine, it took out the service,” Burke explained.
IT uses third-party cybersecurity called Sentinel One to address possible attacks and the company is still unsure of whether the problem was an attack or another issue with the software. “The company is still diagnosing the issue with the software, so I don’t know at this point if it wasn’t an attack,” Burke noted. However, “this was unusual that it was totally from the inside,” he said. Typically, cyber attacks come from outside threats, but the situation that took place on Monday was from something that occurred internally.
However, JCU ITS assured that systems are prepared for attacks or interruptions. “Normally, if we would have had a failure on this service, we have redundant systems in place that automatically take over. However, it affected our redundant systems as well,” Barnett said. If this issue with the DNS had no effect on the redundant systems, the DNS would have remained functional.
Also, due to the level of cybersecurity that JCU has in place, the university has cybersecurity insurance. “Many of our peers can’t get cybersecurity insurance because they are not as secure as they need to be… Every day we look at what we can do to make ourselves more secure,” Burke said.
When working through these issues, the Information Technology Services, or ITS, department attempted to reroute valid users to access sites and SSO while trying to prevent a possible attack. “We were able to do that, but by the time we got that in place… we had identified the issue and were testing the remediation,” Burke explained. At that point, they decided to wait about 15 to 20 minutes to let all users know the service was running again.
If students or faculty want to see live updates on technology services around JCU’s campus, they can visit status.jcu.edu. It is a separate site than Banner or My JCU, but JCU ITS frequently updates the status page to reflect any disruptions and whether or not technology issues have been resolved.
Burke and Barnett noted that they are trying to improve their processes so an outage of this length does not occur in the future and urged students to contact JCU ITS if they notice anything out of the ordinary. “We’re definitely after the facts and still looking at all of that to figure out how we can improve our processes,” Burke said. Barnett explained, “I can’t fix things that I don’t know about… We tend to not get a lot of feedback.”
If you are having trouble with technology services on campus or would like to know more about JCU ITS, email [email protected] or call (216) 397-3005.
