The president of a Portland-based optical network company said Thursday's equipment failure that shut down business and government telephone lines and slowed Internet activity to thousands in Southern Oregon was an "unimaginable event."
Michael Weidman, president of Lightspeed Networks Inc., which operates LS Networks in parts of Oregon outside the Willamette Valley, said built-in safeguards couldn't keep lines open for many of the area's independent telecommunications, Internet and data transmission companies. Those companies' customers in turn lost their voice and data connections.
"We have redundancy upon redundancy built in, but the whole shelf dropped and we were not able to restore the system remotely from Portland," Weidman said Friday. "The system is redundant and resilient with diverse routes in and out of Medford in the event of a fiber cut. But the unimaginable event occurred."
About 200 circuits, each carrying between 1.5 megabits and 10 gigabits of information, failed when software running the system became corrupted, he said. A 10 gigabit circuit can handle thousands of online customers simultaneously.
"The best way to understand the difference in the circuits, is to think of a garden hose next to a fire hose next to an oil pipeline," Weidman said. "There's a lot more flowing through the pipeline (with the larger circuits)."
Exactly what caused the software to go bad is still being reviewed.
"What would corrupt the software, we don't know," Weidman said. "We have our vendor looking at it at their lab."
The outage began just before 1 p.m. Thursday. Telephone and Internet service began returning after Hunter Communications technicians working with LS Networks at the Medford Qwest central office restored connections around 4:30 p.m. It took at least another two hours for service to return to normal.
The crash affected a variety of telephone, Internet and data providers. Medford-Ashland area businesses who subscribe to those providers were particularly hard hit.
Service providers affected included Hunter Communications, Ashland Fiber Network, Charter Communications, Clearwire, Infostructure, Integra and Rio Communications.
Although he could talk about the volume of loss, Weidman was unsure of the number of accounts affected, because many were clients of LS Networks' customers.
A spokesman for Hunter Communications said the "worst-case scenario" event will spur the affected companies to develop back-up plans.
"This occurrence will generate a series of meetings between all communications providers involved and new paths will be put into place to avoid the same issue in the future," Hunter Communications' Chris Cahill said.
Reach reporter Greg Stiles at 776-4463 or e-mail email@example.com.