Our president recently asked me to spend a few minutes with the vice presidents, provost, and deans at his upcoming leadership council meeting, revisiting a Board of Trustees briefing on two matters Auburn managed in October: the eMudhra certificate distrust and critical PHP vulnerabilities. Ten minutes seemed manageable.
Then Vanderbilt was hit by ransomware on Monday. Microsoft 365 failed on Thursday. On Friday, a communications failure led the FAA to halt departures nationwide. The briefing now had to cover what it meant for Auburn. I was happy when the president’s chief of staff called back to say I could have ten more minutes. I spent Saturday updating the presentation, hoping nothing else would happen.
I doubt my president’s request was unique. Presidents and boards across higher education are asking their CIOs some version of the same question: what does all this mean for us, and what are we doing about it? But these events share almost no common cause, and walking through them produces only a chronology of bad news.
In today’s Dispatch, I reveal how I responded to this challenge at Auburn and attempt to provide CIOs everywhere with guidance on how they can explain incidents with different causes as evidence of one institutional condition: systems so interconnected that a small failure in one place can quickly become a problem for the whole university.
The big picture
Each event had its own cause. Vanderbilt is a criminal matter, though the university has not confirmed the ransomware group’s claims. Microsoft called its outage “a technical issue not related to cyber or AI.” Federal officials found no evidence of a cyberattack behind the ground stop, and experts are studying a failed network upgrade.
Root cause matters because it changes what should follow. Ransomware puts identity and endpoint controls under scrutiny and may require breach notification. A cloud outage tests continuity plans and the provider’s contract. A vulnerability tests how quickly the institution patches what it runs. The eMudhra incident tested whether anyone knew which systems depended on the affected certificates. Calling all of these “cyber incidents” hides those differences from executives who must decide what to do.
Different causes, but one institutional condition: digital brittleness. Brittle does not mean unreliable. The analog university was slower and more constrained, and many failures stayed local. Digital systems removed that friction and connected activities once separate. Identity, networks, cloud platforms, vendors, data, laboratories, and administrative processes now depend on one another. That interdependence is the source of the modern university’s capability. It is also the path along which failure travels. An ordinary event can produce a consequence far out of proportion to itself.
A CIO’s briefing must go beyond what happened. It must explain how a compromised account, provider outage, or poorly governed change becomes a university-level event.
Small Things, Large Consequences
Serious incidents often begin with an ordinary, simple action. Someone opens an attachment, works from an account with administrative privileges it does not need, leaves an internal application unpatched, or changes a production system without a tested way back. The action matters less than the conditions that allow it to spread.
Vanderbilt has not said how the attackers got in. Early information shared through open sources points to a familiar sequence: a foothold on an employee’s workstation, then movement to systems never exposed to the Internet. The conclusions remain tentative, but they should change how CIOs think about internal-facing applications. Once an attacker controls a workstation inside the network edge, an “internal-only” server is no longer behind anything. Internal is a location. It is not sufficient protection. Damage depends on whether the account carried unnecessary privileges, endpoint telemetry noticed the foothold, and patching had closed the next door.
The same interdependence that lets an attacker move laterally can propagate an operational failure. Microsoft traced its outage to a network capacity upgrade, and the FBI reportedly examined whether a similar upgrade grounded the FAA. A poorly tested change, made at a consequential time without a reliable way back, can travel just as quickly. Change management belongs in the same briefing as ransomware.
Good risk management in federated IT creates real costs. Separate privileged Workstation accounts are less convenient for end users. Central endpoint management constrains local autonomy. Vulnerability discovery creates an obligation to patch on an institutional schedule. A CIO who dismisses those costs will not be believed. But friction should buy something: privileges kept off everyday accounts, visibility for security staff, or a reversible change. Otherwise it is just delay.
Leaders outside IT sometimes read an incident as a failure of technical prowess. More often, the harder problem is persuading dozens of units, each with its own authority and priorities, to accept a shared inconvenience before anything goes wrong. That is a people problem. It takes negotiation and leadership at least as much as engineering.
Common Protections Must Be Common
At Auburn, those capabilities include Microsoft Defender for Endpoint (Plan 2); Intune for workstation management; and Qualys for server vulnerability management. Ordinary and privileged work occur through separate accounts. The products matter less than the capabilities: endpoint protection and telemetry, management and automation, vulnerability discovery, privilege separation, and institutional visibility.
Enterprise-wide scale turns those capabilities into institutional protection. When thousands of devices report into one security operations center, security staff can correlate evidence across colleges or other units. A compromised machine can be contained automatically, wherever it sits on the organization chart. A device outside that environment is invisible. The tools guarantee nothing, but they allow the institution to respond across federated lines rather than one unit at a time.
Exceptions Are Governance Decisions, Not Technical Accommodations
The strongest objection to common controls comes from units that know what they are doing. A well-resourced college may have a legitimate reason not to implement a control exactly as prescribed, and its compensating control may be as good as, or better than, the institutional standard. Central IT should say so when that is the case.
The problem begins when nobody has documented the exception, assessed the residual risk, or asked the accountable executive to accept it. A technical exception is an institutional risk decision. A credible exception identifies the control being waived, documents the compensating measures and residual risk, and puts that risk before an executive authorized to accept it. And those exceptions should expire.
Technology organizations are often experienced as the “no police.” The better role is a trusted advisor that names the risk and exposure, explains the consequence, evaluates compensating controls, and recommends alternatives. The accountable executive accepts the residual risk. Documentation makes the decision visible.
Accountability runs both ways. If an accepted exception leads to an incident, the executive who accepted it owns that decision. If prescribed controls prove inadequate, the CIO bears greater responsibility for the architecture, controls, and response. The point is to align authority with accountability while leaving room for local innovation.
Where the Work Belongs: Edge, Leverage, and Trust
Exceptions raise a prior question: where should a capability operate? Core capabilities, identity among them, require institutional control. Common capabilities gain value through shared standards, contracts, tools, or providers. Unique capabilities belong close to specialized teaching, research, or administrative work.
Brad Wheeler described the relationship sustaining those placements at Indiana University as edge, leverage, and trust. Edge keeps authority near the work. Leverage supplies what would be expensive, risky, or fragmented if every unit reproduced it. Trust is structural: reliable services, clear obligations, visibility, reciprocal accountability, transparent exceptions, and recourse when either side fails.
A university does not need every technology professional to report through the CIO. It does need shared controls, clear responsibility, and a documented process for exceptions. That need makes most cyber risk a people problem first and foremost.
Anything Connected Should Assume Failure
Anything connected to the Internet carries this risk, and a research university should assume that some controls will eventually fail. Preventive controls matter enormously, but no CIO should promise that good enough prevention will make significant incidents impossible. That’s a lie. A cloud provider will go down. A threat actor will sometimes get through. Someone will click the wrong thing. A tested change will still surprise its creators when a dependency behaves differently than expected.
None of this argues for fatalism. It argues for resilience: prevention, detection, response, recovery, learning, then back to better prevention and detection. Institutions often shortchange learning, filing an after-action report once the pressure passes. Learning belongs between recovery and the next cycle of detection. An incident should change telemetry, thresholds, playbooks, ownership, and sometimes architecture or controls. If nothing changes, the institution recovered without learning.
The same logic applies to board briefings. Challenge, action, and result stop too early. Learning is the fourth step because it shows how the last event improved detection and containment. Presidents and boards deserve reassurance, but credible reassurance comes from response capacity, not a promise that nothing will happen.
The final word
That Saturday, while I worked on the presentation, I talked by phone with Jere Morehead, president of the University of Georgia, where I spent fourteen years before coming to Auburn. He asked me a simple question: “What can we do about all of this?”
My answer was that we had to accept that the digital world is far more brittle than the analog one. Partly, that is the technology, connected in ways even its builders do not fully document or understand. Partly it is dependence, because the university relies on those systems for nearly everything consequential. And the Internet exposes those dependencies to threat actors and events outside the institution’s boundaries.
We will all experience similar incidents eventually. It might be a threat actor or provider, our own mistake, or a dependency nobody understood until something important failed. Preventive controls need to be good. Our ability to detect and respond must be just as good. Recovery matters, and learning closes the loop.
My expanded briefing for Auburn did not change the question or the answer. It added examples. Resilience begins when we stop treating response as evidence that prevention failed and begin treating the full lifecycle, learning included, as the work.
Dispatches from an Internet Pioneer is a free regular essay about technology, leadership, and institutional change in higher education. Each takes up a question large enough to organize decisions around. This essay asks how research universities should respond to a digital environment in which small failures can quickly become institutional events, and why prevention must be joined by equally strong detection, response, recovery, and learning. If this argument would be useful in your institution, subscribe or forward it to a colleague.


Tim,
This is very informative and helpful. Thanks for posting it. Your guidance over the years you were here with us proved invaluable. I'm looking forward to your next Substack post, no matter the subject.