Every water system in the country runs on relationships it does not control. The most familiar one is mutual aid: a state WARN program, an interlocal agreement with the system in the next county, a cell number for the operator who has the spare pump. Others matter as much. Reporting to a primacy agency. A call to an FBI field office. Sector information sharing. The state association that forwards a notification on a Tuesday morning. Each one is a coordination relationship and each one rests on a quiet assumption: when the moment comes, there is a working way to reach somebody outside your organization.

Upcoming Webinar: What CI Fortify Doesn't Say About Communications

For most emergencies the assumption is sound. A main break at 4 a.m. does not take out the phone system. A flood takes out roads and lift stations and the utility two counties over is dry and answering its phone. The relationship works because the failure is local and the help is not.

A cyber incident has a different shape. It can take the email and the phone system in the same event that takes the monitoring and the contact roster with them because all of it sits on one network and authenticates against one directory. In the scenario sector planning almost never contemplates, it takes them at both ends in the same week.

Two Utilities, One Campaign

Two systems, 40 miles apart. Both serve small communities, both belong to the same state WARN program, and both are touched by the same activity inside the same seven days.

The first system needs help. It needs an operator who can cover a manual shift and somebody who has seen this before. Its superintendent has been advised to treat the enterprise network as suspect, which puts the email account he would use to make the request, the phone system riding the same circuit, and the contact roster on the shared drive inside the boundary he was just told not to trust.

The second system is the one he would call. It is also mid-incident. Its people are committed, its manual coverage is thin, and its superintendent is running the same calculation about his own communications channels.

The state is getting partial information from both, later than it wants it, while trying to work out whether this is two events or one.

None of that requires an exotic adversary. Mutual aid works because the responder's spare capacity has nothing to do with the requester's failure and geography usually guarantees that. Correlation is the whole point of a campaign. When the same activity reaches many systems inside one window, the assumption that somebody nearby is fine and available is the first thing to go.

Reachability Comes Before Verification

The security instinct is to treat this as an identity problem. How does the second utility know the request really came from the first utility's superintendent and not from somebody who has been reading his email? How does the state validate what it is being told? Those questions are real and the sector will have to answer them.

They are the second question. The first is whether a channel exists at all: one both parties can reach without depending on either organization's primary network or directory. Nobody authenticates on a channel they do not have.

Continuity planning tends to stop right about here. Plans name the parties and list the numbers. Fewer plans say where those numbers live when the network holding them is the network under suspicion or which channel carries the conversation when both sides are operating under the same warning.

That is a fair question to put to vendors, including this one. A coordination platform delivered as a cloud service is a dependency of the internet connection an isolating utility may have just severed. The default delivery model for BlackBerry® AtHoc® is the cloud. The configurations that hold up with external connectivity cut are on-premises and hybrid and an operator buying coordination capability should know which one is on the quote before signing it.

Detection Is Arriving. Coordination Is Still Yours to Provision

Something real changed for small systems this month. On August 7, DEF CON Franklin and the National Rural Water Association launched the Water Watch Center, which puts managed detection and response providers to work at no cost for utilities serving fewer than 10,000 people with NRWA as the national hub for what those providers see. Roughly nine in ten of the country's community water systems fall under that population line. For a utility with no security staff, this is the difference between having monitoring and having none. The sector has been stuck on that question for years.

Detection tells you it happened. Coordination is how you respond.

It also sharpens the question. A superintendent who gets a credible alert at 11 p.m. is in a far better position than one who learns about it from a customer complaint on Thursday, and he now owns a set of decisions no monitoring service makes for him. Who gets woken up. Which neighbor gets called. When the primacy agency hears. What the city manager is told and what any of it looks like if the enterprise network is part of the problem.

Coordination capability is a distinct thing from detection capability and distinct again from the operational capability to run the plant by hand. Of the three, it is the one most often assumed rather than provisioned because it looks like a list of phone numbers instead of a system.

Five Questions to Answer Before the Next Activation

1. If your email and your phone system were both suspect at 2 a.m., what number reaches you and who outside your organization has it?

2. Where does the mutual aid roster live and can you get to it after disconnecting the network it sits on?

3. When you ask a neighboring system for help, how does that system confirm the request came from you?

4. If the two partners you would call are both handling their own incidents, who is the third call and have you ever placed it?

5. Can you produce a record of who you notified and when without assembling it from memory a week later?

A superintendent can work through those in an afternoon. The answers come out uncomfortable in the same places for almost everybody.

What This Means for the People Writing the Rules

Those five questions double as a policy diagnostic. Primacy agencies and legislators are deciding right now what small-system cyber money can be spent on and eligible-use lists get written around things that can be inventoried: assessments, equipment, monitoring subscriptions. Coordination capability resists itemization so it gets funded only when somebody names it on purpose.

States have a self-interested reason to care. In a multi-system event, the state sits downstream of degraded and delayed reporting from several utilities at once, trying to brief a governor on what is happening in its own sector. The channel carrying a superintendent's status report helps determine whether the state has an accurate picture of the situation at all. That is worth funding on its own terms.

Related reading:

• Siemens S7 Series Programmable Threat Advisory: What Water Utilities Need to Know

• Water Watch Center Highlights Water Utility Cyber Response Gap

• Why CI Security Guidance Keeps Failing Small Utility Operators

• Assume Compromise, Then What? Moving Past the Slogan to Operational Requirements

• What CI Fortify Asks Utilities to Spend — And the Capability It Never Names

• When the Phones Go Down: The Coordination Gap Nobody Plans For

• Lessons from the July Water Utility Attacks: What the Joint Advisory Tells Operators to Do Next

• Why OT Isolation Planning Needs Communications Continuity