This composite account is assembled from published advisories, after-action reports, and conversations with operators at systems of this size. No single utility is described. The narrator runs a water system serving about 25,000 people with eleven employees, contracted IT two hours away, SCADA supported by an integrator, and no overnight shift.

The first call came at 9:40 on a Tuesday morning. Our integrator was in the historian doing something routine and saw outbound traffic from a segment that has no business talking to anything. Not their traffic. Not their maintenance window. They couldn’t account for it, and they were going to pull logs and call me back.

Upcoming Webinar: What CI Fortify Doesn't Say About Communications

Six minutes, that call. I spent the next twenty assembling ordinary explanations, because that’s what you do. The firmware push last month. The cellular backhaul on the new pressure sensor. A vendor laptop somebody forgot to take off the network in April.

At 10:35 our state association forwarded a notification. Systems running a particular remote access product exposed to the internet were appearing on a target list, and here was the guidance. We run that product. We bought it in 2021 so the integrator could troubleshoot at night without driving three hours each way.

Neither proved anything. Together, they meant I couldn't run the day as normal, but I couldn't declare an incident either. Our emergency response plan has a threshold for a confirmed intrusion. It has nothing for two items that might both be nothing.

The Calls You Make Before You Are Allowed to Call It an Incident

The integrator first. What do you actually see? What can you rule out? When can somebody be on site? Thursday, they said, unless we escalate.

Then contracted IT. Our MSP is good. It is also two hours away. The contract covers business hours, and the engineer who knows our environment was at another client. A ticket got opened. So now the sentence “possible unauthorized access to the control network” lives in a ticketing system I don’t administer, with notifications routed to email addresses on my domain.

Then the operator on duty. Change nothing, write down anything that moves, call me if a setpoint shifts on its own.

Then the city manager, before lunch, because if this becomes a public matter the first time she hears about it cannot be from a reporter. She asked whether the water was safe. I told her yes, and that I knew it from residuals we were reading by hand rather than from the system that normally reports them.

Somewhere in there I started typing the situation out for the integrator and stopped with the cursor blinking. Every sentence I was about to write described a weakness in my plant, and the account I was using authenticated against the same directory as the machine we were worried about. Our tenant is shared with the city. I had no reason to think anybody was reading it, but no way to prove that nobody was. So I called instead, and then I had the same conversation four more times by voice, and by 2 p.m. I had a clear picture in my head and not one line of it written down anywhere.

The Decision and What It Actually Costs

Thursday’s plan became Tuesday afternoon’s decision. Pull the remote access, break the segment, run manual until the integrator can rebuild from something we know is clean.

On paper that is three moves. In staffing it is this: chemical feed by hand, residuals logged every two hours by somebody certified to do it, and I have three people who qualify. One of them was at his daughter’s graduation in another state. No overnight shift means somebody has to cover 11 p.m. to 7 a.m., which means calling people at home, then calling spouses, then driving to a house because a phone was off.

The app I would normally use to sort coverage runs on the network I had just cut. The printed call list on the wall by the SCADA workstation was fourteen months old and two of the numbers were dead.

Manual operations are more a chain of custody than a procedure. Whoever comes on at 11 has to know the last reading, what we changed, and what normal looked like before Tuesday. That is a conversation, and it happened six times over the next four days, every one of them on somebody’s personal cell phone.

Going External Before You Know Anything

The primacy agency had to hear from me, and the reporting clock does not wait for you to have a complete technical picture. My first report was mostly a list of things I couldn’t confirm. Unexplained outbound traffic. An exposure matching a notification. A segment disconnected. Manual operations in effect. No evidence of process manipulation, and no evidence ruling it out.

The state asked who was doing the forensics. I said the integrator and our MSP, and I heard exactly how that sounded as I said it.

The FBI field office was professional and wanted artifacts, but those artifacts were sitting on the network in question. I also learned that “do we tell the public” isn’t a technical question, and that the conversation goes considerably better with a city attorney on the line.

Coming Back

Nine days, start to finish. The integrator rebuilt the affected machines from known-good images, we came off manual on a Sunday afternoon, and the last decision turned out to be the one nobody writes a procedure for. At some point I had to believe a sensor again.

Nothing produces a certificate that says clean. What you get is a set of people telling you what they checked, and your own judgment about whether that was enough. I signed off on a Sunday because the hand readings had been steady for nine days and two people whose work I know told me the rebuild was sound. That’s the honest version.

The Part that Comes After Which Nobody Warns You About

Then the paperwork, and it has teeth. The state wanted a timeline. The board wanted to know when the city manager was told and who authorized manual operations. Our insurer wanted dates and durations. What I had was a call log on my personal phone, a legal pad, and a group text thread from the second night that I would rather not hand over to anyone.

Recollections are not a record. The channels you improvise on are the channels expected to produce evidence six weeks later.

Every mitigation on the sector's guidance list was in play that week. From where I sat, none of them felt like a mitigation. They showed up as calls I had to place to people who might not answer, over channels I wasn’t sure of, before I knew enough to say the word incident out loud. The guidance is written for the moment after the declaration. The hard part was the eleven hours before it.

Related reading:

• Water Watch Center Highlights Water Utility Cyber Response Gap

• Why CI Security Guidance Keeps Failing Small Utility Operators

• Assume Compromise, Then What? Moving Past the Slogan to Operational Requirements

• What CI Fortify Asks Utilities to Spend — And the Capability It Never Names

• When the Phones Go Down: The Coordination Gap Nobody Plans For