Security for a day

Had the joy of getting a bunch of old ISA servers to SP3. Seems a pretty simple task until I looked in to it a bit closer. The server had over 90 rules and a lot of custom tweaks made with registry hacks.
Anyhow, did my normal roll back tricks of backups, cloning and breaking disk mirrors then started with the updates. Two of the server threw up this error: msfpcsnp.dll failed to register.

No-go SP3 update!

Was a bit confused and the logs held no useful information, then as a side I though I’d check the uptime of these boxes.

The servers hadn’t been rebooted in over 458 days and they were still performing fine!

A quick reboot of the two machines and the service pack installed with out further issues.

Next problem the Application log started to fill up with:

Event Type:     Warning
Event Source:   Microsoft ISA Server Web Proxy
Event Category: None
Event ID:       14141
Date:           14/09/2008
Time:           10:28:10 AM
User:           N/A
Computer:       WNDMZIA09
Description:
ISA Server detected a proxy chain loop. There is a problem with the configuration of the ISA Server routing policy.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
There’s a helpful piece from MS on solving this normally here but I wanted a fast, simply way to find the root of the problem.  knew that the network options and routing had been set correctly and did a check that nothing else was attempting to hog either port 80 or 8080 with the ol’ netstat command:
c:\>netstat -aon | find “:8080″
producing: TCP    0.0.0.0:8080 0.0.0.0:0  LISTENING       452

c:\>tasklist /svc | find “452″
resulting in: wspsrv.exe                     452 fwsrv (which is the ISA firewall service- that good )

So went in to the ISA monitoring tab, setting the logging to find HTTP status Code equal 12206.

This shows all the error messages for: 12206 The ISA Server detected a proxy chain loop. There is a problem with the configuration of the ISA Server routing policy. Please contact your server administrator.

It showed me the rule and client details to actually see where and what the problem was. In this case some client machines were trying to contact a decommissioned server on a decommissioned ip address space. The ISA’s were the default path for all unknown networks, so the traffic would end up there. The ISA didn’t know what to do with the trafic and thus generated the 14141 errors in the Apps log. A quick deny rule blocking the  decommissioned ip address space sorted out the clients, and then lead another to a misconfigured rule, which was easily fixed.