While tinkering with a TMG issue, a TMG server stopped working as expected. The symptoms I observed were when rebooting the TMG server I had to wait TWENTY minutes of looking at Applying Computer Settings screen before I could log in, only to find out that four out of the five TMG services hadn’t started.

The event logs had these errors:

- The following service is taking more than 15 minutes to start and may have stopped responding: Microsoft Forefront TMG Control

- The following service is taking more than 15 minutes to start and may have stopped responding: SQL Server Reporting Services (ISARS)

- The Microsoft Forefront TMG Control service terminated with service-specific error The wait operation timed out

- The SQL Server Reporting Services (ISARS) service hung on starting.

- The Microsoft Forefront TMG Firewall service depends on the Microsoft Forefront TMG Control service which failed to start

- The Microsoft Forefront TMG Managed Control service depends on the Microsoft Forefront TMG Control service which failed to start

- The Microsoft Forefront TMG Job Scheduler service depends on the Microsoft Forefront TMG Control service which failed to start

This meant the TMG wasn’t a firewall anymore and need all the TMG services to be started to be useful and work correctly.

This was rather annoying, but since I was on the phone to Microsoft on another TMG call, they identified this as a known problem and offered this fix; At the administrative command prompt type:

sc config isactrl depend= RasMan/SSTPSVC/FwEng/ISASTG/bfe/mpssvc/HTTP

- then reboot the server.

The server took its normal two minutes to get to the log in screen and the TMG services were running normally. Marvellous.

Well that was all well and good, but I like to know what this command meant and did.

The sc command invokes the Service Controller and the config part denotes which service (isactrl) to work with but it’s the depend= which brings the magic. Typing sc qc isactrl on a working TMG show these services (RasMan, SSTPSVC, FwEng, ISASTG, bfe, mpssvc and HTTP) are dependencies of the ISACTRL service and that what the depend= means – dependencies that have to be running first. So running this command forces the TMG to waiting for certain services to start, in this case RasMan SSTPSVC FwEng ISASTG bfe mpssvc HTTP before starting the TMG Control service.

Footnote

After adding another patch to the TMG you have to run SC the command again as it seems to reset the dependancies.

The great thing about any new or updated firewall is the coders understand the protocols and processes that little bit more. This means the stuff that worked on the old version might now not work on the new one. This is very much the case on moving from ISA to TMG – especially for FTP.

This has created  issues with Active FTP and FTP over HTTP which is why this piece is now here to help others avoid similar learning pains.

Active FTP

TMG blocks Active FTP by default, which is a good thing as Active FTP is bad.  Well, RFC 959 doesn’t actually define Active FTP as bad,  but the way it works mean the client FTP firewall has to open up a large number of ports and keep track of them inbound from the FTP server. So using Passive FTP means  it’s good for the TMG admin with the FTP clients and possibly very painful to the admin hosting  the FTP site, but we’re all about defense here. Nice explanation here of the two FTP transfers here

The only problem is that there’s a list of Ms products that use Active FTP as the client by default. The most annoying of which is the built in FTP.exe client on every windows machine. This is extremely painful as there is no way to force this to run in passive mode (Don’t get me started on the quote pasv option in Ms’ FTP.exe debacle – why tell the server to use passive FTP and then, as the client, NOT be able to use passive FTP in the first place!)

SQL Server Integration Service (SSIS) and BizTalk also use Active FTP by default but are easy to flip over to passive FTP.

For those zany folks that have to FTP from the command line I really like ncftp. NCftp client is a free piece of software ported from the *nix world, it has a wonderful .msi format that is easy to deploy to Windows systems. Running commands from a .bat file is as easy as this:

ncftpput -a -u username -p password ftp.site.com . c:\downloads\test.txt

The . (period) after ftp.site.com represents the current directory on the FTP server, in this case the default directory for the username used to log in with.

Uploads local file c:\downloads\test.txt  to the current FTP directory on ftp.site.com after login

ncftpget a -u username -p password ftp.site.com . “D:\Server\somewhere\weird folder\” topsecret.*

Downloads remote files topsecret.* from  ftp.site.com after login  to “D:\Server\somewhere\weird folder\”  

The quotes are only used as a space character is used in the folder path. The . (period) after ftp.site.com represents the current directory on the FTP server, in this case the default directory for the username used to log in with.

Plenty of documentation so if you decide to use it, drop some coins in via their paypal donate  button.

Can’t get away from Active FTP?

For those stuck in the hell of using Active FTP for a client to an Active FTP site Yuri Diogenes has written a great piece on how to use the application filter to allow Active FTP here so I won’t repeat his instructions.

FTP over HTTP

Back to FTP over HTTP in TMG. For those wondering what this is open up your web browser and type in ftp://ftp.microsoft.com It presents a web view of the ftp site – FTP over HTTP in its full glory.

In ISA have a rule allowing FTP and HTTP meant everything worked,  but in TMG they add in the new protocol FTP over HTTP. If you have staff using web browsers to download FTP through the browser then add in or update your FTP allow rule to include the protocol FTP over HTTP.  That should keep everyone happy.

Why isn’t my FTP over HTTP rule working

- One firewall too many

You’ve changed your TMG rule but you still get an Error Code 10061 web page from TMG when using FTP over HTTP for any site. For example, in the browser goes ftp://ftp.microsoft.com but this appears:

Technical Information (for support personnel)

• Error Code 10061: Connection refused
• Background: The server you are attempting to access has refused the connection with the gateway. This usually results from trying to connect to a service that is inactive on the server.
• Date: 6/6/666 1:23:45 PM [GMT]
• Server: IloveTMG.honest.com
• Source: Remote server

Now if you have:

A)      use multiple IP addresses on your external interface

B)      Have set the internal to External Network Rule to use a certain NAT IP address

C)      Have a edge firewall/router blocking traffic from certain IP addresses*

You’re problem is the TMG misbehaving and edge firewall device doing its job.

Here’s an example of what the problem is:

I have 192.168.1.66 defined as the IP address the outside world will see from the TMG, despite have twenty other IP addresses on the same external interface. This is defined in the Network Rule tab of the Networking section of the TMG interface.

With a network sniffer, such as Netmon or Wireshark, installed on the TMG server(see below) and listening on the external interface for traffic going to the internet, the TMG uses 192.168.1.66, as it supposed to.

Now connect to ftp://ftp.microsoft.com from a browser and it uses a different IP address (192.168.1.59 in this example)

Feel free to rush to your test lab, try this out for yourself and see it with your own eyes. I’ll wait until you’re back.

But why? WHY?

TMG uses  the IP address chosen by Window 2008 TCP/IP stack to pushes out FTP over HTTP rather than the IP address specified in the Network Rule. So it’s Windows 2008 TC/IP stack that’s to blame, rather than poor old TMG, although TMG should really be in full control, so bad TMG! There’s a number of KB’s and articles explaining how Windows 2008 deals with multiple IP addresses.

So what’s the fix?

Easy, but crap security option is to allow FTP out on that IP address (192.168.1.59 in this example) on the external firewall.

I have a request in with Microsoft’s TMG team to explain and hopefully fix this odd behaviour, so when I get a response I’ll update this post

*Option C is pretty common to have a hardware appliance (Cisco, Checkpoint , F5, Sonicwall, etc.) sitting on the edge of the network between the TMG and the raw internet pipe (router, modem or USB device uplink to the ISP) I like have a box in front of the TMG as it stops the random junk traffic from hitting the TMG and provides another layer of defense.

After doing some normal patching updates on the IAG and client machines, I suddenly had the problem when trying to connect to a Java based application. The SSL wrapper screen appeared but after a minute an error would appear. The app wasn’t working. This isn’t good as

The fix turnd out to be quick and easy:

In the IAG  configuration app, in the URL filter change InternalSite_Rule28 to Ignore and replace InternalSite_Rule29 URL to /internalsite/com/whale/sslvpnclient/whalesslvpnclient/class.class

Getting to this was a hour of head scratch, searching and playing. This is my journey to that two second fix.

I fired up the IAG Web monitor and noticed these errors:

Severity     ID       Type
Warning   55     Parameters not Allowed with URL Security portal (S)

Request failed, URL is not allowed to contain parameters.

Trunk: portal; Secure=1;

Application Name: Whale Internal Site; Application Type: InternalSite; Source IP: x.x.x.x; Method: GET; URL: /InternalSite/applet/sslvpnclient.jar?version-id=3.7.0.14.

Severity     ID       Type

Warning     67     URL Path not Allowed Security csrportal (S) Request failed, the URL contains an illegal path.

Trunk: portal; Secure=1;

Application Name: Whale Internal Site; Application Type: InternalSite; Rule: Default rule; Source IP: x.x.x.x; Method: GET; URL: /InternalSite/com/whale/sslvpnclient/whalesslvpnclient/class.class.

I knew I  had not changed on the rules or configuration.

Clicking on the first error of ID 67  popped up this:

Warning #67: URL Path not Allowed

Symptoms

A remote user requests a page. The request is denied, and the following message is displayed in the browser window: “You have attempted to access a restricted URL. The URL you are trying to access contains an illegal path.”

Cause

The path of the requested URL was rejected by the URL Inspection engine.

Resolution

Take the following steps in the Configuration program:

1. Open the Advanced Trunk Configuration window, and select the URL Set tab.

2. Do one of the following, depending on the rule that caused the failure, as specified in the “Description” filed of the message:

• If the rule that caused the failure is “Default rule”, use the URL List to add a new rule, or edit one of the existing rules, so that the requested URL is allowed.

• If the failure was caused by an existing rule, and the name of the rule is specified in the message’s “Description” field, access the rule in the URL List. In the “URL” column, edit the path of the URL.

Cracking open the IAG configuration tool and searching the URL List I  found InternalSite_Rule29 was very slightly different to the one in the failed error. I swapped it from

/InternalSite/com/whale/sslvpnclient/whalesslvpnclient.class

to

/internalsite/com/whale/sslvpnclient/whalesslvpnclient/class.class

Saved the configuration and tried the Java app again. Still failed.

After a bit of head scratching I found this post from the excellent www.forefrontsecurity.org

InternalSite_Rule28 (/internalsite/applet/(sslvpnclient|detectjava|microsoftclient|oesislocal|runtimeelevator|agent_win_helper|agent_mac_helper|agent_lin_helper)\.jar)
changed Parameters value Reject to: Ignore

Basically this stops the checking on the detection agents and allows the Java applet to do it job.

Another Hum Ho moment.

The folks over at the Technet magazine have publish this little gem on OCS and ISA depolyment

Worth a read if you have OCS and need the outside world to connect up to it

http://technet.microsoft.com/en-us/magazine/dd440949.aspx

We were splitting two web sites to two new servers from the existing one. The sites all worked and resolved internally, so created a new web publishing rules on the ISA for the relocated sites and their host header names.

When attempting to access the web site number two the web page displayed:

Bad Request (Invalid Hostname) – 400 Error

The web sites still worked internally and the ISA could resolve the server host headers and browse the sites. The rule looked good, pointed to the right location had the right ports open.

Turned on logging and watched for access to the site. No hits. Muck around checking event logs, restarting the firewall services and tweaking the rule. No joy.

Got a pencil and paper out and went through my deployment notes, ticking off each step. Got the web listener for site two and immediately found the problem. I had done a copy and paste of the rule and web listener and hadn’t changed the IP address of the second web listener to it new address. It was still using the first listener IP.

Added the correct IP address and everything worked as expected.

Meep

Like any normal day in the office, at some point I’ll get blamed for something not working.

More specifically, one of the security systems I manage gets blamed, but I cop the flack for it. Moaning at an inanimate firewall doesn’t elicit the sheer pleasure of ranting a human being, or me, it appears.

Anyway, over the weekend a financial web site had stopped working and they wanted it to work. It was throwing up authentication errors in a Java Applet screen they never seen before. The Web company’s helpdesk said nothing had changed, so it had to be our problem, or my problem, as normal. Since nothing changed on the network, client machines or firewall rule sets (gotta love change management) and I had a screen shot of the error with times of the problems, I called up the ISA logs, and filtered on the url of the site.

The screen shots filled me with apprehension, as the web site had a big ugly Java error and the pages were .jhtml and it was all running over SSL. After a chat with the staff member, he said the site looked like it had an update.

I’ve had problems before with Java programmers doing dubious things over http and the ISA correctly dropping the traffic, so wasn’t looking forward to getting in to a fight with a big financial web company over coding.

The ISA filter displayed a whole heap of these errors when connecting to the site:

Status: 995 The I/O operation has been aborted because of either a thread exit or an application request

Now this doesn’t tell you much, so after a quick bit of browsing the web I found a reply from Jim Harrison to someone with similar issues

“This is expected even for normal termination of SSL Tunnel traffic.

ISA can’t follow the HTTP conversation within the SSL session and so the session closure is always a surprise.

It does not indicate an error in ISA."

I trust Jim’s advice implicitly, and was sure it was the lovely web company’s fault, but his reply didn’t help nail what was wrong and SSL won’t let me analyse the traffic.

Help came from an unexpected source when the staffer mention he could access the web site at home, after taking a very long time to load to first time.

Did anything else happen while you accessed the site, like some piece of software update? I enquired.

It appears that his machine downloaded the latest version of Java first. Hmm, to the test lab machines!

I fired up a test machine, broke company standard build policy, ripped out the current package version of Java and installed the latest and greatest straight from the web site. This promptly broke the machine. I grabbed the next test machine and attempted to update Java. That broke it too. The third machine, a totally non-standard machine, installed Java without issues and could “magically” accessed the site. No need to touch the ISA rules.

Grinning like a Cheshire cat I promptly handed over the mess of updating everyones Java client to the packaging team with a couple of notes on how it had destroyed two perfectly good machines for no apparent reason.

Take away: if you see status message 995 being logged on ISA, a web app stops working and the site is Java based, then check the Java client and ask what version you should be using to access the site first.

Put money on it that the third party will say “Install at minimum version X”, which won’t be the version you’re running.