WebServer Admin

Friday, 13 June 2014

Web logs and historical traffic analysis may give you insight into the kind of traffic you can expect over time, but how can you tell what your infrastructure can really handle? To find out, you need a tool that can push your infrastructure and its application stack to the breaking point. One such tool is the free Web Capacity Analysis Tool (WCAT), a lightweight load-generation utility that plays back scripted HTTP requests against a Web server (or load-balanced server farm) while gathering performance statistics for later analysis. The latest version (6.3.1 as of this writing) is available for both x86 and x64 Windows systems(x86:iis.net/downloads/1466/ItemPermaLink.ashx; x64: iis.net/downloads/1467/ItemPermaLink.ashx) and has support for IPv6 networks.

I simulate this with my Webserver and found the below result.

Web Capacity Analysis Tool – WCAT :

Purpose:

Web Capacity Analysis Tool (WCAT) is a lightweight HTTP load generation tool primarily designed to measure the performance of a web server within a controlled environment. WCAT can simulate thousands of concurrent users making requests to a single web site or multiple web sites. The WCAT engine uses a simple script to define the set of HTTP requests to be played back to the web server. Extensibility is provided through plug-in DLLs and a standard, simple API.

HTTP.sys Cache:

Microsoft introduced Kernel-mode caching in IIS 6.0.This feature elimantes the need for accessing User-Mode cache in many cases.The HTTP.sys cache helps increase server performance and reduces the disk cost.

You can config frequentHitTimePeriod and frequentHitThreshold in the serverRunTime applicationHost.config section.Here are the default serverRunTime settings in the \Windows\System32\inetsrv\config\schema folder.

</attribute>

</sectionSchema>

A request is cached of more than the number of frequentHitThreshold request for a cacheable URL arrives within the frequentHitTimePeriod settings.

The examples shows how to setup Web Capacity Analysis Tool –WCAT and run the WCAT controller,WCAT Client, and setup output caching polices in IIS manager.

Download the WCAT tool and install.
Create a folder named C:\LoadTest to hold the configuration files.
Create a file called Default.aspx in C:\LoadTest. Type <% =DateTime.Now() %> and save Default.aspx.The file will be used for load testing.
Create a new file called script.cfg in C:\LoadTest and Type the following text:

NEW TRANSACTION

classId = 1

NEW REQUEST HTTP

Verb = “GET”

URL = http://localhost/Default.aspx

5. Create a file called distribution.cfg and type the following text inside the file

1 100

6. Create a file called config.cfg and type the following text inside the file:

Warmuptime 5s

Duration 30s

cooldownTime 5s

NumClientMachines 1

NumClientThreads

Next, After you have configured the LoadTest folder and supporting WCAT files, you can enable the WCAT controller.

Open a command prompt and type the following syntax:

Cd \LoadTest

%Program Files%\ IIS Resources\WCAT Controller\wcctl” –c config.cfg –s script.cfg –d distribution.cfg –a localhost

After enabling the WCAT controller, you can run the WCAT Client to performance tests.

Open another command prompt window to start the WCAT client.

%Program Files%\ IIS Resources\WCAT Controller\wcclient.exe” localhost

The first test has no output cache policy enabled.You can set the output caching policy in IIS Manager. Below screenshot shows no output cache policy enabled.

The second test enables User-mode policy only.The test using user-mode policy assumes you are using the file notifications option displayed in the below screenshot also shows how to enable a user-mode cache policy,

No Output cached policy enabled:

User -Mode cache policy :

As you can see the result in below, After running a test with User-mode caching enabled, the results are 656 requests per second, which is a 13 percent increase over no cache policy enabled.

For the third test, disable the user-mode caching policy and enable Kernal-mode caching.The Kernal-mode caching.The kernel-mode caching test assumes you are using the file notification option displayed in below screenshot shows how to configure a kernel-mode caching policy.

Kernel-mode cache Policy

After running the test with kernel-mode caching enabled, the results are 946 per second, which is 60 percent more than if you had no cache policy.Below results from the three permonance tests.The results can vary depending on what type of hardware you are using.

Test Results :

Request per second	Cache Level
575	No cache enabled
656	User-mode caching only
946	Kernal-mode caching only

Web Capacity Analysis Tool

The IIS 6.0 Resource Kit Tools include WCAT 5.2. To download the IIS 6.0 Resource Kit Tools, visit the following Microsoft Web site:
http://www.microsoft.com/downloads/details.aspx?FamilyID=56fc92ee-a71a-4c73-b628-ade629c89499&DisplayLang=en
The IIS 7.0 Web Capacity Analysis tool can be downloaded from the following Microsoft Web site:
http://www.iis.net/downloads/default.aspx?tabid=34&i=1466&g=6

How to Trace IIS website hangs/Slowness Issue :

Hangs are fairly common for production applications, and can be incredibly frustrating to troubleshoot. The main reason for this are:

· They may be happening only sometimes and can be hard to catch.

· They can be caused by complex and interrelated factors that can be difficult to isolate.

In this article show you how we can systematically isolate and diagnose most hangs in production

STEP 1: Is it really a hang?

An IIS website hangs whenever the it appears to stop serving incoming requests, with requests either taking a very long time or timing out. It's generally caused by all available application threads becoming blocked, causing subsequent requests to get queued (or sometimes by the number of active requests exceeding configured concurrency limits).

It’s important to differentiate the following kinds of hangs:

Full hang. All requests to your application are very slow or time out. Symptoms include detectable request queuing and sometimes 503 Service Unavailable errors when queue limits are reached.

NOTE: Most hangs do not involve high CPU, and are often called "low CPU hangs". Also, most of the time, high CPU does not itself causes a hang. In rare cases, you may also get a "high CPU hang", which we don't cover here.

1. Rolling hang. Most requests are slow, but eventually load. This usually occurs before a full hang develops, but may also represent a stable state for an application that is overloaded.

2. Slow requests. Only specific URLs in your application are slow. This is not generally a true hang, but rather just a performance problem with a specific part of your application.

Here are 3 "reasonable" early detection signs:

1. "Http Service Request Queues\MaxQueueItemAge" performance counter increasing. This means IIS is falling behind in request processing, so all incoming requests are waiting at least this long to begin getting processed.

2. "Http Service Request Queues\ArrivalRate" counter exceeds the"W3WP_W3SVC\Requests / sec" counter for the application pool's worker process over a period of time. This basically implies that more requests are coming into the system than are being processed, and this always eventually results in queueing.

3. And the best way to detect a hang is: snapshotting currently executing requests. If the number of currently executing requests is growing, this can reliably tell you that requests are piling up ... which will always lead to higher latencies and request queueing.

Most importantly, this can also tell you which URLs are causing the hang, and which requests are queued.

You can view all currently executing requests in InetMgr, by opening the server node, going to Worker Processes, and picking your application pool's worker process:

You can also automate this by using the AppCmd command line tool:

%windir%\system32\inetsrv\appcmd list requests /elapsed:10000

This will show you which requests are executing, optionally longer than theelapsed filter you specified. I recommend an elapsed filter of at least 5 seconds or longer.

If you see multiple requests that are taking a long time to execute AND you are seeing more and more requests begin to accumulate, you likely have a hang. If you DO NOT see requests accumulating, its likely that you have slow requests to some parts of your application, but you do not have a hang.

Detecting a hang reliably is suprisingly difficult. While you can almost always tell when you have a hang by requesting your website externally, detecting it internally can be surprisingly hard.

There are many possible places where a hang can happen, and many possible signs of hangs. Most of these signs are unreliable on their own (e.g. ASP .NET queueing counters), and the reliable ones (executing requests, thread snapshots) are prohibitively expensive to monitor all the time. With LeanSentry, we solved this problem by using progressive hang detection, which starts out with lightweight monitoring of more than a dozen different performance counters ... and then confirms a likely hang with executing request snapshots and the debugger.

STEP 2: Diagnose the hang

Once you confirm the hang, the next step is to determine where its taking place.

It's not IIS (but check it anyway).

IIS hangs happen when all available IIS threads are blocked, causing IIS to stop dequeueing additional requests. This is rare these days, because IIS request threads almost never block. Instead, IIS hands off request processing to an ASP .NET, Classic ASP, or FastCGI application, freeing up its threads to dequeue more requests.

To quickly eliminate IIS as the source of the hang, check:

· "Http Service Request Queues\CurrentQueueSize" counter. If its 0, IIS is having no problems dequeueing requests.

· "W3WP_W3SVC\Active Threads" counter. This will almost always be 0, or 1 because IIS threads almost never block. If its significantly higher, you likely have IIS thread blockage due to a custom module or because you explicitly configured ASP .NET to run on IIS threads. Consider increasing your MaxPoolThreads registry key.

Diagnose the hang.

Snapshot the currently executing requests to identify where blockage is taking place.

REQUEST "7000000780000548" (url:GET /test.aspx, time:30465 msec, client:localhost, stage:ExecuteRequestHandler, module:ManagedPipelineHandler)

REQUEST "f200000280000777" (url:GET /test.aspx, time:29071 msec, client:localhost, stage:ExecuteRequestHandler, module:ManagedPipelineHandler)

...

REQUEST "6f00000780000567" (url:GET /, time:1279 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)

REQUEST "7500020080000648" (url:GET /login, time:764 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)

You can use the resulting list of executing requests to learn A LOT about whats happening, including which URL is causing the blockage, and which requests are queued.

Expert tip #1: identifying requests causing the hang. You can identify which requests are the ones causing the hang because they will be at the front of the list, taking the longest time to execute. They will generally all be stuck in the same module and stage, and often the same URL.

If the hang is being caused by a specific ASP .NET controller or page, the module will say "IsapiModule" (Classic mode) or"ManagedPipelineHandler" (Integrated mode), and the stage will say"ExecuteRequestHandler". The URL should then point to the page/controller responsible.

Expert tip #2: Identifying queued requests. See the block of requests at the bottom of the list? These are the queued requests!

In Integrated mode, these will all have the module/stage corresponding to the first ASP .NET module in the pipeline. This will generally be "Windows Authentication" in "AuthenticateRequest" or sometimes "Session" in "AcquireRequestState".

STEP 3: What code is causing the hang? (for developers)

At this point, you've confirmed the hang, and determined where in your application its located (e.g. URL). The next and final step is for the developer to figure out what in the application code is causing the hang.

Are you that developer? Then, you know how hard it is to make this final leap, because most of the time hangs are very hard to reproduce in the test environment. Because of this, you'll likely need to analyze the hang in production while its still happening.

Here is how:

1. Make sure you have Windows Debugging Tools installed on the server (takes longer), or get ProcDump (faster).

Expert tip #3: It always pays to have these tools available on each production server ahead of time. Taking the dump approach is usually faster and poses less impact to your production process, letting you analyze it offline. However, taking a dump could be a problem if your process memory is many Gbs in size.

1. Identify the worker process for the application pool having the hang. The executing request list will show you the process id if you run it with the /xml switch.

2. Attach the debugger to the process, OR, snapshot a dump using procdump and load it in a debugger later.

3.  // attach debugger live (if you are fast)

4.          ntsd -p [PID]

5.          // or take a dump to attach later

6.          procdump -ma -w [PID] c:\dump.dmp

7.          ntsd -z c:\dump.dmp

8. Snapshot the thread stacks, and exit. Make sure to detach before closing the debugger, to avoid killing the process!

9.  .loadby sos clr

10.         .loadby sos mscorwks

11.         ~*e!clrstack

12.         .detach

13.         qq

14. The output will show you the code where each thread is currently executing. It will look like this:

15. OS Thread Id: 0x88b4 (7)

16.         RetAddr          Call Site

17.         000007fed5a43ec9 ASP.test_aspx.Page_Load(System.Object, System.EventArgs)

18.         000007fee5a50562 System.Web.UI.Control.OnLoad(System.EventArgs)

19.         000007fee5a4caec System.Web.UI.Control.LoadRecursive()

20.         000007fee5a4beb0 System.Web.UI.Page.ProcessRequest()

21.         000007ff001b0219 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)

22.         000007fee5a53357 ASP.test_aspx.ProcessRequest(System.Web.HttpContext)

23.         000007fee61fcc14 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

24. Wait 10-20 seconds, and do it again. If you are taking a dump, just take two dumps 10 seconds or so apart.

Alright. Once you have the two thread stack lists, your objective is to find thread ids that have the same stack in both snapshots. These stacks show the code that is blocking the threads, and thereby causing the hang.

NOTE: If you are only seeing a couple threads or no threads with the same stack, its likely because you either a) have a rolling hang where requests are taking a while but are still moving, or b) your application is asynchronous. If its async, debugging hangs is WAY harder because its nearly impossible to tell where requests are blocked without stacks. In this case, you need to implement custom application tracing across async boundaries to help you debug hangs. I will blog more about this in the near future.

Understanding IIS Bindings, Websites, Virtual Directories, and lastly Application Pools

Bindings: Did you say “Bindings?”

So you’ve been tasked with development of a new Web application to be hosted on IIS (any version)? The first thing on your mind is usually the design of the Website, how the application will interact with the middle-tier, and usually security. This is a great start in the design process. However, let’s not forget that often jumping into this level of design will mean that later on your going to make some other decisions a bit more tricky.

It starts with these questions:

Am I going to host everything in one IIS Website?
Will I use an “existing” Website like the Default Web Site or create my own?
Will some of the site require secure authentication using SSL?

The first thing that often happens with developers posed with these questions are they say these aren’t important but I quickly smile and say, “We’ll see”.

The primary reason that these questions are important are around the fact Websites are accessed by every client using bindings. The end-user of your Web application(s) don’t know they are using bindings because they are usually hidden behind a nice, pretty “Web address” using DNS. If you don’t have the answer of how many Websites your Web application will utilize then you are going to be struggling when you are upset that you are limited to “rules” governed by directories.

You see, Websites have something called Server Bindings which represent the underlying address, port, and potentially a host header that your Website is accessed using. Do you think that HR staff would be happy if their Website is accessed using the same bindings as your company’s intranet? I would venture to guess the answer is no.

Bindings 101:

A typical binding for Websites are in the form of IP:Port:HostHeader. For all versions of IIS that anyone reading this in 2010 care about (version 6.0 and higher), the default Web Site binding is set to *:80:* meaning that all requests to that server will land at that site.

Valid Bindings:

IP Field	Port Field	Host Header	Result
*	80	*	All requests to this server’s IP address will access this site.
*	81	*	All requests to this server’s IP address with :81 will access this site
192.168.1.1.	80	*	All requests to this server’s IP address will access this site*
*	80	www.microsoft.com	All requests to this URL will access this site
*	80	microsoft.com	All requests to this URL will access this site

For option where you utilize IP address as the “unique” point for access, you will need to disable HTTP.sys default behavior of listening on all IP addresses configured on your server. For example, if you have 192.168.1.1 and 192.167.1.2 configured as IP addresses on the same server the default behavior “out of the box” is to listen on port 80 no matter if you do the binding in the IIS Manager.

To change this behavior, you will need to configure HTTP.sys’s IPListenList (future blog I guess as there is no MS documentation on the topic) to only listen on a specific address. This is done via the registry or NetSH depending on what you are most comfortable with.

Figure 1: Default setting for IPListen (blank equals *:80:*)

In short, if you plan to utilize a Website then know what your bindings will be and where your application will live in production. If a shared server, you can bet you will need a Host Header or a unique IP address so think ahead and get ‘er going.

Websites versus Application Pools

There are so many reasons that Websites & Application Pools are confused that I don’t have enough time to do a post on it. I’m not going to try and solve the debate here, but instead, I’m going to try and educate you on what the fundamental difference between the two are. In discussions with IT Pro’s & Developers, rarely will you have any of them “admit” they know what each is and when to utilize one or the other but my guess is that over 70% of them don’t know.

Thus, I hope for readers out there who used their decision engine (nice plug, ay?) to find this reading will enjoy learning this topic and we can together reduce this 70% to a much lower number…

Websites: Container of physical and virtual directories

It really is simple. A website is nothing more than a container of physical and virtual directories that have a unique “Server Binding” for clients to access the content. The default container in IIS, for years, has been %systemdrive%\inetpub\wwwroot unless you are doing a unattended install in IIS 6.0 which allowed you to put the files where ever you choose.

Path + Server Binding = Website … It really is easy.

NOTE: Their is a serious obmission completely on purpose here. As you can see, Websites have nothing to do with memory, server processes, bitness, or performance. They simply are a path + binding.

When to choose a “Website”

With that understanding, you can now make an educated guess as to how to answer the question of whether you should create a new Website or use an existing one. However, I will make sure to share it in case you missed it - “You decide whether to create a new Website based on whether you would like to have a unique binding for your Website or if you want to use an existing one.”

The path isn’t important in this equation as I can create a 1000 Websites all pointing to exactly the same path and there is absolutely no problems with doing this (of course, why in the heck would you do this is a great question). The key decision here is that any physical or virtual directory will always use the bindings of the Website so ensure that you understand this.

When to choose directories?

If there is a website which is already running and utilizing a binding that you would prefer to use then you should select this option. This allows you to utilize the resources of the parent site, if interested, as the server (e.g. IIS) will handle any requests over the same connection(s). For example, any physical or virtual directory in the IIS path is still considered “/” to the server as it builds out the URI because the bindings are already mapped at the site level. This means that URLs can be re-written to go various different places within the folder hiearchy over the the same connection since the binding is the “same”…

If you choose to put your Web application in its own Website then you will have to use the HTTP 302 redirection capability (exposed via Server.Transfer or other methods) to push the request elsewhere.

So, as you can see, thinking ahead of time about whether you are building a Website for your application or whether it is a child directory (physical or virtual) is an important piece of information to have locked early, early on!

Application Pools: Container of applications

The very nature of application pools is to do the obvious, contain a single or multiple applications. The introduction of application pools in IIS 6.0 caused some head scratching but in today’s world where IIS 6.0 is very engrain in enterprises and the Web leads to less scratching. However, again, development teams often make mistakes by not “thinking” about application pools and there impact on their new applications they are building. Hence the reason we will chat about this some more today…

First Concept… Windows Process = Application Pools *not* Windows Process = Website

Second Concept… Process Management = Application Pools *not* Process Management = Website

When to create a new Application?

By default, IIS 6.0 or IIS 7.0 must have a single application to run. If the root application (/) is deleted or corrupted then IIS will fail, as in, not serve your application. Both products ship with a default application which is assigned to the Default App Pool. I should not this is only if no other Microsoft features have been installed and instead we have the basic Web server installed.

As you can see, there is also a Classic .NET AppPool but no applications are currently bound to it. In IIS 7.0, any managed code application can choose to utilize the Integrated Pipeline or to use the classic ASP.NET pipeline which is present in IIS 6.0.

By default, you as a developer of a Web application can choose to simply inherit the settings of the parent Application Pool (/) and choose to not create your own. This is absolutely fine. So you might ask, what do I get from choosing this route? I’m glad you asked because it is important to know that you get all the settings of the parent application pool which in this case is the DefaultAppPool.

These settings include the following:

Setting	Purpose
Recycling Settings	How often the App will be recycled such as by time intervals, memory usage, etc.
Process Security	Who is the identity that the W3WP process will run as
Pipeline Type (IIS 7.0 Only)	Whether to use the integrated pipeline, classic pipeline, or no Managed code at all
Bitness	Whether the process runs in native 64-bit or uses a 32-process (64-bit OS only)

As you can see, you need make some important decisions early on or you are going to change a lot during the development process.

When to create a new Application Pool?

Well, it sounds like I’m best to create a new application pool for all my Web applications. I would say you’ve been suckered and convince that this is the best without all the facts. The fact is that creating an application pool includes understanding better your strategies for security such as do you run Network Service, a Domain Service Account, etc. that starts to complicate things very quickly. The one thing that many manage code developers often love to take advantage of is the caching capabilities of processes and manage code. Each time you create a application, bind it to its own unique application pool then you are limiting your ability to share cache with other .NET applications running on the same box. For example, if you have theMicrosoft Enterprise Library in use all throughout your Web applications then you can often utilize caching to improve performance. As soon as you break these out into different process boundaries (e.g. App Pools) then you no longer have that benefit.

There are a number of these types of examples listed above that drives the question – Do I use my own application pool or do use another one already running? I’m happy to get posed a question via comments or email regarding this topic and see what your situation is and make my suggestion :)

Nonetheless, be careful in your planning when utilizing your own Application Pools and share resources where possible is my guidance. There are absolutely situations where one might choose to always go hard line with creation of app pools for every new Web development project. I just caution you and say, “Not so fast my friend… “

Summary

In today’s post, I went for a “blast from the past” theme as to just feel the power between… well let’s leave it there. I spent several years focusing on our IIS product and a great deal more focusing on helping customers struggle with the product with our lest than stellar performance of training on IIS. You now have learn.iis.net so for that I say woohoo! So today was fun…

To summarize, I hope that I gave you non-IIS geeks a bit of understanding in one of the fundamental foundations of ‘Getting Started with IIS’ concepts. I often see many folks fail to understand when to use a Website, or a virtual directory and my aim with this post is to give your brain a bit of a quiz to help you figure out which “way” you want to go when you are developing applications based on IIS.

In yesterday’s meeting, that is all I did was give my feature team a lot to think about and then I will let them choose what avenue they would like to go…