Frequently Asked Questions

How do I Get Started?

Well, you can actually start in several places. A test itself defines a visitor's profile along with the experience they will have at your site. So, if you start at the Test tab of the Test Center, you can go straight to Create a New Test. When prompted for Visitor Experience, you can select "A New Visitor Experience." This will take you straight into the place where you define new Visitor Experiences.

If you are not quite ready to build your tests, you can start by building your Visitor Experiences. From the Visitor Experience tab of the Test Center, choose to Define a New Visitor Experience. After you're done, you'll see this Visitor Experience available later when you create your tests.

How do I create a test?

A StressMy.com test combines a visitor experience with a traffic pattern, and character definitions for those visitors. Before creating a test, you need to decide on the click path, traffic load, and visitor characteristics that will help you to meet your testing goals.

To define a test, go to the StressMy.com Test Center, and click the Tests icon. StressMy.com presents a list of your currently defined tests and a link to define a new test. When defining a test, you may use an existing visitor experience or record and edit a new visitor experience during the definition process.

Choose a traffic pattern that will assist you in acquiring the information you need. A traffic pattern that adds a given number of visitors at a specified time interval will allow you to determine the optimum number of concurrent users supported by your application. A constant load over a specified time interval (or a traffic pattern with one to two significant increases) will allow you to determine the performance characteristics of your site's application under a sustained (and potentially damaging) load.

Currently, the only visitor characteristic you may control is the definition of the visitor's returning visitor behavior. The returning visitors setting determines the percentage of simulated visitors behaving as if they were returning, or visiting the site for a second time. This deals directly with the caching of cookies and graphics. Specifying "0" (zero) in this form field results in all simulated visitors acting like new site visitors; conversely, specifying "100" makes all simulated visitors act as returning visitors.

When you have finished defining a test, StressMy.com returns you to the Test Center. You can then schedule your new test to run immediately or at a later time. To do this, simply click the Run link next to your test.

How can I upgrade my service? How can I use more than five simulated visitors in a test?

By utilizing the StressMy.com complimentary account, you can gain quick information about your site's performance with just one simulated visitor. This is important because it is establishing a baseline. StressMy.com testing engineers have even seen cases where a test load of two concurrent visitors was enough to uncover "threading" issues in the application. But, in most cases, you will need to generate a larger number of simulated visitors to fully understand your site's performance and uncover pesky bottlenecks.

To increase the number of available simulated visitors you may use for testing, and/or increase the number of available visitor traffic patterns, you will need to upgrade your account. You may upgrade your complimentary account at any time by clicking the My Account link. On the Account Management page, you will be presented with the option of changing your service level to Genesis, Dynamis, Quintus, Ultimus and Zenith, with maximum traffic of 100, 250, 500, 750, and 1000 simulated visitors respectively. After you select your desired level of service and click the Submit button you will see a confirmation page. From here, click the Proceed to Secure Payment Page button. This will transfer you to our secure credit card processing service. (Note: StressMy.com does not store your credit card information. In fact, credit card processing is done on an entirely separate system and your transactions with that system are protected by secure SSL links.) Once you have completed your interaction with the credit card processing system, you will see an order confirmation page. Follow the link on this page to complete the upgrade procedure. You may now begin building and running tests using the additional visitors.

How can I download your product?

One important feature of StressMy.com is that you are not required to download or purchase any software to run performance tests against your site's application. In short, there is NO software to download. You are able to access, maintain and use your account from any browser-equipped PC or workstation. Furthermore, you do not need to spend time and effort identifying and setting up hardware resources to generate a test load. StressMy.com provides the software, hardware and communication resources to perform real tests that reveal the performance of your application under load.

For sites requiring more local control over their testing or who require testing loads on intranets that are not accessible through the internet, Downright Software will install and customize the StressMy.com testing engine for a reasonable fee. Contact us at accounts@stressmy.com if you would like to discuss this installation option.

What prevents a person from using StressMy.com in a malicious manner? Can StressMy.com be used for a denial of service attack?

Several features have been built into StressMy.com to minimize the likelihood of it being used in a malicious manner. The nature of stress and performance testing is to generate enough traffic through a site to determine its performance as the load increases, and the site's application may recognize it as a denial of service (DOS) attack. If you are the site's developer, this may be a good way to test your site's vulnerability to DOS attacks and check its preventative mechanisms. It may also be a good test of the efficacy of your load balancing scheme, the capacity of your network, or the capacity of the channels between your web servers and database server or legacy host applications.

Appropriate testing with StressMy.com and malicious use can look very similar. The difference is in whether the test load was invited or not. To minimize the chances of malicious use, StressMy.com makes sure that it has valid identity information its members. Complimentary accounts are carefully limited to a maximum of five simulated visitors, far smaller than the number required to conduct a DOS attack on most sites. To gain access to a greater number of simulated visitors, a member must supply a valid credit card number. These are two levels of security that StressMy.com has set in place to assure identification, and hold abusers accountable.

Another feature that assists in the prevention of misuse is StressMy.com's honoring of a site's robots.txt file. Entries in a robots.txt file such as the following will prevent StressMy.com from generating traffic to a site.

           User-agent: StressMy
           Disallow: /
    

Furthermore, the StressMy.com Terms of Service specify that any account used in a malicious manner will be immediately suspended should a complaint be filed with the StressMy.com staff. After investigation, any account used in an malicious manner will be canceled (with no refund of fees) and the appropriate law enforcement agencies will be notified. The StressMy.com staff will cooperate with any law enforcement inquiries as to the inappropriate use of the StressMy.com product.

StressMy.com is a tool for the serious developer. Like any tool, it can be used to build and improve projects of beauty, grace, usefulness, and importance or it can be misused. The StressMy.com team will support the appreciate use of this tool and will aggressively work to prevent the misuse of it. Feel free to contact abuse@stessmy.com to report abuse or to make suggestions for ways we can further minimize chances of abuse.

May I test sites other than my own?

By accepting the StressMy.com Terms of Service, each member agrees to conduct testing only sites that they are authorized to test. An "Authorized Test" may be defined as such, "you, or the person using your account has the permission and the right to access and/or test that web site."

Unauthorized testing of a site could easily be interpreted as harassment or a Denial of Service attack. If a site operator complains to StressMy.com about a particular test, we will suspend the associated account until we can determine whether or not an authorized test has been conducted. In the event that your account was used to conduct an unauthorized test, (or is used in any abusive, harassing, or threatening manner), your account with StressMy.com will be immediately terminated with no refund of subscription fees. In this case, StressMy.com will cooperate with law enforcement officials investigating any improper use of StressMy.com.

StressMy.com recognizes restrictions placed in a site's robots.txt file, and you may therefore receive a message from StressMy.com refusing to build the visitor experience within that site.

What happens to my account after my subscription expires?

The StressMy.com team recognizes that testing the performance of your web application is not a one-time event. Sites change and evolve , requiring you to perform testing periodically. Archived testing results need to be available for comparison to identify time-based trends. For this reason, your StressMy.com Complimentary account does not expire, and all upgraded accounts will revert to the Complimentary service level at the end of the subscription period. All test results will remain available online for six months, and you will receive notification via e-mail at least one week prior to any test results being archived, to give you time to make a personal copy.

If your account has reverted to the Complimentary service level, you may upgrade simply by visiting the My Account section. This flexibility allows you to periodically re-test your site using the desired traffic pattern, when it is convenient for you . Your test results will remain available to you no matter what your service level may be.

I want to monitor my database server to see how it performs when my web site is operating under a heavy load. May I use StressMy.com to do this?

Modern web applications tend to be complex systems composed of many independent functional elements, ( web server, database server, application server, interface to legacy host applications, etc.). While the performance of each element can be optimized in isolation, the performance of the system as a whole can be difficult to predict. StressMy.com allows you to test your web application as a unit, sending a set of loads that allow you to exercise all the "moving parts" of your application.

An intended (and common) use of StressMy.com is to generate a sustained load on your application to reveal the functionality of its moving parts. Several traffic patterns offered by StressMy.com provide just the type of load required for this test, and the scheduling features of StressMy.com allow you to schedule the test with enough lead time (usually minutes) to prepare and begin monitoring the operational elements of your site.

One of StressMy.com's beta-testers used the site in this manner. StressMy.com has been informed that being able to generate a sustained testing load on his site and, by implication, on the database server supporting the site, allowed him to identify critical timing, deadlock, and capacity issues. He feels that the 24/7 availability of StressMy.com enabled him (and his geographically dispersed development team) to test, identify, fix and retest the database performance of the application within a very aggressive development timeframe.

My site is behind a firewall. Will StressMy.com work with it?

We built StressMy.com to help developers test public Internet sites. However, several of our customers have built Intranets to support their internal operations and would like to test those applications.

For StressMy.com to help you test a site located behind a firewall, that site must be visible on the "public" side of the firewall. Basically, if a site is publicly available, or the firewall can be configured to allow HTTP access by selected hosts, you may use StressMy.com to test the site.

Some organizations configure their firewalls to allow traffic to/from the local network when that traffic is part of the standard HTTP protocol. In that case, you may use StressMy.com to test internal sites with no changes to your firewall configuration. Some firewalls can be configured to allow HTTP traffic only from certain hosts. The StressMy.com support staff can supply you with a set of IP addresses that you can use for configuring your firewall. If you require assistance or information regarding the configuration of your firewall, please contact support@stressmy.com

If you are unable to make your site accessible to the outside world, StressMy,com can still handle your testing needs. Downright Software will configure, install, and support custom configurations of the StressMy.com system behind your firewall, giving your entire organization access to the powerful testing tools included in StressMy.com. For more information about this option, please contact us at accounts@stressmy.com.

How do I handle or generate dynamic content for my site?

StressMy.com provides Producers for helping the simulated visitors generate the information returned to your application in forms. Each form field can have its own Producer and there are a variety of Producers available to simulate different types of responses. By default, StressMy.com assigns a Constant Value Producer to each form field with the Constant Value set to the contents you specified in that form field when you recorded the Visitor Experience. There are Producers available which will return a value from a specified list, a random string based upon a supplied pattern, or the contents of a field as supplied by the web server.

This last Producer, the Current Value Producer is quite useful for applications that use hidden form fields to maintain state (or other transaction linking information) as a value in a hidden form field. In this case, simply repeating the value supplied in the field when the Visitor Experience was recorded will not allow StressMy.com to test your application. (Usually, that condition results in a lot of errors on your server.) However, by assigning a Current Value producer to the hidden form field, you will cause every simulated visitor generated by StressMy.com to return the unique state information your applications assigns in that linking form field.

The List and Random Value producers are useful for generating test responses that differ for each simulated Visitor. The Random Value producer is quite valuable for producing a range of user ids so you can test your application with simulated Visitors that take on different personas (as opposed to the single persona used when recording the Visitor Experience.) This will allow you to test applications that will not allow a user to have multiple concurrent logins or sessions. It also permits more realistic tests by having each new Visitor take on a new persona or identity.

These two producers are also useful for generating a range of values for form fields that are linked to database information. If the data supplied by a (simulated) Visitor is stored in a database or used as parameters for a query by your application, simply repeating the same input information for each Visitor can result in an unrealistically optimistic measurement of the work capacity of your application. (Databases have gotten real good at caching information that doesn't change and producing impressively good performance when retrieving and/or storing that information. And, if the form information is used as an index when storing information in your database, using the same value over and aver can result in an unrealistically pessimistic performance measurement as your database struggles to manage the same index being used for 100s of records.)

What is the difference between new and returning visitors?

When StressMy.com creates a simulated visitor, that visitor is assigned a role of either new or returning based on the mix parameter you supplied when defining the test. Once created, a visitor will not change its role and will continue to follow its assigned Visitor experience until the test completes. A visitor's new or returning role affects how that visitor handles cookies and graphics.

A new visitor will always download a graphics image found in a page sent from your application. During a single iteration, that graphic will be added to a graphics cache and will not be fetched again during that iteration. At the end of an iteration of a Visitor Experience, the visitor's graphics cache will be cleared out and the visitor begins another iteration of its Visitor Experience. The visitor will once again download graphics the first time they are encountered in its Visitor Experience.

A visitor assigned the returning role does not erase the graphics cache at the end of an iteration and thus downloads graphics images only the first time they are encountered during the entire test.

A visitor's cookie handling behavior is also determined by the visitor's new or returning role. During a visitor's first iteration of a Visitor Experience, cookies are stored in a cookie cache for that visitor. New visitors clear out their cache at the end of each iteration. Returning visitors keep their cache until the end of the entire test. Of course, any new and/or updated cookies are stored whenever they are received (by both new and returning visitors.)

How does StressMy.com come up with my page-reading times?

StressMy.com computes a minimum and maximum page reading time based on the total text content of the page being read. This suggested page-reading time is based on the assumption that more text on a page will increase the time a visitor will spend on that page. In the event of a graphics-only page (a link to a .gif, for instance), the page-reading time range is always the same: 2 to 10 seconds.

Since the suggested page-reading times do not take into consideration the number and visual complexity of embedded graphics images, a page with a lot of graphics may end up with a lower-than-expected read time. Similarly, simply counting the number of characters in a complex form may understate the amount of time and effort that is required of the visitor to complete that form. So, while StressMy.com make an attempt to guess a set of appropriate page reading times for your application, you, as the application developer (or tester) should review all suggested read times and adjust them to correspond with your "local knowledge" of the application.

Since I am ultimately responsible for setting page reading times, what are "normal" values?

If you have some traffic history and have access to access log analysis software, you may be able to extract exact information about how much time your visitors spend on different pages of your site. Or, you may be lucky enough to have specific user interface feedback from observations and/or prototypes. Usually though, you, as the test definer, get to make the decisions about how long a simulated visitor stays each page. You will need to juggle factors like the impact of visual design, the amount of text on a page, the complexity (and attractiveness) of the displayed graphical images, function of a page (placeholder, navigation, query definition, information response/display) and compute the times. Some rules of thumb from the StressMy.com staff:

• People comfortable with an application rarely spend more that a few seconds on pages associated with navigation.
• New visitors to a site will take longer to read and process a page than returning viewers. Adjust the read times to reflect the type of population at you a simulating.
• Visitors can take up to several minutes to complete forms that require information that is not on their desktop. Things like credit card numbers, detailed financial history information, detailed school history, etc. can dramatically increase the time it takes for a visitor to complete and submit a form. Adjust your form read-times to reflect the complexity of the form.
• Visitors can become quite adept at completing forms that they use a great deal. Adjust your form read-times to reflect the speed with which your visitors can fill out a familiar form.
• For "text heavy" pages, use a stopwatch to time how long it take you to read the information on a page. Assuming that, as the developer or tester, you have a great deal of familiarity with its contents, this time is a fairly good metric to use as the minimum page read time. You can then triple your measured read time as the maximum read time.

Why are page-reading times important?

Page reading times are a critical factor in your web based application's performance. Fundamentally, the longer a single visitor spends reading, completing or processing a page of information sent by your application before requesting another page (following a link, performing another query, adding another item to their shopping basket), the more concurrent users a given configuration can support.

So, it becomes important that StressMy.com users define appropriate minimum and maximum reading times for pages in a Visitor Experience in order to get a good estimation of the response time for their application at increasing levels of concurrent visitors. If reading times are set too low, then the performance measured by StressMy.com will underestimate the actual work capacity of your application. If think times are set too high, StressMy.com will provide an overly optimistic estimate of your application's capacity. You ability to interpret and rely on the information provided by StressMy.com is directly related to the accuracy of your reading time ranges.

How does StressMy.com handle cookies?

When you are creating a Visitor Experience, StressMy.com lets your favorite browser handle your cookies. When you are running a test, each simulated visitor detects and records cookies sent by your application to that visitor. During a particular visit, those cookies are returned to your application like they would be if the visitor was using a browser. (Their domain name and directory are checked and they are validated for expiration.)

When StressMy.com creates a visitor, it is assigned a role of either new or returning based on the mix parameter you supplied when defining the test. The cookie behavior of a visitor varies depending on its role. For new visitors, at the end of an iteration of a Visitor Experience all cookies received during that iteration are erased. Returning visitors act like new visitors during their first visit, but their cookie cache is not erased at the end of the first iteration and any cookies recorded during the first iteration are returned during the second and subsequent iterations.

Can StressMy.com test my site if it uses JavaScript?

Of course it can... with a few restrictions. When building a Visitor Experience, StressMy.com records the HTTP transactions between your browser and your application. In some cases, like when JavaScript is used to build links, your application can confuse the recording process. If you build a Visitor Experience and discover that some mouse clicks don't seem to have been recorded, you probably encountered this situation. (If you do encounter a situation where mouse clicks are not recorded, and it does not seem to be related to a JavaScript navigation issue, please contact StressMy.com support at support@stressmy.com. We'd like to look into the situation.)

When simulating Visitors to your site, StressMy.com is replaying the recorded transactions; it is not actually simulating a browser (with its embedded JavaScript engine.) So, any browser responses that require computation by your JavaScript will not be generated automatically. Usually, this is not a problem; you (as the application developer) know the "correct" response for a particular set of conditions and can "program" StressMy.com to send the appropriate form fields to application using StressMy.com's form field Producers. So, if you use JavaScript mainly to validate users' input, you will have no problem using StressMy.com. If you use JavaScript to compose dynamic responses, you may have to get a bit more creative. However, StressMy provides form field Producers that should give you the tools you need to handle all but the most complex situations.

How do I test my site if it manages application state?

Web developers use several design patterns to identify a visitor's application state. There are two common design patterns:

• Use cookies to store information
• Embed a hidden form field containing a state ID

A state ID stored in a cookie will be handled seamlessly with StressMy.com's built-in cookie support. Each simulated visitor will remember and report any cookie received during a visit.

A state ID embedded in a hidden form field is handled with StressMy.com's form support. When you build a visitor experience, StressMy.com identifies each hidden form field and creates a unique Producer for that field. The default Producer for hidden fields is the Current Value Producer. The Current Value Producer uses the exact value sent from the web server, each time the page is sent to the visitor (it does not use the value encountered when building the visitor experience). In other words, during a test run, StressMy.com identifies a piece of information in the pages sent by your application, remembers that information, and sends it back to your application in the next request from a simulated visitor. The behavior of the Current Value Producer allows your application to embed a (constantly changing) state ID in a hidden form field and use it to track site visitors.

While recording my visitor experience, my site seemed to respond slower than normal. Why?

While recording a visitor experience, your site may seem sluggish. This is because of the technology used by StressMy.com to record the information in your visitor experience. In order to make StressMy.com non-invasive, the development team implemented a strategy that requires NO installation of software on your workstation. StressMy.com accomplishes this by doing the following:

1. StressMy.com sends a page request through its central servers, where it is recorded in your Visitor Experience.
2. The page request is then sent to your web site, and the resulting web page is sent back to our central servers for processing.
3. Once the resulting page is completely assembled, it is sent to your browser.

The combined effect of these three steps may sometimes result in sluggish behavior while recording a Visitor Experience.

When running a test, the simulated visitors do not experience the same sluggishness because they are communicating directly with your web application.

How do I choose a traffic pattern?

The selection of a traffic pattern is an important consideration when planning a StressMy.com test. To select a traffic pattern, first consider the objectives of your test. Do you need to:

• Test all the working parts of your web-based application while under a constant load?
• Discover the maximum transaction rate of your application?
• Discover the load level at which response times become excessive?
• Discover the load level at which the application becomes unstable?

Based on our experience testing complex Internet applications, the development staff at StressMy.com has defined a set of traffic patterns for each service level. There are several traffic patterns to choose from, with constant or increasing levels of simulated visitors.

Fundamentally, the selection of a traffic pattern is a reflection of your testing goals. Select a traffic pattern that will exercise your Internet application in a way that gives you the information you need.

How do I schedule a test?

When you go to run a test from the Test Center, you are given the option of specifying a schedule date and time. If you specify a date (or time) in the future, StressMy.com will not attempt to run your test until the specified time. Since StressMy.com fills the schedule date and time fields with the current date/time by default, tests are usually scheduled to run as soon as they are submitted.

StressMy.com will run your test as close to its scheduled time as possible. The ability to run your test is governed by the availability of resources. A scheduled test will only run when there are enough available Simulated Visitors to support the test's traffic pattern. While StressMy.com has sufficient resources to support hundreds of concurrent simulated visitors, those visitors may already be assigned to other tests scheduled to run just before your test. Once there are sufficient available simulated Visitors to support your test, it will be run.

Scheduling a test to run at a later time can make testing much more convenient. If the site you want to test is in a production environment, you can schedule a test to run when it will have the least impact on &live& visitors to the site. (And you do not have to be online or even awake for the test to run. You will receive an e-mail notification that the results are ready when the test is completed.) Delaying the start of a test by 5 to 10 minutes is also a useful technique to give you enough time to set up any additional monitoring of your web server or database server before the test starts.

When using the StressMy.com scheduling feature, remember that all StressMy.com times are expressed in UTC (Universal Time Coordinated), AKA Greenwich time. You'll have to know the offset between your local time and UTC and then adjust the schedule time appropriately. The only time this gets tricky is when the offset causes a change in date.(If enough users provide feedback, we will add a user attribute to your profile that will allow you to define this offset and then convert all displayed and entered times to/from UTC.)

How can I see what tests have been scheduled to run?

StressMy.com lets you to schedule tests to run at any time. Your test will run at its scheduled time provided there the necessary resources. (Before executing a test, StressMy.com checks the availability of simulated visitors for that test. This resource allocation scheme minimizes the chances that the results of your test are affected by another member's test.) If a scheduled test cannot be executed at the specified timetime due to resource (or other) constraints, it will be executed as soon as resources become available.

To check the scheduled time for a test, simply visit the Results tab of the Test Center. There you will find a listing of all your tests scheduled, in progress, and completed. (Of course, you will only be able to view results of completed tests.) The test listing contains Schedule column, listing the scheduled start time for each test. The Start and Finish columns document the actual begin and end times. Note that all times are listed in UTC, so you will have to adjust for your local time zone.

Each of the time columns is sortable, with time-based columns sorting newest to oldest. It may be useful to place all of your scheduled tests at the top of the list. The Test Title column sorts alphabetically, then newest-to oldest schedule time within matching test titles.

Can I test with more than one Visitor Experience at the same time?

Yes. A common testing protocol is to investigate the performance of your application when several different types of visitors are all attempting to access the site concurrently. By defining a different Visitor Experience for each type of visitor and then using the scheduling feature of StressMy.com to request several tests concurrently (each using one of the Visitor Experiences), you can subject your application to a testing load that simulates the mix of users you have observed or experienced.

This approach not only allows you to test a mix of Visitor Experiences, it also allows you to adjust the new/returning visitor ratio for each type of Visitor Experience. For example, a Visitor Experience that simulates a prospective site visitor can be scheduled to simulate a mix of 100% new visitors, while the Visitor Experience that represents a subscriber can be scheduled to simulate a mix of 100% returning visitors.

Keep in mind that, just like when scheduling a single test, StressMy.com cannot guarantee that your tests will all run at exactly the same time. Tests are dependent on simulated Visitor availability.

I'm more interested in the total transaction rate that my application can support rather than the total number of concurrent visitors. Can I use StressMy.com to stress test my site with a focus on transaction rate?

Yes. To do this, build a visitor experience that reflects the transactions you need to measure or test. If you set the minimum and maximum reading times for each page to "0" (zero), successive steps in the visitor experience will be executed without latency. In this configuration, individual simulated visitors become sources of unremitting requests generating return traffic to your application as fast as the application can respond to those requests. As the total number of simulated visitors increases, the total number of requests per second (or minute, or hour) increases. (This is also true when tests are configured with non-zero reading times; however, the effect is more dramatic when reading times are set to zero.)

A fair estimate of the transaction rate supported by your application may be calculated by doing the following: multiply the number of completed visitor experiences by the number of pages in the visitor experience, and divide by the duration of the test interval. (Since testing intervals correspond with a particular load level, you will also have a transaction rate that corresponds to a given level of concurrent connections.).

Can I test my application if it uses frames?

Of course! StressMy.com has been carefully designed to support all common browser functions, including frames. Frames will be handled as you build a visitor experience. As it records, StressMy.com will download the appropriate pages that the simulated visitors are to view during the actual test.

How can I share my test results with my colleagues?

Now, why would you want to do that?

When StressMy.com has completed running a test, it prepares a report of the test results and then sends you an email notifying you that the results are ready for your review. That email contains a very long hyperlink to your report. You can share that test report with your colleagues simply by cutting and pasting the long URL into your e-mail directing your colleagues to review the results. When they get your e-mail, they simply click the hyperlink, or cut and paste it into their browser (depending on their mail client) and they are reviewing your results.

It is very important that you pass the entire URL. It contains not only a reference to identify your test results, but also a "magic code" that helps keep your results confidential. StressMy.com will not display your test results without both the id and its corresponding magic code.

How come my graphs show no data?

StressMy.com tries to apply some intelligence to the information it presents. One of the things it does is filters the results to remove data items that can confuse the analysis and interpretation of the results. So, StressMy.com removes any measurements for Visitor Experiences that run longer that a testing interval. (A testing interval is the period of time between planned increases in the level of simulated Visitors.) An empty graph can result from all your Visitor Experiences running longer than the specified testing interval.

So, take a look at the total reading time for your Visitor Experience. It's very possible that just the reading time alone is longer than the testing intervals on your selected traffic profile. Check the minimum possible time to complete an iteration of a Visitor Experience by adding up all the minimum think times. You may also want to check that the expected value of the think times (the minimum possible think time plus half the range between the minimum possible and the maximum possible think time for an iteration) is less than the testing interval.

To reduce total reading time, edit your Visitor Experience. For each page, you can specify a minimum and maximum time (in seconds) that a visitor will take reading that page. As long as the total page reading time falls within the range of the test you selected, you should see results in your graphs when you run your test. Keep in mind, however, that there's also page request and download time in addition to reading time, so if your web server is using a slow connection, or if pages take several seconds to download, you may still exceed the time allocated in your selected test. In this case, either choose a longer-running test or shorten page reading times.

What are the different parts of a StressMy.com test report?

StressMy.com prepares a report to document the results of each test. In this report, you will see a summary of all the important aspects of your test design and the results of the test in both graphic and spreadsheet formats . The report consists of three main sections:

• Test Overview
• Test Results
• Visitor Experience

The report begins with a Test Overview that describes the traffic and visitor experience parameters that were specified for this particular test run. (The Visitor Experience listing is logically a part of the Test Overview, but appears at the end of the report to reduce clutter.) The Test Overview is the basic documentation of the testing parameters and testing environment. It presents all information necessary to re-create a test and to compare several test runs.

The overview is followed by the Test Results section that begins with a summary of the start and stop test times, total number of Visitor Experiences completed and any errors encountered during the test run. Following the summary information are detailed graphs and spreadsheets documenting the test load over time, completion times for the Visitor Experience and response times for each page visited in the Visitor Experience. The response time graphs plot the minimum, maximum, and average response time at each discreet load level. All data presented in graphical form is also presented in a spreadsheet that follows the graph. Detailed examination of the information in this section can help you identify the bottlenecks in your application (by identifying the slowest and fastest page response times) and/or the point at which the number of concurrent visitors starts causing application errors or excessive response times.

Finally, the report documents the Visitor Experience used for the test. This section can be quite useful when comparing test results and trying to understand the impact of differing page reading times or the impact of various form input options.

How do I interpret a Load vs. Time graph?

The first graph in the StressMy.com testing report is the Load vs. Time graph. This graph plots the number of concurrent simulated visitors over the elapsed time of the test. This graph is usually stair-shaped with the number of steps defined by the traffic pattern you selected for your test. (There's a tiny bit of time between the arrival of each visitor, so the leading edge of each step will probably not be a strictly vertical line.). If your test used a single load, the graph will show a horizontal line instead of a stair shape.

The Load vs. Time graph also plots detected warnings and errors. Each green tick on the graph represents a warning and each red tick represents a fatal error. (A warning is logged but does not stop a visitor experience. A fatal error causes a simulated visitor to stop executing its assigned visitor experience at the error point and restart at the beginning.) A detailed listing of the first 100 errors is available by following the Details link in the Test Results report section.

Careful examination of warning and error timing may reveal important information about the work capacity of your application. Errors that occur at all load levels are probably systemic errors (missing graphics, broken links, etc.). Take a look at the onset time and error rate as the number of simulated visitors increases. This can be as good an indicator of the application's work capacity as the overall per-page response time. Look to see if most of the errors start to occur after a particular load level. Does error frequency go up as the number of simulated visitors increases? If so, you probably identified a key performance parameter of your application Errors that appear to come in clusters (unrelated to load level, or appearing only above a given load level) usually indicate some sort of network congestion or a periodic load on one of the system components unrelated to the tested application. (Like periodically scheduled database maintenance.)

How do I interpret a Visit Time vs. Load (Includes Reading Times) graph?

The Visit Time vs. Load (Includes Reading Time) graph shows a plot of elapsed times for each visitor experience completed for a specific load level during your test. At each load level specified in the traffic pattern, the minimum, maximum and average elapsed visit times are plotted using a range bar. In addition, each visitor's average time is plotted in blue ticks on the range bar. This data allows you to see the distribution and trends in average visit time as the load increases.

Only data from simulated visits that began and ended within a load level are included in the graph. This minimizes the distortion in measured time that could be caused by visits that span an increase in load level. (Is that visit's elapsed time counted in the lower or higher level?) Remember that the visit time includes the inter-page reading time, so the visit will not be completed any sooner than the sum of the minimum reading times and may take as long as the sum of the maximum reading times to complete. Be sure to choose a traffic pattern where the load levels are long enough to allow a single visit to complete.

When interpreting this graph, you should look for the load level where the average elapsed time makes a dramatic change. This knee in the curve indicates the point at which your application can no longer comfortably support the concurrent workload. Examination of the blue tick distribution at each load level will give you information about the consistency of the user experience your application provides. An even distribution of blue ticks across a large min/max elapsed time range reveals that your visitors can experience a wide range of response times. If the blue ticks are clustered around the low end of the range with only a few scattered at the high end (resulting in a high maximum elapsed visit time), your visitors are generally receiving acceptable response times while a few wait for an occasional page.

This graph plots data that includes the reading times specified in the visitor experience. Examination of the other response time graphs will give you information about the performance of each page without reading time included.

This graph is followed by the Visit Time vs. Load (Without Reading Time) graph. This graph plots the same information as Visit Time vs. Load (Includes Reading Time) but excludes reading times. It is sometimes easier to notice subtle trends in the elapsed visit time data when the stochastic reading times are excluded from the plot.

Finally, if your choice of data analysis techniques includes actual inspection of the relevant data, the information used to generate the graphs are presented in a spreadsheet format immediately following the graph.

What does the Page Contribution to Visit Time graph tell me?

The Page Contribution to Visit Time graph shows a stacked bar of the average page response times at each load level in a traffic pattern. Only the response time data for each page is included in this graph, excluding any inter-page reading times.

This display makes it very easy to uncover which pages (if any) in your application are contributing to slow response times on your site. Simply look for the largest bars in the stack and compare the change in relative contribution as the load level increases. You can identify which parts of your application are not scaling linearly with the load. Both of these conditions may lead you to the inspection of operations and components supporting a page with an excessive response time.

Additional detail on the response time behavior of each page may be obtained from the individual page response time graphs.

What does the error report tell me?

The error section of the StressMy.com Test Report summarizes the number and type of errors encountered while running a test. (Remember that the Load vs. Time graph also gives a visual display of the errors.) Details of the first 100 errors may be obtained by following the Details... link in the error section of the Test Report.

The error detail report lists the time each error was encountered, the page encountering the error, the action taken when the error occurred and detailed information about the error. (Specified warnings will be logged, allowing the simulated visitor to continue its site visit. Fatal errors force the simulated visitor to restart the visitor experience.) Pages are identified by their page number in the Visitor Experience.

Examination of the details listed in this report may identify missing graphic objects and help pinpoint the cause of a stress related failure. When web servers are attempting to throttle the amount of traffic, they will typically return a connection refused error. When a network component is overtaxed (or a web server is not gracefully handling an overload situation), the error reported is usually a socket error. Interpreting the error listing may also be enhanced by defining appropriate user defined errors that identify application-specific warnings and fatal errors.

What are the industry standards for response time?

The common industry standard for response times is the "eight-second rule." This is commonly interpreted as a requirement that NO response take longer than eight seconds. Experience, and common sense, have shown that no rigid rule exists.

Some common design practices can mask response times longer than eight seconds. For instance, interleaving text and graphic responses, or sending the first part of a page while generating the remain parts, let many users feel like they are seeing satisfactory response times, even when the application may take much longer to build and download a complete page.

In general, our experience shows that users will leave a site when response times average above eight seconds. It also shows that users will tolerate apparently long waits (30 seconds or more) interspersed in a generally snappy application when they believe that some operation is going on in the background (like credit card processing) and they receive some indication that the system is still alive (like a scrolling "Processing...." banner.)

So, while industry standards such as the eight-second rule exist, you will have to carefully analyze your user's behaviors and expectations in order to set appropriate response time goals.

Why am I getting so few completed visitor experiences?

One of the most common reasons that StressMy.com users observe a low visit count is a mismatch between a Traffic Pattern used for a test and the Visitor Experience selected for a test. Traffic patterns specify how StressMy.com produces a test load over time. Usually, a traffic pattern specifies a step pattern of increasing levels of simulated visitors, with each step occurring for a specific length of time. StressMy.com reports statistics based on each step in the traffic pattern and only counts visitor experiences that start and complete within a step. If the elapsed time for the visitor experience exceeds the length of time covered by a step, the visitor experience will not be counted. Since you can specify random reading times, some instances of a visitor experience may be short enough to be counted during a step, and some may be too long.

So, if you have observed unexpectedly low numbers of completed visitor experiences, compare the step times used in your traffic pattern to the overall expected elapsed time of your Visitor Experience. With a 15 or 20 page Visitor Experience, and a reading time averaging 30 seconds on each page, you can expect the Visitor Experience to run for 7.5 minutes on average. So, select a Traffic Pattern that will allow for that elapsed time in each load step.

If you are using a complimentary StressMy.com account, the longest step time available is 5 minutes so you will have to adjust your Visitor Experiences accordingly. Upgraded accounts offer step times of up to 15 minutes.