What it is and how it works
ACE Pipeline takes the stress out of stress testing in the CI pipeline by allowing your team to agree on the criteria that will cause the test to fail or allow it to pass.
Only by eliminating false negatives and preventing false positives can you be sure your test will pass every time unless there's an unexpected error or slowdown.
Once the team agrees on pass/fail criteria and approves the contract, test results will be analyzed automatically after each test run and the pass/fail status returned to
the caller. Define response time and request rate limits at the summary and page level. Historical trend analysis shows how each test performed in relation to the previous one, to a baseline and an overall average.
ACE integrates seamlessly with any environment for testing in the CI pipeline and lets you run JMeter tests of any size in the cloud. GitHub integration is built in along with
a Grafana Dashboard for viewing test results.
ACE Pipeline is a cloud platform that deploys on your AWS account. This gives you a dedicated server that only you control and have access to, not a SaaS server shared by many.
Along with seamless integration with the CI pipeline, ACE lets you run JMeter tests of any size and duration in the cloud. You can spin up as many load agents as you need in
the ACE Cloud Portal and terminate them when finished with your test. Since there is no extra charge for this you can save hundreds or even thousands of dollars compared to what the competition
charges. Watch this short video to see how easy it is to deploy ACE Pipeline on AWS.
A Brief Tutorial
An ACE contract defines all the criteria for passing or failing your test and it must be approved by the project stakeholders. Each change is stored in an audit log along with
approvers, approvals and test results. When the test is run, a small Java executable on the build agent connects to the ACE server and runs the test in the cloud, or downloads
it, runs it locally and uploads the results, where they
are analyzed according to the contract and a pass/fail is returned. This lets you integrate your stress tests in the CI pipeline with a single line of code. Since the Java app runs
inside the firewall it can test lower level environments that are not exposed to the internet. When tests are run there are notification emails users can subscribe to or the results can
be posted to a Slack or Teams channel. For the simplest integration model you can invoke your stress test from a functional test after it passes.
Load Testing on Steroids
For testing outside the CI pipeline, the same consensus-based analysis is used but the number of load agents is up to you – the ACE load agent supports up to 500 virtual users,
so to run a test with a million virtual users you would need 2,000 load agents. This can be handled easily by the t3a-Large instance of the ACE server while the t3a-XLarge can support
up to 5,000. While your tests are not likely to be anywhere near this big, they can be as big as you need them to be for only 2 cents per hour per agent and 5 cents per hour for the server.
The same contract used
for the CI test can be used as a template for your other tests, requiring you to change only the response time and request rate limits as necessary. A typical series of tests run
outside the CI pipeline might include a load test, a peak test, a stress test, and an endurance test. Having a separate contract for each one lets you apply historical trend analysis to them and catch any new issues or slowdowns as soon as they happen.
A Brief Tutorial
Load Testing on Steroids
ACE Pipeline offers a revolutionary new approach to load and stress testing called Consensus-Based Analysis. It gives your team a way to reach a formal agreement about what will cause a test to fail and what will allow it to pass, including which errors can and can’t be ignored and what the response time, request rate, and throughput limits should be. This becomes a self-enforcing contract that gets applied automatically to each test run, resulting in a pass/fail status that is every bit as dependable as the result of a unit test or functional test. In fact, it’s the only way to have a fully automated load or stress test that doesn’t require anyone to examine the result unless it fails. And you can be sure the test won’t fail unless something is broken so most of the time it will pass as expected.
This is far more robust and reliable than using error rates to pass or fail a test – a small error rate can mask serious issues by producing a false positive, just like a false negative can
cause a test to fail when it shouldn’t. This requires a granular level of control over every metric, including every type of error – network error, script execution errors, 400 errors and 500 errors.
In addition to response time limits, expected hit rates and throughput can also fail a test if they drop below a certain threshold. You can even eliminate response times for calls to external services
that you have no control over.
Consensus-Based Analysis is the result of many years of experience running stress tests in the CI pipeline for many applications, discovering what works and what doesn’t and finding solutions to the problems that come up every time. It integrates seamlessly with the CI pipeline no matter which OS, platform or build orchestrator is used. Make all of your load testing more robust and reliable and save all the time normally spent discussing and analyzing test results by letting ACE do it for you!