OroCommerce Performance Load Testing with k6 

k6 is an open-source load-testing tool designed to test the performance of APIs, microservices, and websites. It allows developers and testers to simulate traffic to a system and analyze its performance under various conditions. The key features of k6 include:

  • Script-based tests using JavaScript

  • Support for HTTP/1.1, HTTP/2, and WebSocket protocols

  • Metrics collection and real-time reporting

  • Cloud execution and distributed testing capabilities

  • Integration with CI/CD pipelines

This documentation outlines the steps for performing performance and load tests on the OroCommerce application using the k6 load testing tool.

Repository Structure 

The repository is structured as follows:

performance/
├── scripts/                  # Load test scenarios for k6
│   ├── warmingUpTheApp.js
│   ├── storefrontTests.js
│   └── checkoutTest.js
└── summary.html              # Summary test result output (generated by k6-reporter)

Setup Instructions 

Installing k6 

Debian/Ubuntu 

To install k6 on Debian/Ubuntu:

  1. Add the k6 GPG key and repository:

    sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update

  2. Install k6:

    sudo apt-get install k6

macOS 

To install k6 using Homebrew, run:

brew install k6

Windows 

To install k6 using the Windows Package Manager, run:

winget install k6

Docker 

To pull the latest k6 Docker image, run:

docker pull grafana/k6:latest

Running Tests 

Test Parameters 

Before running the tests, specify the following parameters in the test commands:

  • BASE_URL: Base URL of the OroCommerce application.

  • USERNAME: Storefront username.

  • PASSWORD: Storefront user password.

  • VU: Number of virtual users.

  • DURATION: Duration of the test (e.g., 60s, 10m).

  • THRESHOLD_95: Pass/fail criteria for your test metrics (95th percentile).

  • SL_ID: Shopping list ID (optional).

  • SHIPPING_METHOD: Shipping method (e.g., "fixed_product_5", "flat_rate_6").

  • PAYMENT_METHOD: Payment method (e.g., "payment_term_1").

Test Commands 

Run the following commands to execute the tests:

Debian/Ubuntu 

  1. Warm up the application:

    k6 run -e BASE_URL="https://example.com" -e USERNAME="AmandaRCole@example.org" -e PASSWORD="AmandaRCole@example.org" -e VU=1 -e DURATION=60s -e THRESHOLD_95=3000 scripts/warmingUpTheApp.js

  2. Run storefront tests:

    k6 run -e BASE_URL="https://example.com" -e USERNAME="AmandaRCole@example.org" -e PASSWORD="AmandaRCole@example.org" -e VU=1 -e DURATION=600s -e THRESHOLD_95=850 scripts/storefrontTests.js

  3. Run checkout tests:

    k6 run -e BASE_URL="https://example.com" -e USERNAME="AmandaRCole@example.org" -e PASSWORD="AmandaRCole@example.org" -e SL_ID=2 -e SHIPPING_METHOD="fixed_product_5" -e PAYMENT_METHOD="payment_term_1" -e VU=1 -e DURATION=600s -e THRESHOLD_95=850 scripts/checkoutTest.js

Docker 

  1. Warm up the application:

    docker run --rm --network host -u "$(id -u):$(id -g)" -v ${PWD}:/home/k6/performance -w /home/k6/performance grafana/k6:latest run -e BASE_URL="https://example.com" -e USERNAME="AmandaRCole@example.org" -e PASSWORD="AmandaRCole@example.org" -e VU=1 -e DURATION=60s -e THRESHOLD_95=3000 scripts/warmingUpTheApp.js

  2. Run storefront tests:

    docker run --rm --network host -u "$(id -u):$(id -g)" -v ${PWD}:/home/k6/performance -w /home/k6/performance grafana/k6:latest run -e BASE_URL="https://example.com" -e USERNAME="AmandaRCole@example.org" -e PASSWORD="AmandaRCole@example.org" -e VU=1 -e DURATION=600s -e THRESHOLD_95=850 scripts/storefrontTests.js

  3. Run checkout tests:

    docker run --rm --network host -u "$(id -u):$(id -g)" -v ${PWD}:/home/k6/performance -w /home/k6/performance grafana/k6:latest run -e BASE_URL="https://example.com" -e USERNAME="AmandaRCole@example.org" -e PASSWORD="AmandaRCole@example.org" -e SL_ID=2 -e SHIPPING_METHOD="fixed_product_5" -e PAYMENT_METHOD="payment_term_1" -e VU=1 -e DURATION=600s -e THRESHOLD_95=850 scripts/checkoutTest.js

Interpreting Results 

After running the tests, a summary of the results will be available in performance/summary.html. This summary is generated by k6-reporter and provides detailed metrics on the performance and load handling of the OroCommerce application.

Default Metrics 

Metric

Description

http_req_duration

Measures the total time for the request. Includes sending, waiting, receiving, connecting, and TLS handshaking.

http_req_waiting

Measures the waiting time of the request. It is the time spent waiting for the server response.

http_req_connecting

Measures the time spent establishing a TCP connection to the remote host.

http_req_tls_handshaking

Measures the time spent on the TLS handshake. Includes all blocking phases of the TLS handshake.

http_req_sending

Measures the time spent sending data to the remote host.

http_req_receiving

Measures the time spent receiving a response from the remote host.

http_req_blocked

Measures the time spent blocked before initiating the request. Includes time blocked on DNS resolution, TCP connecting, and TLS handshaking.

iteration_duration

Measures the time it takes for one iteration of the script to execute.

group_duration

Measures the time it takes for a group to complete. A group is a logical grouping of actions in a script.

Custom Metrics 

Metric

Description

authentication_post_request

Measures the performance of the POST request for authentication.

check_failure_rate

Measures the rate of failed checks in the test.

create_sl_request_post_request

Measures the performance of the POST request for creating a shopping list request.

create_sl_widget

Measures the performance of creating a shopping list widget.

load_about_page_cms_guest_user

Measures the performance of loading the About page for a guest user.

load_home_page_guest_user

Measures the performance of loading the Home page for a guest user.

load_home_page_logged_in_user

Measures the performance of loading the Home page for a logged-in user.

load_login_page

Measures the performance of loading the Login page.

load_product_detail_page_guest_user

Measures the performance of loading the Product Detail page for a guest user.

load_product_detail_page_logged_in_user

Measures the performance of loading the Product Detail page for a logged-in user.

load_product_listing_page_guest_user

Measures the performance of loading the Product Listing page for a guest user.

load_product_listing_page_logged_in_user

Measures the performance of loading the Product Listing page for a logged-in user.

load_product_search_page_guest_user

Measures the performance of loading the Product Search page for a guest user.

load_product_search_page_logged_in_user

Measures the performance of loading the Product Search page for a logged-in user.

Key Performance Indicators (KPIs) 

Average Response Time 

Average response time is calculated as the average duration for a request to be completed. Use the http_req_duration metric in the report to measure this.

Peak Response Time 

Peak response time is the maximum time it takes for a request to be completed. Use the http_req_duration metric in the report to find the maximum value.

Error Rate 

Error rate is the percentage of requests that results in an error. Use the check_failure_rate metric in the report to measure this.

Throughput 

Throughput is the number of requests that your application can handle per unit of time. It is usually measured in requests per second. In the provided HTML file, this information is given as Concurrent Requests Per Second (CRPS) in the title of each tab.

Concurrent Users 

Concurrent users is the number of users that are simultaneously connected to your application. In the provided HTML file, this information is given in the title of each tab (e.g., 1 VU is 1 Virtual User, 25 VU is 25 Virtual Users, etc.).

Additional Resources