.png){kind=icon}
Every year around the holidays online transactions break new records. In 2016, Black Friday online sales hit $3.34 Billion including $1.2 Billion spent via Mobile devices. Although the focus is usually on Black Friday, Cyber Monday gains an increasing part of the online purchases over the years. Last year, Cyber Monday broke a new U.S. e-commerce record of $3.45 Billion.
Testing on Mobile Devices
The number of transactions on mobile devices continues to increase and it now represents over a third of the purchases made online. Based on data published by Comscore, smartphone usage has doubled between 2013 and 2016 and that change is reflected by online shoppers. A large number of users still remain on desktop (both Mac and Windows), which is why it is critical when creating the test matrix and list of browsers to support and perform e-commerce testing on.
Get Your App Ready Early
Large retailers prepare for the holidays weeks to months in advance. Typically, new features are implemented a few weeks before the holidays to be tested. Last year, Kohl’s and Walmart were some of the companies offering a mobile payment app.
This year, Walmart is launching a new service called Mobile Express Returns to allow their customers to quickly return items in store. This will allow their customers to return items in any of the 4,700 Walmart stores through the Mobile Express Lane. You can read more about it on Business Insider.com.Meanwhile, the list of stores closing in the US continues to increase. In a recent article, Business Insider listed 6,401 stores shutting down in 2017.
Plan to scale up!
One of the most important tasks in testing will be load/performance tests and monitoring the results and traffic on an e-commerce website over the holidays. Every year a large e-commerce makes the news on Black Friday at midnight when their website goes down. A few years ago it was Lego.com. At midnight the website was not accessible (probably due a huge amount of traffic) and their customers were not able to access the sales and deals that were supposed to be available online.
If you are interested in Load and Performance testing you can find some great tips about this subject on BlazeMeter.com blog.
Defining a Testing Strategy
Different teams have different approaches for e-commerce testing. From our experience, the test planning combines front-end testing and automated testing. The user will be directly impacted if some of the features aren’t working as expected on an e-commerce site: missing links, pricing issues, search not functional, payments, etc.
If you work on e-commerce testing a small to medium-size website you’ll want to prepare a testing checklist and create some test cases based on the different pages and user flows of the site. You can break different sections of the website into categories so you can test the pages separately and the different features related to the Search, Login, Account Management and Cart.
The functionality and compatibility testing will need to be done across the whole website:
Overall, it’s best to use a test cases management tool such as Test Lodge or Test Rail. At PLUS QA, we usually recommend that our clients pick a solution early in the project for creating and managing test cases. If you decide to use a spreadsheet document, make sure you keep it up-to-date through the testing cycles so the testers do not lose time trying to execute outdated test cases or invalid user scenarios.
A good, updated test case document will allow you to track test results through each test cycle and also track bugs referenced inside a test case.
What about Automation?
Even if you decide to test the majority of a website through manual testing, automation testing will improve the quality of your application. Once you have a stable environment you will need to identify what to automate and what type of framework you want to use for testing. Running automated scripts will be helpful when going through multiple rounds of regression before, during and after the holidays. The purpose of the regression tests is to ensure no new bugs are reintroduced throughout the development cycles and the changes made to the website. In addition, it’s also quicker to test an entire checkout flow through automation than it is through manual testing.
Be prepared to test on Web, Mobile and Mobile Web
Prioritizing your browser and device list
For a few years, experts predicted that desktops would soon be gone and replaced by tablets, phablets and phones. There is a good chance this will happen in the coming years but this year you should still plan to test on desktop and mobile browsers, if you test an e-commerce site. A challenge when testing the functionality and compatibility of a project is deciding which browsers to test.
Desktop Browser Stats - US September 2017
Source: gs.statcounter.com
We typically recommend our clients look at the user analytics and define 2 browser priority lists based on their user’s statistics: the first one will be the list of browsers that are included in 90% of visitors of the site. The secondary list will be older devices and browsers that represent a small amount of visitors. In some cases, low-end devices might be 1% of the users but they can represent hundreds of thousands of people depending on the popularity of the website.
It’s important not to ignore those users. If you are interested in Browsers and Mobile OS statistics check out the stats we post each month on our blog.
Connectivity Testing
One last thing to consider when you release a mobile app is the connectivity testing. How does the app respond when you use it in a store with a low signal? How long does it take to load a product page? What error messages are in place if you are out of network? These are test scenarios you want to include in your test plan and also include in your regression test cycles. If your website or your app performs poorly when a person is out of a cellular network or if they switch between a 3G and Wi-Fi network the user should not be negatively impacted.
Security Testing
It’s becoming more common to hear about data breaches and hacked accounts. Just recently, Instagram and Yahoo made it in the news when billions of Yahoo user accounts and millions of Instagram accounts were hacked.When it comes to testing an e-commerce, it is critical to ensure that the incoming and outgoing data is handled correctly. There are different security testing methodologies and the most standard ones consist of making sure that the website can prevent SQL Injections, XSS, Authentication and Authorization vulnerabilities.
For Web apps testing, if you use a platform such as WordPress, Magento or Drupal you can use tools such as SiteGuarding to scan your domain for malware, or Web Inspector to scan your site for malware, backdoors or trojans.
Conclusion
The holidays are the most critical time of the year for the large retailers and for small companies that sell their products online. Your test strategy will be different based on the size and the nature of your project. Ultimately, you want to make sure your e-commerce or your app is not impacted by a poor user experience, a bug in your app or a site crashing at midnight on Black Friday. In the past few years, we have helped large retailers perform e-commerce testing on their apps and e-commerce sites so millions of users could access them and purchase their products seamlessly. The tools and the experience are changing over the years for QA teams but the goal remains the same: help our clients launch stable products with a great user experience.