Immediately after migrating your systems and applications to the cloud, it’s important to validate that everything is working as expected. This includes verifying if applications are functional, checking for any compatibility issues, and ensuring the reliability of data. Following a QA migration checklist can help you make certain that your deployment went smoothly and that business operations can safely go back to normal.
The validation stage follows the Deploy stage—the last stage in the 5Ds of IT-Conductor's Cloud Migration Strategy. We've published a series of blog posts about each stage and how we approach cloud migration. You can read about them here:
-
DISCOVER: The Role of Application Discovery in Cloud Migration
-
DISTILL: Benchmarking in the Cloud: Generating Cloud Baselines
-
DEVELOP: Building a Cloud Infrastructure: Automated Migration
-
DEPLOY: Deploy Applications to the Cloud
Table of Contents
Pre and Post-Baseline Comparison
Pre and Post-Baseline Comparison
When generating cloud baselines, we've recommended capturing resource utilization metrics such as CPU/Memory, storage, and network. We also suggested capturing snapshots of the last working configuration state of your source application environment. After migration, you need to capture the same set of performance metrics and configuration baselines which you can then use to compare to those you've captured before the migration activity.
Figure 1: Tenant Overview of Source & Target Systems
It's also important to confirm that all objects from the source environment have been created in the new environment. For instance, when migrating databases, tables, views, procedures, or functions should also exist in the new environment.
Figure 2: Migrated Systems in AWS
Questions to Ask:
-
What are the peak values (lowest and highest) of the metric being measured?
-
What is the average value of the metric being measured?
-
What time range to use when measuring the peak and average values?
-
How many data points to use when measuring performance metrics?
-
What are the error rates of transactions/processes/metrics in the application and infrastructure domain, if any?
Functional Validation
When migrating an application to the cloud, you need to ensure that the application is deployed in the desired state as closely as possible. This can be done through functional validation which aims to validate the end-to-end readiness of the application environment for production.
All functional aspects of applications should be retained after the migration activity. From testing client applications to testing user interactions and workflows, it is imperative to verify the end-to-end behavior of applications in a production environment before actually going live. Check for any unexpected errors or issues with system functionality.
Questions to Ask:
-
Are applications behaving as expected?
-
Are applications returning the expected results?
-
Are there any unexpected errors or issues with system functionality?
-
Are there any incompatibility issues that are prohibiting the application to run as expected?
-
Are changes made to the legacy systems support the new functionalities delivered in the new environment?
After cross-checking the functionality of systems and applications in the new environment, you can move on to conduct performance testing, which will help you further identify any potential issues with the deployment.
Performance Testing
Performance testing checks whether the application environment and all of its underlying components remain functional under different scenarios. It can be done using various methods, such as load testing, stress testing, soak testing, and spike testing.
Figure 3: Performance Testing Depiction
Load Testing
Load testing is performed to measure system response under extreme load conditions. It aims to determine the breaking point of a system or application by simulating different loads until they reach the highest utilization peak. This allows you to ensure that your systems remain usable even with the surge of users and lower the risks of them becoming frustrated with slowness and system crashes. During the migration project, load testing profiles should be planned to use a rule of thumb to sample about 20% of transactions that generate at least 80% of the expected workload. See SAP Performance Best Practices for Implementation, Migration, and Upgrade.
Questions to Ask:
-
What is my workload transaction mix between online, batch, and interfaces which need to be load tested?
-
What is the average utilization of my application in %?
-
How many concurrent users can my application handle at n% utilization?
-
How many concurrent users can my application handle at peak utilization?
Stress Testing
Stress Testing, on the other hand, is performed to measure system response beyond the threshold that resulted from the load testing. For instance, you've determined that the application can only handle 300 concurrent users at maximum. During stress testing, you must test the environment with at least 310 users to check how the application behaves when it's utilized past its threshold. Stress testing also helps you verify the redundancies and/or automatic recovery actions in place should your application experience failure.
Questions to Ask:
-
What is the utilization in % if the number of users exceeds the maximum set during load testing?
-
How does my application handle a gradual increase in the number of users past the threshold? If slowness or latency is experienced, note the response time.
-
If the system crashes, how does my application handle failure? How long does it take to recover?
Soak Testing
In soak testing, system reliability is examined through the prolonged execution of test scenarios at high loads. In the previous example, you may need to continuously simulate 300 concurrent users over a sustained period of use. This is important in cases where systems or applications run at high utilization indefinitely. Should results go beyond the expected output, you may need to add or upgrade resources which rarely happens if the sizing was correctly done from the beginning.
Questions to Ask:
-
How long can my application sustain the maximum number of concurrent users without experiencing latency or system crashes?
-
Latency experienced; how does it compare to the results I have during stress testing?
-
System crashes, how long does it take to recover? How does it compare to the results I have during stress testing?
-
Are values almost identical to that in stress testing? If yes, you may need to re-evaluate the threshold set.
-
How do we ensure the stability of my applications going forward?
Spike Testing
Spike testing, from its name, is performed by introducing random load spikes to the system or application being tested. Unlike stress testing, loads are presented at random rather than increasing gradually. This is much closer to the real-world scenario if your application, for example, is a web application developed for ticket-selling sites where users are not pre-determined, or a health information service system during the peak benefit enrollment period.
Questions to Ask:
-
How does my application handle load spikes?
-
What is the utilization in % recorded when load spikes were introduced?
-
Does my application experience slowness when load spikes were introduced?
-
Does my application crash when load spikes were introduced?
-
What measures can be proactively planned to scale up the environment to address application load spikes?
These four testing scenarios must be able to measure performance from the perspective of the application, the infrastructure behind it, and the end-user experience. Target results should be as close as possible to the desired values set during project initiation.
Learn more about cloud migration KPIs and how they can help you measure cloud migration success the right way.
Integration Testing
For modern applications that are built in microservices architecture, it is critical to conduct integration testing as systems are most likely integrated with other systems via API. It's important that communication between two systems is seamless to ensure that issues will not occur. Checking the dependencies among other siloed components are also valuable when it comes to meeting defined SLAs with other external vendors.
Figure 4: Tenant Dashboard
Questions to Ask:
-
What are the different components of my application environment? Is there any integration existing among them?
-
Are those components that have existing integration communicating with each other?
-
Does my application have dependencies with external applications?
-
Does my application meet the SLAs?
Data Migration Validation
For database migrations, the reliability of data is of utmost importance. Immediately after migration, you should be able to cross-check data quality at rest, in transit, and in use. This may be challenging but data reliability is key to cloud migration success. In addition, these data migration validations may be required by your specific company compliance processes.
Figure 5: IT-Conductor Monitoring Approach for SAP Data Services
Data validation can be done in several phases if it is to be manually performed. One method of doing it manually is sampling, where a random subset of the migrated data will be chosen and inspected for quality. This, however, may not guarantee 100% reliability. So, when conducting data validation, determining the level of acceptable risk should be defined and remediated on a regular basis.
Questions to Ask:
-
After conducting data sampling, what percentage of data is inaccurate?
-
Is my database schema still intact? Are there any changes?
-
Are data from legacy applications migrated to cloud resources successfully?
-
Are there any ambiguities seen in the migrated data?
-
Is data flow functioning as expected?
Read more about Monitoring SAP Data Services with IT-Conductor.
Analyze SLOs
Service level Objectives are analyzed and compared with the metrics collected in the source environment prior to the migration activity. At its core, SLOs help you measure business objectives by quantifying criteria. After migrating to the cloud, you should be able to meet the expectations initially set during project initiation. For instance, if you define 99.99% application availability, your application in the cloud should be available 99.99% of the time or higher.
Questions to Ask:
-
Are my business objectives met after migrating to the cloud?
-
Are there any items that need to be remediated to meet the defined SLOs?
Security Testing
One of the main hurdles for organizations moving to the cloud has been concerns over security. This gives you more reason to perform security testing after migrating to the cloud. Security testing, however, does not start after the migration activity. Risk assessments should already be performed prior to the migration activity. Creating an incident response plan should come next. This helps organizations identify and respond to any potential security threats quickly and effectively.
After migration, you need to implement security controls around identities wherein only the right users are allowed to access the right resources at the right times for the right reasons. This ensures that your resources in the cloud can be accessed only by authorized users. You also need to take into account where sensitive data are stored and build stronger security measures to protect them from security breaches.
Questions to Ask:
-
Who has access to data and why?
-
What sensitive data were transferred? Where was it stored after migration? Is it safe from prying eyes?
-
What security policies are in place to protect my data?
Learn more about how to secure your cloud environment.
User Acceptance Testing
We have discussed quite a long list to perform after migrating to the cloud, but none is more important than user acceptance testing (UAT). It is the process of verifying that your migrated applications meet the needs and expectations of end users. It ensures that end-users can continue doing their day-to-day jobs without any disruption. By ensuring that your systems are tested thoroughly before they go live, you can avoid embarrassing customer service incidents later down the line.
UAT comes in two forms, mainly functional and non-functional testing. As discussed above, functional testing aims to validate the end-to-end readiness of the application environment for production. With non-functional testing, tests should focus on how users interact with the new system and how they feel about using it. It's also important to regularly get feedback from end-users as you continue to develop and improve the experience of your organizations in the cloud.
Questions to Ask:
-
Are you able to perform your daily tasks without any problems?
-
How do you compare the experience working now versus working before the migration?
Business Continuity Testing
Most companies, especially publicly listed ones, are required to have business continuity (BC) test plans and test at least once annually. These may include high availability (HA) testing within the local geography for the cloud or data center, and disaster recovery (DR) testing between geographically separated regions to ensure the business systems can survive a local disaster.
When migrating to the cloud, BC plans are definitely affected due to the nature of the system infrastructure changes, thus adapting the BC plans to the technology and process changes is often required.
Questions to Ask:
-
How is HA design and configuration to be tested, such as manual failover and automatic failover by clusters?
-
How is DR going to be tested, both technically and with the business groups or customers?
What's Comes Next?
Migrating to the cloud is only the beginning. Once you've moved your applications to the cloud, it's important to make sure that all of your QA processes are in place so that you can guarantee a smooth transition for your users. This checklist will help make sure that everything from functional testing to UAT is covered and executed properly. By conducting these testing scenarios, you'll be able to minimize any potential impacts on your users and make your move to the cloud a success.
