Testing SAP R/3

A Manager's Step-by-Step Guide
By Jose Fajardo Elfriede Dustin

John Wiley & Sons

Copyright © 2007 John Wiley & Sons, Ltd
All right reserved.

ISBN: 978-0-470-05573-1


Chapter One

Introduction

SAP testing is complex, difficult, and esoteric. The perils and risks to the intended SAP production system are maximized when the project team does not have enough skilled testers and a robust approach for testing the system, tracing the entire system design and architecture to testable requirements, and eliminating defects. A comprehensive plan or approach for testing an SAP system even for a small project includes assembling a test team, acquisition of test tools, establishment of a test lab, construction of a test calendar, monitoring of testing activities, training for testing participants, and completion of a test cycle based on predefined criteria.

A test plan for SAP includes the approach; description; roles and responsibility for conducting usability testing; white box and black box testing for interfaces; negative testing; security testing for roles, integration, scenario, unit, and performance; user acceptance testing; and regression testing. Few companies implementing SAP are structured or organized to address all these various types of testing.

This book offers guidance and assistance to project managers, test managers, and configuration leaders who want to establish a testing framework from the ground up based on industry-established principles for evaluating requirements, providing coverage for requirements, diagramming processes, test planning, estimating testing budgets and resource allocation, establishing an automation framework, and reviewing case studies from large SAP implementations.

WHY THIS BOOK?

SAP is by far the world's largest enterprise resource planning (ERP) application, and this position is not likely to be relinquished anytime soon. SAP traces its origins to its mainframe-based R/2 version from the 1970s. Even though SAP and other vendors have developed implementation methodologies for SAP that provide emphasis for activities needed to design the system and for going live, many of these methodologies fall short of addressing robust testing practices for SAP in particular in light of the prominence of automated test tools and outsourcing testing activities.

The roles and activities for SAP configuration, for SAP advanced business application programming (ABAP) development, and SAP Basis are for the most part well understood and clearly defined. In contrast, when it comes to SAP testing, the roles and activities are a mystery and subject to interpretation. It is possible for a person who is entertaining becoming an SAP consultant to take courses on how to configure a particular SAP module, how to build security roles, or how to develop ABAP programming code but not how to test SAP R/3. Functional testing of SAP is often left to individuals without a testing background whose main tasks are to configure the system. Other types of SAP testing such as technical and system testing for system performance and backup and recovery are also left to individuals without a testing background whose main responsibilities are to design and maintain the technical architecture for SAP.

This book was written to help SAP projects address weaknesses in the SAP testing life cycle, define testing and quality assurance activities, and overcome misconceptions about SAP testing. The book contains contributions from industry leaders in the fields of SAP testing, test tool automation, and templates to help project managers and test managers establish immediate testing best practices. It covers all aspects of SAP testing from preparation to resolution of defects.

WHAT DOES THIS BOOK OFFER ME?

This book is written from the point of view of the company or entity requesting SAP services from a systems integrator. The book emphasizes testing practices predicated on the following principles:

* Building a system with quality as opposed to merely testing for quality.

* Adhering to testing practices from SAP's ASAP Roadmap Methodology, IBM's Ascendant(tm) methodology and guide for implementing SAP, Institute of Electrical and Electronics (IEEE) standards, and the Capability Maturity Model (CMM) from the Software Engineering Institute (SEI).

* Drafting of clearly stated requirements that can be validated with test cases.

* Construction of a requirements traceability matrix (RTM) to verify all in-scope requirements.

* Supporting each testing effort with an exit, entrance, or suspension criterion.

* Validation of production support changes through thorough regression testing.

* Subjecting all test results to third-party verification and approval (sign-offs) from appropriate project stakeholders.

* Diagramming processes and requirements with Unified Modeling Language (UML) notation.

* Documentation and adherence to test plans and test strategies that are subjected to version control.

* Early formation of a test team that participates in design workshops during the blueprint phase, change control boards meetings, and the "go/no go" decision.

* Inclusion and enforcement of quality assurance (QA) standards.

* Peer reviews and inspections for testable requirements.

* Independent verification and validation of system design.

* Independent "hands-on" testing with participation from end users who execute test cases.

* Functional testing with manual testing and automated test tools.

* Maintaining testing deliverables such as test cases, test results, and testing defects in a test management tool that includes security features and audit trails.

* Compliance with company and industry audits.

The book is a primer for testing SAP from unit testing through regression testing for production support. The intended audience for the book includes test managers, project managers, integration leaders, test team members, QA personnel, and auditing groups. The book provides templates, case studies, and criteria to establish a framework for SAP testing, establish a test case automation strategy, mentor junior testers, and identify tasks and activities that the SAP system integrator is expected to accomplish for the client requesting SAP services. Specifically, this book will address the following activities that are prevalent yet poorly conducted at most SAP projects:

* Identifying the testable requirements.

* How to ensure that requirements are testable, unambiguous, clearly defined, necessary, prioritized, and consistent with corporate policies or industry standards.

* How to retain testing documentation to support audits (i.e., Section 404 from Sarbanes-Oxley).

* Defining the scope of testing.

* Creating a test plan and test strategy.

* Developing a strategy for acquiring automated test tools and for automating processes that includes verification at the graphical user interface (GUI) and back-end layers.

* How to create a library of automated test cases for production regression testing that can be executed unattended.

* How to define and verify service-level agreements (SLAs). * Boundary testing for negative and positive testing.

* How to apply quality standards from the SEI and Rational Unifying Process (RUP).

* Creating robust flow process diagrams that include narratives.

* Techniques for estimating the testing schedule, duration for testing activities, and number of testers needed.

* How to compress or reduce the necessary number of test cases with the technique of orthogonal arrays (OATS) for projects implementing SAP variant configuration or projects that have multiple variations for end-to-end processes and are time constrained.

* Defining criteria for deciding which processes or scenarios are suitable for testing.

* How to estimate testing costs and budget.

* Creating an RTM to ensure that coverage has been provided for all types of testable requirements (i.e., functional, security, performance, archiving, technical, and development requirements).

* Creating a test schedule and a test calendar.

* Defining objective criteria to commence, finish, and stop testing.

* Managing, categorizing, and resolving test results.

CHALLENGES IN SAP TESTING

Established commercial implementation methodologies for SAP typically fail to address how requirements will be met, the criteria for testing, the framework for utilizing test tools, necessary resources for testing, estimating testing budgets, specific testing roles and responsibilities, and how test defects will be managed and resolved. Furthermore, many factors hamper successful testing at most SAP projects such as unclear requirements, inability to trace the system design to requirements, missing dedicated test teams, waiving defects without appropriate workarounds, and inadequate involvement of needed participants for testing such as subject matter experts (SMEs) for capturing requirements and end users for user acceptance testing. Despite these testing challenges, many SAP project managers perceive that their SAP implementation is successful or "fine" even when the production help desk team is flooded with complaints that the system does not perform necessary functionality, the production system does not meet intended performance SLAs, security roles are not defined and implemented correctly, the system produces short dumps because it cannot perform exception handling or not enough negative testing was conducted, data is not converted properly from legacy systems, and end users cannot find even the most basic data or necessary reports.

The SAP arena is replete with functional, development, and technical consultants that moonlight and parade as SAP testers for various testing efforts but often lack sufficient knowledge to establish a successful testing strategy and framework. What is more puzzling and baffling at SAP projects is that it is the individuals with the least amount of knowledge and skills in the area of testing who are the ones in charge of leading and managing the testing effort since many SAP projects do not have dedicated test managers or centralized test teams. Admittedly, testing at any SAP project is an integrated effort that requires the expertise and skills of several resources such as SMEs, functional configuration resources, ABAP developers, and business analysts. Yet executing testing activities without the guidance and help of testing professionals is analogous to taking a trip without knowing what the final destination will be.

Frequently, an individual moonlighting as an SAP tester will state that "testing is breaking or exploring the system" or that he "knows how to test," which undoubtedly leads to a misconception about what SAP testing is really all about. The truth is that many companies fail to adequately test an SAP system and rather deploy the system into production because testing is taking too long, which consequently forces the production support team to fix the system for the first six to eight months after the system is deployed because it was never properly tested prior to its release into the production environment. Whether by accident or on purpose, often the modus operandus for many corporations is to deploy an unstable and/or poorly tested SAP system into production because defects and system problems can be dealt with at a later date in the production environment, even while there is substantial and empirical data that demonstrates that removing system defects is least expensive when done in the early stages of testing.

Industry data shows that removing system defects in a live production environment is at least 20 to 40 times more expensive than doing so in the unit-testing phase or during the requirements-gathering phase. Many defects can be eliminated or prevented altogether with thorough evaluation and peer review of requirements. Many corporations pay expensive consulting fees to fix production problems arriving at the production help desk rather than address these problems or defects during the applicable testing phase. The main reason that this occurs is that SAP projects often do not spend the time or have the appropriate resources to ensure that the captured requirements are peer reviewed and evaluated with objective criteria, or to construct an RTM to provide coverage for all requirements and establish objective testing criteria for each testing phase. Another critical or overlooked reason that causes defects that should have been resolved during testing to slip into the production environment is that individuals acting as SAP testers cannot reach consensus on testing nomenclature or the test approach.

The mere term testing in the SAP world is in and of itself broad enough to create ambiguity, since different individuals will have different perceptions and experiences about what testing means. Testing encompasses many activities such as requirements gathering and traceability, test planning, test design, test execution, test reporting, test results, and resolution of defects to cover a wide range of testing efforts such as unit, boundary, scenario, development, white/black box, security, smoke, integration, performance, user acceptance, and regression testing. Rarely, if ever, do two or more individuals from the configuration, development, or technical teams have the same nomenclature or understanding for a particular type of test. Chaos and inconsistency are the ensuing results from misunderstanding about what the term testing entails or what activities are associated with testing. Dedicated test teams can establish consistency for all testing terms for all project members based on established guidelines and nomenclature from credible sources such as the Software Engineering Institute (SEI), IEEE standards, and the SAP ASAP Roadmap methodology.

A common theme repeated at many SAP projects is that conclusive evidence is missing to show that requirements have been met before releasing the system into the production environment. Most project managers or functional managers cannot answer with any degree of confidence or objectively whether the in-scope requirements captured during the requirements phase have been met before releasing or deploying a system. This occurs because the concept of an RTM is not embedded within most, if not all, of the mainstream or conventional methodologies for implementing SAP for either initial SAP implementations or SAP upgrades.

Test tools pose a challenge for many SAP projects. SAP tools hold the promise of unattended test case playback at any time, increased testing coverage, testing of processes with multiple data and process variations, verification of objects and calculations, and generation of automated test logs with time stamps for audit purposes and compliance. Many SAP system integrators and test tool vendors are adept at convincing companies to spend hundreds of thousands of dollars in acquiring automated test tools and test tool training only to have the test tools gather dust. Test tools can sit idle because the company acquiring the test tools is missing an automation framework and thus cannot successfully engage the appropriate resources to maintain, install, and utilize the test tools. The payback period or return on investment (ROI) for test tools is not maximized or even reached until a series or library of automated test cases can be constructed and reused frequently for future system releases or to support production regression testing to the point where automated execution is cheaper than doing the same tasks with manual labor hours. Constructing a library of automated test cases is rarely achieved even by companies that have had SAP in the production environment for years because they do not allocate the necessary skilled resources to maintaining and utilizing the test tools. Companies that commit hundreds of thousands of dollars to the acquisition of test tools that sit idle compromise their testing budgets. Consequently, these companies resort to testing SAP exclusively with manual testers.

(Continues...)



Excerpted from Testing SAP R/3 by Jose Fajardo Elfriede Dustin Copyright © 2007 by John Wiley & Sons, Ltd. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.