5.1.1.6.2 Process for testing and evaluating the effect of changes to the repository’s critical processes

The repository shall have a process for testing and evaluating the effect of changes to the repository’s critical processes.

Supporting Text

This is necessary in order to protect the integrity of the repository’s critical processes such that they continue in their ability to meet the repository’s mandatory requirements.

Examples for Meeting the Requirement

Documented testing procedures; documentation of results from prior tests and proof of changes made as a result of tests; analysis of the impact of a process change.

Discussion

Changes to critical systems should be, where possible, pre-tested separately, the expected behaviors documented, and roll-back procedures prepared. After changes, the systems should be monitored for unexpected and unacceptable behavior. If such behavior is discovered the changes and their consequences should be reversed. Whole-system testing or unit testing can address this requirement; complex safety-type tests are not required. Testing can be very expensive, but there should be some recognition of the fact that a completely open regime where no changes are ever evaluated or tested will have problems.

Evidence Provided

APTrust staff adheres to a Software Development Styleguide (APTrust members only; supplied to auditors on request) that includes the use of unit tests and continuous integration testing. Changes to our software and repository are tested by developers with localized integration tests and by automated tools (Travis-CI) before being put in place. Every commit to our codebase triggers automated unit tests. Integration tests are run locally as well as on the staging environment as needed. Integration tests mock all of the repositories services to test ingest and restore workflows during the CI/CD on Travis. These tests will be updated in the new CI/CD platform for more dynamic testing options including DevSecOps processes.

APTrust maintains a staging environment that is replicates the environment of both demo and production for pure development and testing purposes. Demo is a mirror of production that is similar to a true staging environment. Changes to the repositories software are deployed there first to assure that changes won’t adversely affect the production system.