IT Infrastructure Qualification Planning IQPC – IT Infrastructure Compliance, Security & Risk Management, 16Nov2009
Carolyn C l St Stockdale kd l Director, Global Computer System Validation Premier Research Group, Ltd.
Welcome and Introductions • Introduction • A short poll... • What’s the mix of engineers vs. managers, quality assurance? • Has your organization qualified their infrastructure? • What did the activity include? ? • What do you hope to get out of this presentation?
“The validated status of GXP applications that are dependent upon underlying IT Infrastructure is compromised if the IT Infrastructure is not maintained in a demonstrable state of control and regulatory compliance.” li ” GAMP Good Practice Guide: IT Infrastructure Control and Compliance, ISPE, 2005
Overview • • • • • • • • •
Some Basics Typical System Development Lifecycle (SDLC) pp y to Infrastructure Q Qualification Applicability What is Infrastructure? What is Infrastructure Qualification? Why Qualify Your Infrastructure? Infrastructure Qualification Planning Qualification by Platform (An Approach) Maintaining the Qualified State
Some Basics • Qualification • qualification, f installation. (FDA) Establishing confidence f that process equipment and ancillary systems are compliant with appropriate codes and approved design intentions, and that manufacturer's recommendations are suitably considered. • qualification qualification, operational. operational (FDA) Establishing confidence that process equipment and subsub systems are capable of consistently operating within established limits and tolerances. • POINT IN TIME
• Validation • Establishing documented evidence which provides a high degree of assurance that a specific process [system] will consistently produce a product meeting its predetermined specifications and quality characteristics (FDA) • FROM CONCEPT TO DECOMMISSION
GOOD SYSTEM DEVELOPMENT PRACTICE
Typical SDLC vs.. Validation Lifecycle Validation Methodology Plan System y and Validation Approach
Requirements Analysis
Define Requirements Select a Vendor Design the System
Architectural Design Detailed Design Coding and Debugging System Testing I l Implementation t ti
Construct the System Integrate and Install the System Qualify the System Evaluate the System
G Gover ning Process P s and Proced P dures
Waterfall SDLC Software Concept p Development
Typical SDLC vs.. Validation Lifecycle Waterfall SDLC Software Concept p Development
Validation Documentation Project and Validation Plan(s) User Requirements Risk Analysis
Requirements Analysis
Requirements Specification/Documentation Traceability Matrix Vendor Audit
Architectural Design Detailed Design
Technical Design/Hardware Specification(s) Design Review(s)
Coding and Debugging System Testing Implementation
Code Review(s) System Test Plan(s) and Scripts Installation Verification Operational Verification Performance Verification
Applicability to Infrastructure Qualification • Prospective Validation/Qualification • Either model, and others, are worthwhile and recommended
• Retrospective R t ti V Validation/Qualification lid ti /Q lifi ti • Less Applicability • Requires more customization and ingenuity • Infrastructure is already in place, can’t pre-define • Changes tend to be more frequent and driven by management of the infrastructure • Dynamic operational needs.
So What is Infrastructure Qualification? • First, what is ‘Infrastructure’? • • • •
Depends on whom you are talking to… to Depends on the services offered by IT… What do you think ‘Infrastructure’ is? Th ’ no absolute There’s b l right i h answer
• Typical yp to consider ‘Infrastructure’ analogous g to ‘Network’ • This idea could result in gaps in your approach, i.e. • • • •
Database D b S Services i Backup and Restore Services Storage Area Networks Security (authentication and authorization)
Then What is ‘Infrastructure’? On the Technical Side… • All of the computer systems with their associated h d hardware, operating ti software ft ((other th th than software ft applications), and networks used to run the business On the Regulated Side… • An aggregation of platforms and services including their associated processes, procedures and personnel . GAMP GGoodd PPractice ti G Guide: id IT Infrastructure I f t t Control C t l and d Compliance, C li ISPE, 2005
What is ‘Infrastructure? • Most will typically include the basics… • Network (LAN/WAN) • Cabling, Switches, Routers, Firewalls, Protocols, VPNs, etc.
• Data Center/Computer Rooms
• Must also include… • Operating O ti S Systems t • Software Tools • Service Management and Associated Procedures • • • • •
Incident Management B&R Security Performance P bl Problem Reporting R ti and d Resolution R l ti
• People • Governing Processes and Procedures, i.e. • Change Control • Maintenance, Administration, Support • Documentation and Document Management
What is ‘Infrastructure? • May include: • • • • •
Servers and Desktops Database Infrastructures Storage Devices (SANs) File Servers SLA and SLAs d OLA OLAs
• What else?
So what is ‘Infrastructure Qualification’? • Infrastructure Qualification - Establishing confidence that infrastructure process equipment, ancillary systems and sub-systems are compliant with appropriate codes and approved design intentions, are capable of consistently operating within established limits and tolerances. Infrastructure as scoped from above slides, does • Basically verifying the ‘Infrastructure’ what you said it was going to do and will continue to do so. • Ensuring you include: • Documented Evidence • High Degree of Assurance • Quality Attributes
Why Qualify the Infrastructure? • “The validated status of GXP applications that are dependent upon underlying IT Infrastructure is compromised if the IT Infrastructure is not maintained in a demonstrable state of control and regulatory g y compliance.” p • In theory, an application can not be considered validated unless it sits on a qualified infrastructure • Ensures you have good control over your baseline configuration and any changes to that baseline. • Ensures Ens res change impact is more accurately acc ratel assessed prior to implementation • The business and business applications (regulated and non-regulated) are dependant on the infrastructure and need assurance of it’s stability. • Ensures a positive cultural shift in understanding regulatory expectations within an IT organization.
REMINDER: GOOD SYSTEM DEVELOPMENT PRACTICE/GOOD BUSINESS PRACTICE
Infrastructure Qualification Planning • Things to consider when planning… • Most likely Retrospective in nature • Is typically an extensive effort (can last 2 2-3 3 yrs. depending on scope and resources) • Breaking the effort into manageable pieces is key • Be a aware, are it will ill result res lt in additional documentation doc mentation and possible rere design • It will involved the development and execution of multiple protocols
Infrastructure Qualification Planning • To Start… • Need to understand your organization • What does yyour organization g look like? • How do IT, IS, Application Development fit in? • Are all the infrastructure pieces centralized in one group? • Is application development centralized in one group? • Who and what depends on your infrastructure? • Is there a Quality Group that must be involved • Who are the key stakeholders?
Infrastructure Qualification Planning • Be sure… • To Identify the Sponsor and… • To Identify a preliminary cross-functional team • • • • • • •
System Engineers/Architect(s) Service Manager(s) Security Manager(s) Quality Assurance/Compliance I Impacted t dS System t Owners O Impacted User Representatives Impacted Business Representatives, as applicable
Infrastructure Qualification Planning • Perform an initial assessment of the current infrastructure, and document • Immediately create high level “as-built” diagrams to help determine scope including: • •
Servers, Ser ers routers, ro ters switches, s itches hubs, h bs firewalls, fire alls protocols, protocols and locations (IDF (IDF, MDF, MDF Data Centers etc.) WANs relationships with LANs
• Establish a master list of significant infrastructure components • Review any existing procedures and process to ensure usability and modifications, as needed • Conduct risk analysis of entire infrastructure including a graded identification of potential points of failure and gaps
• Map out an approach…in manageable pieces!
Qualification by Platform – One Approach • Break out the effort into ‘Platforms’ and specific ‘Platform’ qualifications efforts. efforts • Where Platform = “The underlying architecture of hardware and software configurations constituting the computing environment hosting specific services, components, and applications” • May include: • Network and Data Communications • Data Center • Desktops and Servers • Authentication and Authorization Services • Backup and Restoration • Storage Area Network
Qualification by Platform • Create one ‘Infrastructure Infrastructure Qualification Plan’ that… • Describes the overall effort (all platforms) • Includes the High-Level Diagram(s) • Identifies the Platforms and Scope of Each Platform Qualification • Prioritizes P i iti th the Pl Platforms tf iin tterms off criticality iti lit and dd dependencies d i • s/b based on Risk Assessments and Gap Assessments
• Includes references to and conclusions made from anyy assessments p performed • note: if you didn’t do them, DO THEM
• Identifies Platform Specific PMs and Teams
Qualification by Platform • Ensure it defines the standard methodology that will be used for each Platform Qualification, Qualification i.e. ie • Each will have a Qualification Plan (QP) • • • • •
Platform Description Scope Platform Impact and Risk Assessment Defined Qualification Activities Defined Roles/Responsibilities, etc.
• Each QP will include consistent steps such as: • • • • • • •
Plan Design Installation Q Qualification • IQ/OQ/PQ Ongoing Monitoring and Maintenance Procedure Development Training
• Each Platform Qualification will be summarized in a Platform Qualification final Summary Report
Qualification by Platform • In addition the Infrastructure Qualification Plan should… • Define consistency in: • Having identified documented inputs and outputs (note: documents may vary, but should be called out in the QP) • Defined, applicable reviews and approvals • Project Timelines, Milestones, Deliverables
• Ensure regular reporting to Sponsors and Stakeholders on the status of each Platform Qualification • Define Standard Procedures that run across all platforms • • • • •
Protocol development and execution methods Document Management Ch Change C Controll Training Procedure Development Processes
• Id Identify if Consistent C i A Approval l Expectations E i for f each h Pl Plan and d associated i d documents (i.e. PM, Business, QA, Tech.Rep.)
Time check and segue options… options • Change Management Management, Administration and Maintenance of your Infrastructure • Continue with Platform Planning • Discussion
Platform Qualification Planning and Execution • Documented Requirements • No matter what the platform, f high level requirements should be documented to help in planning the effort and scope. • These may include: • • • • •
High Level Diagrams User Requirement Specifications RFPs, Business Cases SLAs,, OLAs Maintenance Agreements
• May be one document, may be many • These should be created/verified before performing any next steps.
Platform Qualification Planning and Execution • Risk Assessments • Should S always be performed f as they: • Provide the documentation back-up for risk-based decisions (i.e. reduction in effort/scope)
• Are key in.. in • • • •
Defining the approach and scope Identifying potential design/process gaps Identifying y g critical mitigation g activities Assisting in risk and time management activities
• Note: Will show an example with case study
Platform Qualification Planning and Execution • Design Documents • Design Documents should exist for all Platform Qualifications • May include: • • • •
Design Reviews Development/Update of Specifications, Configuration Records Additional next level diagrams CMDB Data
• Mechanisms for ensuring they are maintained in a current state is critical
Platform Qualification Planning and Execution • Installation Activities: • Installation changes may need to occur as a result of your design reviews and Risk Analysis. • It is best to have a stable configuration baseline for your platform once you begin protocol executions. • Any changes should be made following formal change control procedures. • Some changes may also be incorporated into the Installation Qualification activities • Design and Requirements Documentation should be modified in accordance with any changes made
Platform Qualification Planning and Execution • Qualification Activities • Protocols should be developed in order to ensure expedited execution wherever possible • Templatize where possible (repeatable execution) • Pre Pre-Approved Approved templates for ongoing execution • Perform subset executions and perform configuration compares instead
• Identify early, what changes can be considered administrative • Allows changes g to move into Admin. SOPs • Removes them from requiring protocol execution in the future
Platform Qualification Planning and Execution • Qualification Activities • E Execution i • Protocols should be followed as written, changes documented per procedure • Execution most likely will not be in test environment so • Mitigate any potential impact
• Evidence and Results • Can’t reallyy gget awayy from the need for evidence • Be smart • Automated scripts • Reports vs. screen shorts • Cumulative activities evidenced 1 time
• Take advantage of your Prioritized Requirements and Risk Analysis • Some failures may be O.K.
Platform Qualification Planning and E Execution i • Qualification Activities • Summarize Results in Protocol Summary Reports • Will later be summarized in the Platform Qualification Summary Report
Maintaining the Qualified State • Most Important – Control Changes Made to the Infrastructure • • • •
Change g Control Procedures Administrative and Maintenance Procedure Auditing Tools Using a Configuration Management Database or Master System Inventory
• Develop Platform Specific Process/Procedures for: • Administration and Maintenance • Operation • Performance P f M it i g Monitoring
• Other Non-Platform Specific Process/Procedures • • • • •
Training Internal Auditing Incident Management Problem Reporting and Resolution Performance Monitoring
Q&A
Contact information: Email:
[email protected] Phone: 617 617-489-7319 489 7319