Project: AOS Technology Performance Audit

Project Number: AOS-2019-001
Location: Columbus

Purpose: To ensure that the Ohio Auditor of State's Office (AOS) is serving its clients and the taxpayers of Ohio in the most efficient and effective manner possible, AOS is soliciting proposals from qualified vendors to conduct a comprehensive performance audit of the AOS’s use of electronic tools and capabilities, including data analytics tools.

In the Financial Audit Division, AOS is requesting an analysis of emerging tools (Robotic Process Automation (RPA), Extract, Transform, Load (ETL), Artificial Intelligence (AI), and Machine Learning (ML), technologies and data analytics trends in external financial audits.

This audit will focus primarily on AOS’s use of audit data analytics and other computer-assisted audit techniques in performing risk assessment procedures, tests of internal controls, substantive analytical procedures, tests of details, and procedures to help form an overall conclusion related to specific audit objectives in financial audit matters to ensure the AOS optimizes the use of available resources and reflects best practices in the government auditing field.

RFP

Questions & Answers

Questions and answers will be added here and updated as they are received, with the most recent questions at the top. To submit a question about this project, email BIDQuestions@ohioauditor.gov.


Q1. Is there an expectation that vendors will travel to the government agencies that AOS audits or will the work be performed at a centralized location? If travel is required to government agencies that AOS audits, how many different locations does AOS anticipate the vendor will travel to?

A.  We do not anticipate the vendor needing to travel to governmental agencies that AOS audits. If the vendor requires information from our auditees, AOS will facilitate gathering that information centrally and will provide it to the vendor. The vendor may, however, need to travel to one or more of AOS’s seven regional office locations to interview AOS audit managers and staff. Regional interviews can also be conducted by phone or by using AOS teleconferencing and videoconferencing equipment in Columbus to minimize travel at the vendor’s option.

Q2. Can the final report and all deliverables be issued under the accordance of the AICPA Consulting Standards?
A.  The reference from the RFP was intended to make vendors aware of the standards AOS follows in the audits we perform. We ask that the successful vendor keep this in mind as it develops recommendations or provides advice on the tools, staffing, or process changes the AOS should make to enhance and transform its data analysis work in audits.

Conversely, the RFP does not specify the standards the vendor must follow for conducting the performance audit. This decision is left to the discretion of the vendor. Part Four of the RFP, however, does specify the deliverables and dates. We would consider AICPA Consulting Standards to qualify as “other similar standards” as described for the Final Audit Report deliverable in Part Four (i) of the RFP.

Q3. The scope of work is laid out in the RFP (Part Two: Scope of Work). Can AOS provide additional information on what is being requested in the Proposed Statement of Work (Section 5F)?
A. It is the discretion of the vendor to decide what information constitutes the Statement of Work; however, it is AOS’s assumption that the vendor’s Statement of Work will incorporate the Scope of Work presented in Part Two of the RFP.

Typically, a Statement of Work addresses project-specific activities, deliverables, timelines for a vendor providing services to the client, etc. It is also the vendor’s discretion to include in the Statement of Work any assumptions of which the AOS should be aware.

Q4. How long has D&A been a core component of the AOS audit process?
A. It is unclear what “D&A” is referring to. If it is intended to mean “Data & Analytics”, we have been performing planning, substantive, and final analytics since the 1990s.

However, a growing collection of data sources and stronger analytic tools are revolutionizing the way external audit firms conduct audits.
For AOS, embracing changes in the way we use data and analytics to enhance our audit risk assessments, testing, and conclusions is crucial to our office remaining efficient and effective in the audits we perform.

Q5.How much training have auditors received relating to D&A?
A. Generally some training on the use of analytics is provided every two to three years.

Q6. How many auditors are there in the AOS ISA Group?
A. Roughly 30.

Q7. Have there been any specific recognized challenges to adopting analytics?
A. The sheer volume, variety, and velocity at which data becomes available present technological challenges in how it is secured, stored, and analyzed.

We recognize that access to audit-relevant data, the availability of qualified and experienced resources to process and analyze the data, and the timely integration of analytics into the audit planning process in particular, will be a challenge as we transform and enhance our use of data analytics in our office.

Q8. Will the transcript from the vendor teleconference be posted?
A. No.

Q9. Can examples of mandatory audit staff trainings be provided?
A. Examples of mandatory audit staff trainings may be provided to the successful vendor. 

Q10. Refrencing page 7, Part 2.A.ii “People”, “Selected vendor shall study and analyze AOS audit staff’s capabilities related to using audit tools and techniques as well as evaluating the results of the outcomes to avoid making inappropriate assumptions, overlooking unusual circumstances, and over-generalizing in drawing conclusions.”
Considering AOS has roughly 550 audit staff, could you provide further clarification on the expected depth of procedures to complete the scope within the required 120 days?

A. AOS anticipates the successful vendor will interview an appropriate number of staff, managers, and technical professionals to evaluate the current state of AOS and recommend a plan for the future development and implementation of the appropriate electronic tools.

Q11. On page 12, Part 5.C.iv “Expertise”, “Vendor must identify the designated Engagement Manager and provide the following: references regarding the performance of the Engagement Manager.” Some states do not allow written references, would you be open to a direct phone conversation with identified references, in lieu of a formal write-up?
A. No, follow the instructions in the RFP.

Q12. On page 13, Part 5.D.i “Project Approach”, “Vendor must provide a description of its approach for conducting a performance audit of the scope requested.” provide the specific objectives that you’d like the vendor to audit against in its performance audit of the tools, people and processes?
A. Refer to the Executive Summary and scope of the RFP for a description of AOS objectives.

Q13. Is the selected vendor precluded from competing for related work with AOS in the future?
A. No.

Q14. Can you clarify the protocols AOS has in mind (if any) to select such states outside of Washington?
A. The National Association of State Auditors, Comptrollers, and Treasurers (NASACT) annually publishes “Auditing in the States," a state-by-state summary of data about state audit offices, key positions within each office, operations and functions. The Ohio Auditor of State (AOS) is a NASACT member and has access to this publication. 

Additionally, NASACT can provide AOS with a list of states (including contact information) that, in their assessment through participation in various audit and technology committees, could be comparable to our office. 

A potential vendor could use these resources to identify comparable peer states, considering whether they perform external financial audits, employ data analysis tools, and perform audit data analytics that substantially enhance efficiency and/or effectiveness, particularly in the planning and risk-assessment phases.

Q15a. How many Uniform Accounting Network (UAN) entities are among the 5,900 offices within the AOS scope?
Q15b. Can you provide further information around Uniform Accounting Network (UAN) entities?
A. The Uniform Accounting Network (UAN) is a financial management system developed by the AOS that is designed specifically for Ohio local governments — Townships, Villages, Libraries, Special Districts and Cities. UAN consists of five modules: Accounting, Payroll, Inventory, Budget, and Cemetery. UAN is used by 2,043 entities that are audited biannually. The table below describes the entities utilizing UAN. Audit Workbench is an add-on to the UAN application, also developed by the AOS, that is used by AOS auditors and IPAs for the audit of UAN entities.

  UAN Clients % on UAN
Townships 1,224 94%
Villages 506 74%
Libraries 159 63%
Special Districts 152 26%
Cities 2 1%

Q16. This project is labeled a “performance audit.” As such, does the AOS expect vendors to follow GAO yellow book standards?
A. The RFP did not specify. We are leaving it up to the vendor to decide what standards it will conduct the project under.

Q17. If a vendor conducts the services under GAO yellow book standards to maintain independence with the AOS, will the AOS consider awarding this contract to a vendor which also provides audit services under contracts with the AOS?
A. Yes.

Q18. In RFP scope of work page 9 section B iv, the AOS asks for advice related to legislative changes that may be necessary. Will the AOS supply the vendor with the AOS legal counsel’s time to make these determinations?
A. We would consider making Legislative Affairs services available to assist with a legislative analysis.

Q19. Will the AOS share the names and titles of individuals on the review committee to evaluate vendor responses?
A. No.

Q20. Does the AOS have a budget reserved for this project?
A. No.

Q21. Does the AOS have a preference for how vendors can propose optional additional services, such as optional implementation support?
A. Interested vendors should follow the criteria specified in the RFP.