Paraguay ECT Final Report
i
Office of
Technical
Assistance:
Rwanda
Revenue
Program
Program Evaluation
Report
Final Submission
Aug 1, 2023
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
i
Table of Contents
I. EXECUTIVE SUMMARY .................................................................................................. 1
KEY FINDINGS AND CONCLUSIONS .............................................................................................. 1
KEY RECOMMENDATIONS ............................................................................................................ 3
II. EVALUATION OVERVIEW .......................................................................................... 5
PROGRAM DESCRIPTION .............................................................................................................. 5
EVALUATION PURPOSE ................................................................................................................ 6
EVALUATION QUESTIONS ............................................................................................................ 7
EVALUATION METHODOLOGY ..................................................................................................... 7
DATA COLLECTION ...................................................................................................................... 8
QUALITY ASSURANCE .................................................................................................................. 8
LIMITATIONS ................................................................................................................................ 9
III. FINDINGS AND CONCLUSIONS ................................................................................. 9
PROGRAM COMPONENT 1: KEY PERFORMANCE INDICATORS (KPIS) ......................................... 10
PROGRAM COMPONENT 2: COMPLIANCE IMPROVEMENT PLANNING .......................................... 13
PROGRAM COMPONENT 3: TAX DEBT COLLECTIONS ................................................................. 15
PROGRAM COMPONENT 4: SPECIALIZED SECTOR AUDIT............................................................ 22
PROGRAM COMPONENT 5: COMPUTER-ASSISTED AUDIT ........................................................... 25
PROGRAM COMPONENT 6: GENERAL AUDIT CAPACITY ............................................................. 28
PROGRAM COMPONENT 7: TAX DISPUTE RESOLUTION .............................................................. 32
PROGRAM COMPONENT 8: INTERNAL AUDIT ............................................................................. 36
PROGRAM COMPONENT 9: RISK MANAGEMENT (INFORMATION TECHNOLOGY RISK
MANAGEMENT) .......................................................................................................................... 39
EVALUATION QUESTION 3 ......................................................................................................... 44
IV. PROGRAM LEVEL RECOMMENDATIONS ........................................................... 44
PROGRAM DESIGN CONSIDERATIONS ......................................................................................... 44
STAFFING AND ONBOARDING CONSIDERATIONS ........................................................................ 46
V. ANNEXES ........................................................................................................................... 48
ANNEX A. LIST OF DOCUMENTS REVIEWED .............................................................................. 48
ANNEX B. KEY INFORMANTS INTERVIEWED .............................................................................. 49
ANNEX C. EVALUATION TEAM .................................................................................................. 50
ANNEX D. EVALUATION TEAM’S RESPONSE TO THE OTA REVENUE TEAM’S COMMENTS ......... 51
ANNEX E. OTA RESPONSE TO “OFFICE OF TECHNICAL ASSISTANCE: RWANDA REVENUE
PROGRAM EVALUATION REPORT” ............................................................................................. 52
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
ii
This report was prepared by Bixal
August 1, 2023
Bixal Solutions, Inc.
3050 Chain Bridge Road, Suite 305
Fairfax, VA 22030
[email protected]
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
iii
ACRONYMS & ABBREVIATIONS
CAA
CAS
CIP
DTD
EO
EQ
FATAA
IA
ICTD
IRS
IT
KII
KPI
Logframe
M&E
MIS
OTA
QRS
RRA
TA
TADAT
TIATA
USAID
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
1
I. EXECUTIVE SUMMARY
From 2015–2020, the U.S. Treasury’s Office of Technical Assistance (OTA) engaged with the
Rwanda Revenue Authority (RRA) to increase domestic resource mobilization in support of
Rwanda’s economic and development priorities. OTA focused on building technical capacity in
nine program components: 1) key performance indicators (KPIs) and reports; 2) compliance
improvement planning; 3) debt collection; 4) specialized sector audit; 5) computer assisted audit;
6) general audit; 7) tax dispute resolution; 8) internal audit; and 9) risk management, including
information technology (IT) risk management. Over the course of the program, OTA provided
technical assistance by periodic deployment of short-term expert advisors who worked closely
with RRA counterparts.
In compliance with the Foreign Aid Transparency and Accountability Act (FATAA)
requirements, Bixal
1
conducted a third-party, summative, program evaluation of OTA’s Rwanda
Revenue Authority Program, under contract to OTA. The evaluation of the OTA/Rwanda
revenue sector program is meant to provide OTA with in-depth, third-party, retrospective
insights of its activities in Rwanda and lessons learned for future programming.
The qualitative evaluation methods included a desk review of key project documents and 22 key
informant interviews (KIIs) with the OTA advisors and RRA counterparts.
The evaluation focused on the following evaluation questions (EQs):
EQ 1. Which of the expected project outcomes were achieved?
EQ 2. Which of the achieved outcomes have been sustained up to the present?
EQ 3. OTA took a new approach to program design in Rwanda by involving
counterparts in the design and writing of key operational manuals. Did OTA’s new
approach contribute to the sustainability of the outcomes, and if so, how?
EQ 4. For any expected outcomes that were not achieved, which factors hindered
success?
Key Findings and Concl u sions
PROGRAM COMPONENT 1: SUPPORT FOR THE DEVELOPMENT OF KEY
PERFORMANCE INDICATORS (KPIs)
While RRA had already adopted some KPIs (related to audits) at program inception, OTA’s
assistance was limited to some training for audit and debt staff. Further development of the KPI
component was limited because RRA was planning to implement a new Management
1
www.bixal.com
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
2
Information System (MIS) system that will support expanded collection and reporting of KPI
data. The evaluation team found that data already collected by RRA were not currently utilized
to their full extent—data were used at the managerial level but not at a corporate or strategic
level.
PROGRAM COMPONENT 2: COMPLIANCE IMPROVEMENT PLANNING
(TAXPAYER)
OTA’s engagement with this component was limited because it was discovered that RRA
already had a compliance improvement plan (CIP) in place. However, OTA initiated an idea that
RRA commission a taxpayer behavior study, which was commissioned late in the program cycle
(2020). OTA assisted with updating the CIP as informed by the study findings.
PROGRAM COMPONENT 3: TAX DEBT COLLECTIONS (ARREARS)
OTA provided overall guidance on debt policies to be incorporated into RRA’s tax debt manual,
including extending the payment period for delinquent accounts. However, the evaluation team
concluded that OTA’s advice in this case may lead to increased delinquencies, as some taxpayers
may take advantage of the extended leniency and withhold tax payments. They also concluded
that OTA’s support did not help RRA address a growing issue with the lack of write-offs that
distorts RRA’s accounting. The RRA needed to revise its write-off approval processes before
they could apply OTA’s technical advice which led to delays. Finally, OTA introduced the use
of a timesheet to track “productive” time (engaged in cases) and “unproductive” time (meetings,
travel, etc.) of auditors. However, the timesheet was not well received and is only used to
measure auditors’ performance rather than improve the efficiency of RRA’s audit department.
PROGRAM COMPONENT 4: SPECIAL (BUSINESS) SECTOR AUDITS
OTA provided specialized training to a select group of RRA’s auditors on the construction sector
businesses. However, RRA deployed the trained auditors back to their previous general-sector
audit work and, as a result, the newly trained auditors lost the ability to apply and develop their
newly acquired skills to the construction sector.
PROGRAM COMPONENT 5: COMPUTER ASSISTED AUDITS
While providing some training to prepare key RRA staff for future adoption of computer-assisted
audit practices, OTA did not achieve most of its objectives for this component because RRA did
not have an MIS or software in place to support computer-assisted audits.
PROGRAM COMPONENT 6: GENERAL AUDIT CAPACITY
OTA’s key achievement under this component was supporting RRA to produce a high-quality,
detailed audit manual. This manual has been digitized by RRA and is frequently updated to
maintain its utility and relevance. Moreover, RRA has since initiated the development of other
procedures manuals without inputs from OTA. In addition, OTA introduced enhancements for
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
3
RRA’s quality review process for audits that remain updated and useful. However, the
evaluation team concluded that OTA’s recommended enhancements to the audit manual may
have been too detailed and included too many prescriptive procedures. While the inclusion of
detailed audit cases was intended to eliminate ambiguity across a wide range of audit cases,
OTA’s approach should have been to create high-level guides and build capacity of the auditors
on how to make decisions.
PROGRAM COMPONENT 7: TAX DISPUTE RESOLUTION
OTA helped draft new dispute resolution policies, advised on legal support for consistent, fair
and effective dispute processes, and provided training to enforcement staff on improved
procedures. However, despite these outputs, RRA’s dispute resolution process remains largely
the same because not all affected departments were consulted in this process. More importantly,
there was inadequate outreach and education to taxpayers to raise awareness of the changed
resolution process.
PROGRAM COMPONENT 8: INTERNAL AUDIT
OTA reviewed RRA’s internal audit procedures, recommended revisions including enhanced
reporting, and developed a process strategy for addressing RRA’s backlog in aged, unaddressed
past internal audit recommendations. However, OTA did not follow up with internal audit
training or support for related interdepartmental communication practices that would capitalize
on the assessment activities and support RRA to engage internal audit practices more fully.
PROGRAM COMPONENT 9: RISK MANAGEMENT (INFORMATION
TECHNOLOGY RISK MANAGEMENT)
OTA assessed RRA’s physical and IT risk environment, supported RRA in developing a risk
register used to identify and manage risks, and provided risk management training to appropriate
RRA staff on a regular basis. Follow-up activities included building security enhancements,
using staff key cards to control access to different building floor sections, and developing an IT
security plan for equipment, password, and network security practices. However, the evaluation
team found that security measures that protect taxpayer confidential records remains weak,
indicating that IT security remains at risk. Likewise, a comprehensive risk management plan for
taxpayer information and records that includes risk reduction and disaster recovery planning is
needed.
Key Re co mmen datio ns
While OTA did make progress in achieving some outputs and outcomes at the component level,
the evaluation team found that the design and approach to implementation could be strengthened.
This section contains high-level recommendations that relate to Rwanda but may also be
applicable to OTA’s work more broadly.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
4
PROGRAM DESIGN CONSIDERATIONS
Theory of Change
A theory of change (or similar) should have been created at an early stage of the project. The
process to create the theory of change would have helped the participants to better identify the
development needs and to obtain a common understanding of a feasible way forward. The
process should have followed a common process, using standard formats and terminology.
Benchmarking at Program Inception
The theory of change could also serve as the basis for establishing a monitoring framework to
measure progress towards outputs and outcomes over the course of the project implementation.
Baselines and targets should be included to set expectations and to serve as a point of
comparison.
Systems Approach
The evaluation team recommends that OTA advisors take more time to understand the
counterpart’s organization, business processes, and priorities prior to developing a workplan.
This would ensure the organization is ready for the activity and is able to implement the changes
and sustain them after the OTA engagement. This system’s approach would also ensure OTA
advisors understand the interdependencies and can address barriers to specific changes.
STAFFING AND TRAINING CONSIDERATIONS
Selection of Advisors
Many OTA advisors were very senior but lacked practical day-to-day operations expertise to
translate recommendations to frontline workers to implement the reforms. OTA should consider
the composition of the teams of advisors to encompass a wider range of experiences and
backgrounds. While OTA engaged a local project coordinator to provide communications,
logistics, and administrative support for the program, the evaluation team recommends that OTA
also needs a higher-level technical role with specific country knowledge and capable of working
across multiple program component areas.
Orientation Package
The evaluation team also identified that an orientation package for the intermittent advisors is a
missing component of training. It should have included information from RRA staff on
organizational structure and identified issues, business processes, and information on Rwandan
cultural and their political and legislative process. Technical assistance would be more effective
if local cultures were understood.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
5
II. EVALUATION OVERVIEW
Pr og ram Des cripti on
Since 2015, the U.S. Department of the Treasury’s OTA has provided technical assistance in
public financial management to Rwanda using a resident-based project coordinator and
intermittent expert advisors who worked closely with counterparts in RRA. The goal of the
project has been to support RRA's efforts to increase domestic resource mobilization in support
of Rwanda's economic and development priorities by building its capacity in nine areas:
1. KPIs and reports.
2. Compliance improvement planning.
3. Debt collection.
4. Specialized sector audit.
5. Computer assisted audit.
6. General audit.
7. Tax dispute resolution.
8. Internal audit.
9. Risk management, including IT risk management.
At the time this project began, RRA was reasonably well developed, compared to its regional
counterparts, and had motivated, educated employees. Third-party Tax Administration
Diagnostic Tool (TADAT)
assessments were conducted in both 2015 and 2019.
2
These
assessments identified some deficiencies to be addressed by RRA. Some of the main issues
identified were:
• RRA’s strong performance of promoting voluntary compliance was outweighed by the
relatively weak execution of audits and debt management functions, resulting in a high
rate of tax payment arrears.
• RRA had an established risk management structure but lacked the depth needed for a
comprehensive compliance improvement strategy, and it faced challenges to identify and
mitigate corporate risks, especially those in the IT sector.
• The cumbersome dispute resolution process required high-level internal approvals of
relatively minor individual taxpayer objections, causing undue delays.
• Auditors did not have the specialized skills required to audit specialized sectors, such as
the construction industry, or engage in complex computer-assisted audits.
2
Internationally recognized tax administration assessments conducted by the TADAT organization.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
6
• Internal auditors were not trained to conduct high-quality audits, and there was no
overarching process to monitor departments’ responses to audit recommendations.
• RRA did not use KPIs to monitor organizational progress.
According to the OTA advisors, these assessments informed the annual work plans, which in
turn constituted the basis for the drafting of the statements of work that outlined the activities for
OTA advisors engaged with RRA. Advisors’ intermittent visits typically lasted two weeks. The
visits included following up on the advice given during an earlier visit, discussions with RRA
staff, new assessments of RRA capacities and operations, and the presentation of new or updated
advice to the senior management at RRA.
A Program Manager who also served as an advisor coordinated OTA’s activities and provided
technical oversight and produced monthly and annual reports.
At the close of the program, the OTA team provided RRA’s Commissioner General with future
recommendations. The recommendations focused on RRA’s Domestic Tax Division (DTD) in
the areas of quality assurance, human resources, administration, and training. The OTA team
also drafted a six-month action plan to implement the recommendations. The registers and the
action plan were included with the final Closing Memo to the Commissioner General.
Ev al uati on Purpose
OTA commissioned Bixal to carry out a summative program evaluation. The evaluation
activities took place during January–July 2023, with a field visit by the evaluation team to
Kigali, Rwanda, during the period of March 23–April 9, 2023.
This summative program evaluation was designed to serve two purposes. The first is for OTA to
comply with the Foreign Aid Transparency and Accountability Act (FATAA) of 2016.
3
The
second is to provide OTA with in-depth, third-party, retrospective insights of its project activities
in Rwanda. Some aspects of OTA’s experience in Rwanda may be extrapolated to other current
or future projects in Rwanda or other countries to inform adaptation, improve projects, discover
ways to reduce the time to achieve project outcomes, identify new ways to effect positive
change, and document project achievements and success.
3
Foreign Aid and Transparency and Accountability Act of 2016, Public Law 114-191
https://www.congress.gov/114/plaws/publ191/PLAW-114publ191.pdf
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
7
Ev al uati on Q uesti ons
Bixal’s evaluation team collaborated with OTA’s revenue and RRA teams to develop the
following evaluation questions (EQs). The EQs provided the foundation for the remaining
design elements of the evaluation, such as methodology and approaches to data collection and
analysis.
EQ 1. Which of the expected project outcomes were achieved?
EQ 2. Which of the achieved outcomes have been sustained up to the present?
EQ 3. OTA took a new approach to program design in Rwanda by involving
counterparts in the design and writing of key operational manuals. Did OTA’s
new approach contribute to the sustainability of the outcomes, and if so, how?
EQ 4. For any expected outcomes that were not achieved, which factors hindered
success?
The evaluation findings found in Section III are grouped by OTA’s nine program components,
each of which includes multiple objectives. For each program component objective, the report
addresses findings, conclusions, and recommendations for EQ 1, EQ 2, and EQ 4. Note that
EQ 3, a high-interest line of inquiry for OTA, is focused on assistance with developing
operational manuals and is broken out and discussed explicitly at the end of Section III. The
evaluation team determined that EQ 3 findings are integral to several program components, so
EQ 3 is largely addressed in context together with the other EQs.
Ev al uati on M et h odology
The evaluation methods included a desk review of nearly 350 project documents provided by
OTA, followed by individual interviews of OTA’s advisors, their counterparts, and other key
respondents. The overall evaluation design was developed using a theory-based approach with
OTA’s program logical framework (logframe) as a starting point, which divided the program into
nine project components. The evaluation team produced an evaluation framework (see Annex
A) that mapped specific areas of inquiry for each of the nine project components. The evaluation
team used this to develop detailed interview guides that were tailored for each KII. The KIIs
provided multiple perspectives and sources that support findings related to the EQs.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
8
Da ta Coll ecti on
OTA and RRA shared vital project documents with the evaluation team. Among these
documents were the terms of reference for the program, the program logframe, workplans, trip
reports, statements of work, monthly reports, project narratives, project performance reports, a
debt management manual, closing memo, the end of project review, and an evaluation
coversheet. Annex B contains a detailed list of the documents used by the evaluation team. The
team reviewed each document to provide context for the program, as well as to elicit findings to
add to the base of evidence.
The desk review was complimented by 22 KIIs, 18 with RRA officials and 4 with the OTA
advisors. Of the 22 interviews, 15 were conducted in person and seven were conducted using
virtual technology. A detailed list of key respondents is included in Annex C.
DATA ANALYSIS PROCESS
The evaluation team conducted qualitative analysis of KII data on an ongoing basis, concurrent
with interviews, to ensure quality in real time and to identify findings and trends in a timely
manner. Interview notes were completed by the team members and voice recordings were made
with the respondent’s consent. A selection of KIIs were also transcribed to cross-check and
validate for coding.
KII notes and transcriptions were coded and analyzed using ATLAS.ti, grouping similar data
under different categories and themes. Thus, the evaluation team could locate, retrieve, and
combine the textual data that corresponded to a category of interest. These data were further
triangulated and contextualized with the findings from the desk review.
When the data collection was nearly complete, the evaluation team members convened for a joint
analysis. The evaluation team discussed potential findings, conclusions, and trends. Insights and
ideas from this process were shared in a presentation of preliminary findings to OTA for vetting
and discussion before the final evaluation report was drafted.
Qualit y Assur a nce
The evaluation team followed standard quality assurance best practices that included the
following:
1. Notetakers uploaded their notes at the end of each day of data collection. Notes were
reviewed to ensure completeness and comprehension and to identify emerging themes.
2. The evaluation team met at the end of each day of data collection in Kigali. The team
discussed what they heard and observed to ensure consistency in collecting and reporting
data.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
9
3. The ATLAS.ti application enabled the team to check for accuracy and completeness of
qualitative data as they were entered and coded.
4. The joint analysis and presentation of preliminary findings provided further opportunity
to cross-check and validate data and findings.
The evaluation team took steps to ensure data security, including using secure communication
platforms, storage of information on secure and password-protected sites, and deletion of
recording and transcription files after the completion of the study.
Limita tion s
In general, the evaluation team faced very few limitations. Most KIIs could be carried out as
planned and sufficient documentation was provided for a comprehensive desk review. However,
a few circumstances limited the data collection and consequently the analysis:
• Lack of program framework, action plan, coherent documentation
• The Logframe was designed very late into the project and did not include sufficient
indicators that could help measure progress over the life of the project.
• For some respondents, it was difficult to remember, in detail, the activities that OTA had
carried out and to what activities OTA advisors had contributed.
• Some staff had left RRA and were not available for interviews.
• In most cases, KIIs with Rwandan counterparts were conducted offsite, outside their offices.
This was due to both the lack of quiet meeting spaces, a result of ongoing construction in
RRA facilities, and the counterparts’ frequently stating their preference for interview privacy
away from their colleagues. While this had some advantages for candid discussions with the
evaluation team, it also limited opportunities for direct observation of RRA’s workspace and
environment.
• The structure of the workplans changed from year to year and were not fully compatible with
the program logframe. This made longitudinal tracking of achievements difficult.
III. FINDINGS AND CONCLUSIONS
Section III presents each of the evaluation findings for OTA’s nine program components for the
RRA program. Section IV discusses additional conclusions and recommendations at a systems
level and considers OTA’s overall program.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
10
Pr og ram Compon e nt 1: Key Per fo rm a nce I ndicators (KPIs)
COMPONENT 1 OBJECTIVE
Component 1 focused on assisting RRA’s leadership in adopting and using KPIs to monitor
and manage organizational performance, set expectations, and allocate resources in key tax
administration functions, especially in audit and debt management. The planned project
activities included: 1) designing KPIs; 2) identifying data sources; 3) training staff on collecting
KPIs; and 4) training on how to analyze KPI data.
In assessing this objective, the evaluation team considered that KPIs need to be developed
carefully to align with activities, provide baselines and targets, identify data sources, collect data,
and, finally, analyze and communicate data to decision makers to contribute to the operation of
the program.
Table 1 summarizes the findings for Component 1, and a detailed discussion follows the
summary.
Table 1: Component 1 Outcomes—Activities and findings
Expected Outcome 1.1: RRA develops and regularly collects KPI data. RRA leadership uses
KPI data to measure the efficiency/effectiveness of core functions (i.e., audit and debt
collections), set expectations for organizational performance, and allocate resources in
accordance with organizational goals.
Activities
Findings
1.1.1: Assist RRA Planning and
Research Director to identify
and design RRA KPIs
• RRA was collecting some KPI data manually, mostly
focused on the number and types of audits.
• OTA assisted RRA to update and further develop an
existing KPI template.
• RRA updated KPI measures with OTA’s support.
• Counterparts reported satisfaction with OTA’s assistance
and that department managers did continue to use KPIs.
However, KPIs were not used at the corporate or
strategic level.
1.1.2: Identify data sources to
enhance KPIs
• OTA assisted RRA to identify relevant data; however,
they did not have the appropriate MIS in place to begin
collecting the information.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
11
1.1.3: Train specialized audit,
debt, and planning and research
staff to collect KPI data
• Some audit and debt staff were trained in the use of
KPIs, but this was on a limited scale.
• This activity was delayed because the MIS needed to be
developed prior to the training.
1.1.4: Train management to
analyze and use KPIs
• OTA was not able to train management staff on KPIs, as
the MIS system for collecting more strategic level KPIs
was not yet in place.
COMPONENT 1 FINDINGS
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Partially achieved
The outcomes for Component 1 were partially achieved. The evaluation team found that RRA
has developed KPIs and regularly collects KPI data at the managerial level. These data were
collected manually and tracked in an Excel document. The information collected was minimal
and, during the tenure of the OTA project, RRA leadership did not use KPI data to set
expectations for organizational performance or to allocate resources. KPI data continue to be
used only to perform basic revenue functions (e.g., measuring the efficiency and effectiveness of
the core functions of audit and debt collections) but do not inform higher-level decision-making.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Outcomes Sustained: Monthly KPI reports
The evaluation team found, through counterpart interviews, that RRA continues to produce five
monthly KPI reports that were supported by OTA advisors, so this practice is sustained. This
information can inform day-to-day operations; however, without a broader organizational effort,
there continued to be missed opportunities.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: 1) No single, coordinated technical assistance action plan; 2)
Lack of organizational level MIS for KPI data collection and analysis
The evaluation team found significant differences regarding the formulation of the achieved
objectives as reported in the End of the Project Review Document compared to the objectives for
each program component in the project logframe document. This indicated that OTA did not
follow a single, coordinated technical assistance action plan among the multiple advisors to
guide activities and track progress across the duration of the program.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
12
Additionally, the evaluation team learned from counterpart interviews that RRA lacked an MIS
with the functionality to collect and consolidate KPI data across the organization. Due to this,
OTA was unable to provide training to staff on how to produce usable analysis that leadership
could use to set expectations for organizational performance and allocate resources in accordance
with organizational goals.
COMPONENT 1 CONCLUSIONS
OTA provided relevant advice to RRA on Component 1, but RRA lacked relevant IT
infrastructure to execute fully.
The evaluation team found that RRA had been using some KPIs to track audits prior to OTA’s
assistance and RRA was interested in further developing additional indicators and using data to
inform management decisions. Unfortunately, RRA lacked an organization-wide MIS to collect
and process KPI data. OTA provided relevant advice and RRA retained one Oracle contractor to
make improvements to their MIS. Overall, this component is still in progress toward OTA’s
anticipated outcomes indicated in the program logframe. Training did not happen as planned as
the MIS was not yet in place.
COMPONENT 1 RECOMMENDATIONS
Continue improvements in reaching and sustaining outcomes by adopting a formal approach to
KPI technical assistance; forming a dedicated unit of technical experts focused on knowledge
management and implementation; collaborative use of a plan/tracker; clearly defining future
steps to achieving outcomes and measuring progress.
A formal approach by OTA to delivering technical assistance on the development and use of
KPIs could have produced more substantial and sustainable outcomes. For example, when RRA
requested support on KPIs, securing agreement on the use of KPIs from all management levels at
RRA would have achieved the expected outcome under Component 1.
The formation of a dedicated unit composed of technical experts to track, manage, and
implement OTA’s advice and continue implementing the new knowledge gained from OTA’s
support may have provided better and more sustained outcomes for RRA. RRA maintained their
own internal tracker, but we recommend using an agreed-upon action plan/tracker between
counterparts and OTA for any future programming.
Structuring the work plan, activities, outputs, and outcomes in a way that is clearly defined and
can be measured over time is key to making this sustainable. The new templates are a step in
this direction, but the need remains for further buy-in at the different levels of the organization,
training to operationalize, and a system to ensure accurate analysis and use.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
13
Pr og ram Compon e nt 2: Complia nce I mpro v eme nt Plan ni ng
COMPONENT 2 OBJECTIVE
OTA’s objective was to improve voluntary taxpayer compliance by identifying and
mitigating barriers. Project activities included assisting RRA to develop a comprehensive CIP
that addresses different steps of the process including taxpayer registration, tax filing, payment,
and mitigating accuracy in reporting/filing. OTA also advised RRA on how to identify and
mitigate revenue risks.
Table 2 summarizes the findings for Component 2, and a detailed discussion follows the
summary.
Table 2: Component 1 Outcomes—Activities and findings
Expected Outcome 2.1: RRA to research taxpayer behaviors and attitudes to understand barriers to
voluntary compliance in the areas of registration, filing, and payment. RRA adopts and implements a
CIP to increase taxpayer registration and mitigate filing and payment accuracy risks.
Activities
Findings
2.1.1: Develop compliance
research plan
• OTA suggested RRA conduct a study to research taxpayer
behaviors. RRA conducted the study with help from the
International Center for Tax and Development (ICTD) in 2020.
OTA did not contribute directly to the research.
2.1.2: Develop a CIP
• RRA already had a well-developed CIP. OTA contributed to
the plan by recommending the audit teams be better focused
and trained before attempting to work in the field, but this
activity was limited in scope.
Expected Outcome 2.2: RRA identifies areas that need additional focus on an ongoing basis and
dedicates resources to mitigation activities.
2.2.1: Assist RRA to develop
ongoing monitoring
methodology
• Since the CIP was not operationalized until after the ICTD
study, OTA was not able to complete this activity. The
evaluation team found that identification and mitigation of
compliance risks through the CIP was not being monitored.
OTA provided guidance for developing risk reports. However,
senior management had not used the risk reports to inform their
annual workplans or measure performance against the plan.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
14
COMPONENT 2 FINDINGS
EQ 1: Which of the expected project outcomes were achieved?
Outcome Status: Partially achieved
Overall, OTA’s expected outcomes under this component were partially achieved. At the
suggestion of OTA, RRA engaged ICTD in 2020 to conduct research on taxpayer behaviors and
attitudes—to better understand barriers to voluntary compliance in the areas of registration,
filing, and payment. Since this study was not conducted until near the end of OTA’s program in
2020, there was not enough time within OTA’s program cycle to incorporate the findings.
However, the evaluation team learned from counterpart interviews that RRA recently used the
findings from the ICTD study to update and implement a revised CIP to increase taxpayer
registration and mitigate filing and payment accuracy risks. OTA helped identify some areas
that needed additional focus and, in a final memorandum, recommended that RRA dedicate
additional resources to the mitigation of the risks that arise from taxpayer noncompliance.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Outcomes Sustained: Improved voluntary taxpayer compliance
Counterparts reported to the evaluation team that voluntary taxpayer compliance has improved,
although RRA is not measuring taxpayer compliance against an established baseline, and
quantitative data as evidence for improved compliance was not available.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: Sequencing and timing
OTA’s contributions were limited due to the sequencing and timing of events. RRA needed data
to design effective activities. While RRA acted on that recommendation, research was not
available until after the OTA program cycle.
COMPONENT 2 CONCLUSIONS
Resource allocation decisions need to factor in country context and best practices for optimal
allocation (e.g., number of auditors per case)
OTA’s contributions to this Component 2 were minimal as RRA already had a CIP and a process
to update and revise it in place. Nonetheless, OTA suggested conducting a study researching
taxpayers’ behaviors that could inform and improve compliance behaviors. Since this study was
only conducted in 2020, OTA’s engagement was over prior to the findings being available.
For better resource allocation, OTA recommended a reduced number of auditors assigned to each
case (lowered from a team of two auditors per case to only one auditor per case). The
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
15
recommendation was adopted by RRA without the necessary investigation. In our view, RRA’s
practice of assigning two auditors per case was providing a safeguard against corruption and
malpractice that often arises when there is only one auditor for a case. Corruption mitigation
plays a significant role in the allocation of resources to the revenue authorities in most
developing economies and OTA might have discussed with RRA to investigate the logic behind
the choice of two auditors per case. We iterate this gap under program-level recommendations,
which highlights the need for OTA to better align expertise and advice with the best practices
suitable to the country context and consider the reasons that lead RRA to making some of these
operational decisions.
COMPONENT 2 RECOMMENDATION
For increased efficiency, stage efforts to complete joint planning, target and goal setting, and
monitoring framework prior to start of project activity
Prior to initiating development work of this type, joint planning should be completed before the
engagement—this should include a desk review and interviews. This planning process would
help identify data gaps and the need for further research, which could be done prior to the start of
the OTA engagement. This would save valuable time and increase efficiency of the OTA
advisors.
In addition, in line with the first component, establishing a baseline and setting up targets and
milestones to achieve those targets would help quantify gains due to OTA’s technical assistance
efforts.
Pr og ram Compon e nt 3: Tax De bt Co llecti ons
COMPONENT 3 OBJECTIVE
OTA’s objective for this project component was to guide the development of standardized
procedures for increasing RRA collection of established tax debts. The planned project
activities included revising tax debt policies and procedures, developing a case selection process,
updating the procedural manual, training employees in the use of the manual, designing
management information reports, training staff to analyze management information reports,
designing timesheets, and training management in the analysis of timesheet data. Due to the
relatively large scope of OTA’s activities under this component, the findings are categorized into
three outcome areas under Component 3.
COMPONENT 3; OUTCOME 3.1 FINDINGS
Table 3 summarizes the findings for Component 3; Outcome 3.1 and a detailed discussion
follows the summary.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
16
Table 3: Component 3; Outcome 3.1—Activities and findings
Expected Outcome: 3.1
RRA adopts and implements revised tax debt policies, procedures, and manual to select cases with
a higher likelihood of collectability, resulting in increased revenues from collections.
Activities
Findings
3.1.1: Assist RRA in
revising tax debt policies
and procedures, including
write-offs of uncollectible
accounts, installment
agreements, improving
data on tax arrears.
• OTA provided overall support on debt policies, including
revising the tax debt manual and guiding RRA to put in place
standard operating procedures.
• Of note, under advisement by OTA, RRA extended the
maximum possible repayment period for tax debtors from 12
months to 36 months.
• Support for the write-off of uncollectible debts did not
materialize and support for improving data on tax arrears was
minimal.
3.1.2: Assist RRA to
develop a tax debt case
selection process
• OTA supported RRA to revise or streamline procedures for case
selection and case management; this led to a higher likelihood of
debt collectability.
• OTA recommended a matching program that would offset debts
against government contract payments, but it was not
implemented because the e-Tax case management system was
not sophisticated enough to accept the changes. This suggestion
was recently submitted to the Ministry of Finance and is
awaiting approval.
3.1.3: Draft updates to the
Debt Management
Procedures Manual
• OTA’s most significant contribution was the support provided by
advisors to develop and update the debt management manual.
• RRA continues to have out-of-date debt write-off procedures in
its policies. The process is cumbersome and involves approvals
from the top level of government.
3.1.4: Conduct training for
debt management
employees in use of the
revised manual
• OTA provided brief training to the enforcement officers and
managers in the use of the revised debt manual.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
17
EQ 1: Which of the expected project outcomes were achieved?
Outcome 3.1 Status: Partially achieved
Outcome 3.1 was partially achieved. OTA assisted RRA in streamlining the audit and
debt management processes for increased efficiency. One of the major achievements under this
activity was the reduced level of involvement by the senior management in audit cases and their
approvals. This delegation and streamlining of functions resulted in reduced interactions
between managers and taxpayers in the audit and debt departments, which led to more efficient
processes.
OTA developed a memo with recommendations for the RRA legal department to consider
increasing the use of legal actions to recover money from the largest debtors. This includes
garnishment actions, seizure and sale of assets, and other significant enforcement measures that
better utilize already available tools. OTA recommended improvements to tax arrears
management data as well to inform RRA’s process.
EQ 2: Which of the achieved outcomes have been sustained up to present?
Outcomes Sustained: Streamlining of the audit and debt management process and use
of updated collections manual
RRA has maintained changes to the audit and debt process that made it more efficient. RRA
does not have benchmark data prior to the changes, which will make it difficult to quantify the
specific resulting impacts on revenue. The Rwandan Parliament amended the law to lower the
penalty and interest rates on arrears to encourage delinquent taxpayer compliance. This action
will ensure application; however, it still needs to be incorporated into policy at RRA.
Based on feedback from interviews, the collections manual is being used by RRA staff in its
updated form.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: Approval delays and OTA’s broad scope
Some of the changes that OTA recommended are not yet enacted due to high-level decision-
making. For example, the e-Tax case management system changes have been submitted to the
Ministry of Finance but are still awaiting approval. Additionally, some tasks did not get OTA’s
focus given the amount of work in this area. These have been addressed in the final
Memorandum of Recommendations to Commissioner General of RRA for further follow up.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
18
OUTCOME 3: COMPONENT 3.1 CONCLUSIONS
Component 3.1: The support of OTA was well received and many of the recommended changes
have already been adopted or are underway. Additional recommendations for future work are
included in the OTA Memorandum of Recommendations to the Commissioner General. Several
areas for improvement were identified, including 1) delinquent tax payment deadlines, 2)
collections officer capacity building, 3) joint development of a comprehensive collections
approach, 4) ending automatic assessment of non-filers, and 5) guidance for best practices in
arrears balance management.
Overall, OTA’s support was well received in this area and RRA adopted many of the
recommended changes or is in the process of doing so. Due to the large number of issues to be
addressed, OTA recommends continued work, which is detailed in the Recommendations Memo.
Despite OTA’s advice to RRA, the evaluation team concludes that extending delinquent payment
periods is not considered best practice. OTA’s recommendation has been adopted, but the
evaluation team concluded that OTA may have misunderstood the root causes of taxpayers’
delinquencies. An efficient collections operation requires several enforcement stages because
delinquent accounts become less likely to collect over time. OTA recommended policy changes
that assumed delinquent taxpayers needed more time to pay; however, the evaluation team
concludes that extensions of delinquent payments to 36 months contributed to noncompliance.
Arrears become increasingly more difficult to collect as they aged, eventually becoming
uncollectable. Devoting efforts to educate taxpayers and enforce timely payment of tax
obligations in a consistent way is widely considered best practice.
The evaluation team also concludes that OTA’s support to develop RRA’s collection manual is
only part of a capacity building strategy that, to be fully effective, should have included
specialized training in improved investigative techniques that enable collections officers to work
through the best strategies and solutions for the many variations in circumstances and challenges
they encounter. RRA’s manual should describe how the business process works in any given
sector, describe the necessary documentation, and provide guidance for where to look for assets
and income. Rather than simply following prescribed solutions to anticipated situations, the
manual should enable critical thinking for auditors based on principles and critical questions for
each situation. The evaluation concludes, based on multiple counterpart interviews, that RRA’s
officers face limitations to investigate taxpayer's arrears properly, particularly where
circumstances vary beyond the limited cases found in the manual.
OTA also advised RRA to prioritize collections of easy case collections. However, the
evaluation team concludes that, based on evidence from counterparts, this resulted in more
complex cases to be almost entirely neglected, resulting in an increase in aged arrears. Instead of
selecting easy cases, OTA needed to work with RRA to develop a more comprehensive
collections approach that included enforcement interventions at each stage of the collections
process that would provide sequential enforcement and increased consequences as cases age.
Every case should see the same initial level of effort applied as the case moves up the
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
19
enforcement ladder. The easy resolution cases will be finalized early and the more complex will
continue up the ladder. OTA’s advice to focus on the easiest cases effectively allows
enforcement staff to pick the cases on which they wish to work, reducing incentives for resolving
all but the least revenue-recovering, more consequential, cases and generally erode a culture of
taxpayer compliance.
The evaluation team also concludes that OTA should have advised against RRA’s policy of
unrestrained, automatic assessments for non-filers. Automatic assessments should stop after one
or two late payments to confirm that the taxpayer’s business is active, as counterparts reported
many temporary and short-lived businesses remain on RRA’s registry, contributing to higher
arrears rates.
Finally, the evaluation team concludes that OTA should have provided guidance regarding
policies for write-offs or the setting aside of uncollectible debts that reflect best practices. Write-
offs provide a critical step in collections cycle, without which total arrears balances grow without
the appropriate enforcement action being taken and distort the Government of Rwanda’s revenue
accounting. OTA did not appear to assist RRA with arrears balance management best practices
through generally accepted, standard write-off policies.
COMPONENT 3: OUTCOME 3.1 RECOMMENDATIONS
OTA needs to use a more systematic approach to planning engagements.
It is critical that OTA advisors are current on country context and best practices in the field of tax
administration. It is an evolving field; as such, it is critical that OTA advisors recommend
changes that advance RRA’s practices. The conclusions section outlines a number of examples
where OTA guidance may not have been as efficient or effective due to a lack of understanding
of RRA’s larger business process and context. See Section IV for further information.
COMPONENT 3; OUTCOME 3.2
Table 4 summarizes the findings for Component 3.2, and a detailed discussion follows the
summary.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
20
Table 4: Component 3 Outcome 3.2—Activities and findings
Expected Outcome: 3.2
RRA adopts and implements management information reporting and debt management reporting to
track and analyze trends in arrears over time. RRA management adjusts policies/procedures as needed
to facilitate debt collection.
Activities
Findings
3.2.2: Assist RRA in
designing management
information reports
• RRA had some management information in place, produced
manually in Excel spreadsheets. OTA supported the revision and
improvement of these reports. For example, OTA assisted with
refinement of the Top 25 arrears case reports and the debt
management report.
• RRA revised, adopted, and implemented some of the tax debt
policies suggested by OTA. However, RRA was primarily focused
on tax debt processes and procedures. Industry sector notes
(construction sector only) were produced to increase efficiency
when working with this key sector.
• RRA was already tracking one KPI for the aging of debts when the
OTA program started.
3.2.2: Train an RRA analyst
to conduct trend analyses
and develop monthly debt
management analytical
reports for management
• Training was completed for a number of staff, but few managers
were included in the sessions. Most trainings were single sessions,
not components of a systematic training approach.
• Training materials were developed by OTA but were not
complemented by use of a train-the-trainer model.
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Partially achieved
The outcomes under outcome 3.2 were partially achieved. Full achievement of Component 3.2
would have included training managers and building in a more systematic implementation, such
as a train-the-trainer model, to ensure training would continue to be replicated.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Outcomes Sustained: Report revision and construction sector industry notes
The evaluation team has found that OTA’s assistance with the revision of the Top 25 reports has
been carried forward and continues to be used. Respondents noted that construction industry
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
21
sector notes have been continued but they have not been developed for the other sectors, such as
telecoms, banking, etc.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: Time restraints prevented training
RRA was not trained to analyze data because of time restraints at the end of the project.
COMPONENT 3; OUTCOME 3.3 FINDINGS
Table 5 summarizes the findings for Component 3.3, and a detailed discussion follows the
summary.
Table 5: Component 3; Outcome 3.1—Activities and findings
Expected Outcome 3.3
RRA analyzes data from timesheets to understand trends in productivity and assign cases to match
enforcement officer skills with caseloads.
Activities
Findings
3.3.1: Assist RRA in designing
timesheets for enforcement
officers (EOs)
• OTA supported the adoption and implementation of timesheets
for debt management as requested by RRA leadership.
However, OTA did not socialize this change or do any change
management, which led to poor adoption by staff.
3.3.2: Train Debt Management
managers and EOs on timesheets
• Training for RRA to analyze and use data from timesheets for
management decisions did not happen due to time constraints.
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Partially achieved
Outcome 3.3 was partially achieved. OTA introduced the concept of timesheets and the
importance of tracking staff time for accountability and task management. RRA put timesheet
protocols in place and is using them to monitor individual auditors’ performance.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Outcomes Sustained: Timesheet protocols and practices
RRA has maintained the timesheet protocols and practices, but adaptation continues to be limited
in scope and not systematic across the organization.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
22
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: Incomplete timesheet use
Timesheets are not used by all staff or fully completed; therefore, RRA is not able to use and
analyze timesheet data to understand trends in productivity across the organization. Ideally, this
information would be used to identify efficiencies and inform resource allocation, but data are
not sufficient for this level of analysis.
COMPONENT 3; OUTCOME 3.3 CONCLUSIONS
Timesheet use is not consistent across RRA and may not fully meet operational needs.
Timesheets were introduced by OTA, but never fully operationalized. While the use of
timesheets is a best practice, OTA advisors should begin the process by understanding RRA’s
operational process and ensuring the timesheets met their business needs. This would have made
the tool more useful and alleviated some of the resistance by staff to using the timesheet.
COMPONENT 3; OUTCOME 3.3 RECOMMENDATIONS
Train OTA advisors in change management principles and employ this approach when
introducing and operationalizing new tools or processes, such as timesheets.
OTA advisors should be trained in change management principles in the future to ensure they are
taking a more holistic approach when introducing new tools or processes. Any organizational-
level change should use a change management approach to ensure the new process meets the
needs of the organization and to gain buy-in by staff and managers. All levels of staff should
also be consulted to ensure the practice is applied and used throughout the organization.
Pr og ram Compon e nt 4: Sp eci alized S ect or Aud it
COMPONENT 4 OBJECTIVE
The objective of this component was to strengthen RRA’s capacity to audit construction
sector businesses and improve taxpayer compliance. The project components focused on
strengthening RRA’s ability to understand the sector’s business environment and regulations,
building them into their own audit policy and procedures. The expected outcome was to increase
the number of cases to be upheld in appeals as the result of RRA auditors using a completed and
more accurate audit manual.
COMPONENT 4 FINDINGS
Table 6 summarizes the findings for Component 4, and a detailed discussion follows the
summary.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
23
Table 6: Component 4—Activities and findings
Expected Outcome 4.1
RRA adopts and implements a construction sector section of the audit manual that complies with the
sector’s policies, procedures, laws, and regulations. RRA staff use audit manuals to conduct audits that
are complete and accurate, which are more likely to be upheld when appealed, leading over time to
increased voluntary tax compliance.
Activities
Findings
4.1.1: Assist auditors in
reviewing the laws and
regulations pertaining to the
construction sector and propose
revisions to the sector audit
policies and procedures
• OTA supported auditors to review the laws and regulations
pertaining to the construction sector and develop sector-specific
notes.
4.1.2: Assist in drafting a new
construction sector section of the
audit manual
• OTA helped adopt and implement a construction sector section
of the audit manual that complies with the policies, procedures,
laws, and regulations specific to the sector.
4.1.3: Train designated managers
and staff in the use of the
construction sector manual
• OTA provided some training materials and trained designated
managers and staff in the use of the construction sector section
of the audit manual. However, trained staff were moved to do
general audits right after receiving construction sector
training.
4.1.4: Mentor RRA auditors and
management on conducting
construction sector audits
• OTA provided on-the-job training and provided some live-case
reviews (one case was examined); however, OTA fell short on
securing commitment from RRA management to keep the
trained auditors in the same section.
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Achieved
OTA provided comprehensive support for all aspects of the component including devising and
updating the special sector audit manual, sector notes, and providing experiential training using a
live case. In general, the evaluation team found that the support provided in drafting the sector
notes was very useful and appreciated by the auditors at RRA.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
24
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Outcomes Sustained: Use of construction sector audit notes and updated manual
section
RRA still uses and regularly updates the sector notes and revised construction sector audit
manual.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: Transfer of trained sector auditors to nonspecialized audits
After OTA provided the construction sector audit training, RRA management transferred the
trained auditors back to general, nonspecialized audit work. As a result, according to several
counterparts, OTA’s training has not been used.
COMPONENT 4 CONCLUSIONS
The range of construction sector audit support and advice of OTA was well received, gains were
made in all aspects of construction sector audits. Efforts are needed to keep auditors with
specialized training assigned to work in their new area of specialization.
This is one of the components where OTA provided comprehensive advice and their support
covered all aspects of the construction sector audit activity ranging from devising and updating
the special sector audit manual, sector notes, demonstration through a live case, and training.
OTA’s support on this component remained available consistently through the program. The
counterparts were very pleased with the model of support provided by OTA. However, the
auditors trained in construction were pulled away for work in other sectors immediately after
they completed their construction sector training. Here, OTA could have used the leverage and
sought assurance from the senior management that the trained auditors will be allowed to
practice their newly acquired skills.
COMPONENT 4 RECOMMENDATIONS
For future programs, direct and frequent meetings of OTA advisors and audit (technical)
managers, along with the user of a wider range of cases to demonstrate techniques during
training, can improve direct knowledge transfer. Securing an RRA leadership commitment to
retain auditors in the department of their (sector) training will also help sustain knowledge
transfer and future training.
OTA advisors worked only on one live case during the training on the construction sector. It
would have been more useful if OTA had demonstrated audit techniques by using a variety of
cases during the training to give a wider perspective. Feedback from respondents was that the
OTA advisors should meet directly and more frequently with the audit (technical) managers
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
25
instead of the senior managers. This would provide for more direct knowledge and skills transfer
to audit managers that are more involved in the day-to-day operations.
RRA would have seen greater benefits from OTA efforts if there was a commitment on their part
to train and deploy special sector auditors in a more efficient manner. RRA has a culture of
shifting employee resources at will and this practice could have been addressed in the areas
where OTA provided assistance, such as the construction sector audits. This could have been
done by securing an agreement from RRA leadership to keep the auditors in the same department
for which they received training until they cement their expertise and then train more auditors for
more sustainable outcomes.
Pr og ram Compon e nt 5: Computer -Assis te d Au dit
COMPONENT 5 OBJECTIVE
The objective of this component was to enhance the capacity of RRA auditors to conduct
complex audits and streamline the audit process in high-risk compliance sectors, such as
telecommunications and banking, using a Computer-Assisted Audit (CAA) program.
Component activities focused on training Computer Audit Specialists (CAS), educating
taxpayers on the relevant filing requirements that enable RRA to use computer-based audits, and
teaching auditors how to use audit data.
Table 7 summarizes the findings for Component 5, and a detailed discussion follows the
summary.
Table 7: Component 5—Activities and findings
Expected Outcome 5.1
RRA operationalizes and appropriately staffs a CAA Manager and a CAS. RRA adopts and
implements the CAA audit manual to analyze large datasets of taxpayer substantiation and efficiently
and effectively identify issues and adjust or verify tax returns.
Activities
Findings
5.1.1: Advise RRA management
on recruiting and staffing CAA
and CAA manager
• OTA suggested the idea of CAA and advised RRA
management on recruiting, staffing, and software needs
necessary for CAA management.
5.1.2: Train CAS auditors on data
management, analysis,
presentation skills, and other
fundamental CAA skills
• OTA developed some training materials for CAA and trained
three auditors. However, the team of three auditors who
received the training are no longer on staff. RRA is currently
training new CASs.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
26
• Due to fast staff turnover, RRA wanted OTA to train all
auditors on CAA; however, OTA did not have the resources
to do so.
5.1.3: Identify sources for online
and classroom training in
advanced Excel and IDEA Data
Analysis Software for the CAS
auditors
• The evaluation team did not find evidence, such as training
materials or sources for online or classroom training, for this
activity.
5.1.4: Assist RRA in developing a
new CAA section of the audit
manual
• OTA assisted in adopting and implementing the CAA manual
to analyze large datasets of taxpayer substantiation.
5.1.5: Train CAA manager in
CAS group management
responsibilities and how to
conduct CAA case reviews
• OTA did not provide any training to the CAA manager in
CAS group management.
5.1.6: Assist the CAA manager in
reviewing the development of the
CAS auditors and, as needed,
provide on-the-job training for
CAA
• This output was only partially achieved as CAA had not
actually started during OTA’s program. RRA is currently
preparing their officers so they can launch a formal CAA
program.
5.2.1: Assist RRA in developing
guidance to taxpayers on
maintenance of electronic records
for use in taxpayer audit
• RRA has not yet issued any guidance or materials to
taxpayers to educate them on how to maintain electronic
records for use in CAAs.
• Furthermore, OTA suggested revising the tax law and
procedures (the legislation was passed just a few weeks prior
to the evaluation team’s visit). The new law requires
taxpayers to file returns in an approved format, which enables
RRA to use the information and data from the return for CAA
purposes.
COMPONENT 5 FINDINGS
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Partially achieved
The expected outcomes for Component 5 were only partially achieved. OTA was able to
introduce CAA as a concept and begin initial steps of implementation, including the training of
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
27
staff. However, CAA is a complicated and resource-heavy endeavor. Full implementation will
take time and both human and financial resources beyond the scope of the current OTA project.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Outcomes Sustained: Not achieved, prerequisite infrastructure needed
This component cannot be considered achieved or sustainable because of prerequisites
that must be in place for operationalizing CAA—including software that can process CAA data
and expertise to use the CAA tool. In addition, regulations that allow RRA to perform CAAs
were only recently passed in 2023 after the program ended. The passage of the regulations
demonstrates buy-in by the Government of Rwanda to this process. RRA expects to conduct
further training to implement CAA now that a legal framework is in place.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: Prerequisite infrastructure needed
As mentioned above, RRA does not yet have the relevant infrastructure to support the
outcomes—the software and systems and the expertise to carry out such audits. The needed
legislative changes were enacted after the program ended. While OTA successfully introduced
CAA, systematic barriers prevented undertaking or achieving most of the workplan activities.
COMPONENT 5 CONCLUSIONS
Technical advice on CAA was provided by OTA before RRA had the established infrastructure
necessary to fully adopt and operationalize a CAA system. Further technical assistance on
establishing CAA, from taxpayer education to computing infrastructure, will be needed.
CAA is generally a very useful concept. This concept was recognized by the OTA team when
they recommended setting up a dedicated CAA team in RRA. However, the evaluation team
concludes that RRA had neither the capacity nor the systems in place to utilize technical advice
on CAAs. OTA provided technical advice that would have been useful (and needed) only after
RRA had established infrastructure to fully adopt a CAA system.
There will be continued work by RRA to implement the CAA that may require ongoing technical
assistance. This initiative requires a significant investment in taxpayer education for this type of
program to be successful. Implementation of a CAA program also requires software that
communicates with other RRA systems to be able to interpret information provided by taxpayers.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
28
COMPONENT 5 RECOMMENDATIONS
A whole-package approach to CAA in Rwanda, which includes sequential planning and is
informed by a thorough understanding of the local context, is key to effectively introducing and
operationalizing CAA.
CAA needs to be seen as an all-encompassing program from RRA and needs to be planned
accordingly. The recommendations made by OTA should therefore have included the whole-
package approach with an outline and sequencing of the necessary steps to be taken.
Due to high staff attrition, RRA wanted OTA to train all auditors for CAA. However, OTA did
not agree to the proposal and provided CAA training only to a couple staff members who have
since left RRA. This component highlights another aspect where close attention to the local
context rather than prescribing/following standard practice would be beneficial.
Pr og ram Compon e nt 6: General Audit Capacit y
COMPONENT 6 OBJECTIVE
The objective of this component was to strengthen RRA’s audit quality by establishing
Quality Review Staff (QRS), documenting procedures and standards for audit quality, and
developing a staffing plan and organizational structure to assure and expand adherence to
standardized good audit practices. Better quality audits were anticipated to result in accurate
assessments, allowing for better targeted collections and reducing the burden on the tax authority
in resolving disputes via the formal dispute resolution process.
Table 8 summarizes the findings for Component 6, and a detailed discussion follows the
summary.
Table 8: Component 6—Activities and findings
Expected Outcome 6.1
RRA establishes a QRS team with qualified employees, selected in accordance with stipulated position
descriptions and selection criteria. QRS personnel understand and execute their duties using defined
roles and responsibilities. RRA establishes a QRS team that operates a fully developed QRS that
improves audit case quality and reduces taxpayer appeals.
Activities
Findings
6.1.1A: Assess baseline audit
quality
• The evaluations assessed that this activity had not happened.
6.1.1B: Review current audit
organizational structure and
• OTA supported the establishment of a QRS team with qualified
employees, selected in accordance with stipulated position
descriptions and selection criteria.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
29
recommend establishment of a
QRS team
6.1.2: Assist RRA in creating
position descriptions and
selection criteria for QRS
auditors
• OTA helped QRS personnel understand and execute their duties
using defined roles and responsibilities and set up a fully
developed QRS that improves audit case quality and reduces
taxpayer appeals.
6.1.3: Assist RRA in developing
plans and procedures for quality
control in the audit process
• OTA supported the development of a quality review plan, a set
of procedures and audit techniques for audit quality control.
• OTA supported digitization of the audit manual, which
enhanced usability of the manuals. OTA also helped RRA set
up a process to update the audit manual.
Expected Outcome 6.2
RRA adopts and implements timekeeping, standardized audit reports, tax computation worksheets, and
audit planning. Auditors apply good practices for audit to their cases, identify weaknesses in the audit
process and within individual audits, provide constructive feedback through an established procedure,
and initiate a corrective follow-on process for audit quality improvement. This leads to progressively
improved audit outcomes.
Activities
Findings
6.2.1: Assist RRA in developing
audit tools including
timekeeping, standardized audit
reports including a tax
computation worksheet, and
audit planning
• OTA supported adoption and implementation of timekeeping,
standardized audit reports, tax computation worksheet, and
audit planning. More specifically, OTA:
o Provided a template that RRA has modified and used to
produce a risk-based annual audit plan that includes case
selection and case management.
o Supported the development of procedures and standards for
audit quality.
o Provided a timesheet template to help managers gauge
auditors’ performance.
o Developed a staffing plan and organizational structure to
assure and expand adherence to standardized good audit
practices.
• Most of the advice provided by OTA was reported to be useful.
The audit manual jointly developed with support from OTA is
maintained and being used by RRA.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
30
• Although the review team did not inspect the audit manual,
advisors referenced its size and complexity.
• The timesheet was specifically noted as having limited utility,
and it is only being used for distinguishing direct and indirect
time spent on a case by the auditor.
6.2.2: Train employees and
managers in the use of the audit
tools
• OTA developed training materials, and a limited amount of
training was given to audit staff.
COMPONENT 6 FINDINGS
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Partially achieved
Most of the outcomes identified were achieved to some degree. The quality review process was
enhanced, audit reports and computation worksheets were developed, an audit manual was
improved, and audit planning templates were developed.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Outcomes Sustained: Audit manual use and updates, quality review process
improvements
The audit manual has been fully adopted by RRA and continues to be updated regularly. The
quality review process that was enhanced and the risk-based audit plan template developed by
OTA continues to be used and was reported to be effective in staff interviews.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: Change management not used to introduce timesheets
The introduction of timesheets has been a contentious issue and limited in use. These control
measures were introduced without prior groundwork to socialize the change and explain the
utility. This change would have allowed RRA to link individual audit case times to the e-Tax
system so audit results as well as time and level of effort factors could be uploaded to risk
management and to performance management units. This information would have been useful
when planning the workloads for coming periods. Unfortunately, this process was not explained
to staff and most felt like it was an intrusive level of oversight.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
31
COMPONENT 6 CONCLUSIONS
The OTA support in producing a high-quality, audit manual and introducing enhancements for
RRA’s quality review process for audits were key successes. These tools and OTA’s advice could
have been improved by a better understanding of the RRA context—producing higher-level, less
prescriptive guides and providing advice that reflected RRA’s organizational situation.
Overall, OTA provided support that was relevant and well received under this component.
The evaluation team found that OTA advisors were able to assist in getting several tools in place
within RRA including standardized audit reports, tax computation worksheets, and audit
planning. Unfortunately, the feedback from those interviewed was that the operations manual,
while useful, was very prescriptive and cumbersome to use effectively. An effective audit
manual should include tools to guide a decision-making process and be a complement to
classroom training rather than include detail on each case.
OTA supported changes to the delegation of authority, which led to fewer levels of approvals
required to process an audit case and resulted in faster processing procedures and time spent on
each case.
The most significant achievements include setting up QRS and a quality review process. OTA
helped establish a constructive feedback procedure to assist auditors to improve their audit skills.
OTA dug into the issue of sending untrained auditors into the field and made recommendations
accordingly.
OTA also recommended reducing the number of field auditors for each case from two to one.
This is an example of not fully understanding RRA’s business needs and the policy was soon
reversed due to potential for corruption with a single auditor. It is also good practice to pair a
new auditor with a well-trained person to provide a better opportunity for on-the-job mentoring
and closing knowledge gaps.
Some other initiatives by OTA, including the suggestion to move desk audit under the
enforcement (previously being done by the general audit department), seemed to have missed
RRA’s needs. Desk audit activities are linked directly to the main audit function and should be
used as a training ground for new audit staff. The strategy to mix the unit into enforcement
would need to be reviewed.
As in some other components, OTA did not establish a baseline for audit quality and there were
no quantitative means to assess if audit quality had improved.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
32
COMPONENT 6 RECOMMENDATIONS
It is important for OTA advisors to invest the time to understand the organization, business
practices and rationale, and the local context to better facilitate the introduction of appropriate
new procedures. To gain insight into the organization’s day-to-day processes and needs and to
improve uptake of suggested changes, advisors should both work with staff at all levels and use a
project-wide change management approach.
As discussed in the conclusion section, while OTA has made some progress, there were also
many oversights. When designing future programming, the evaluation team recommends that
OTA advisors take the time to understand the organization and their business process, including
the rationale for that current structure. This would avoid missteps in introducing new procedures
counter to the organization’s requirements. Any proposed modifications should also be
accompanied by a more holistic change management approach—from senior leaders down to
more junior-level staff. As in the case of timesheets, even if there is buy-in at the leadership
level, the change will not work if staff do not understand the intent and are brought into the
process at an earlier stage. Finally, working with staff at all levels of the organization can help to
understand the day-to-day needs and the practical application of any new tools.
Pr og ram Compon e nt 7: Tax Dis p ute Res olution
COMPONENT 7 OBJECTIVE
The objective of this component was to improve voluntary compliance among taxpayers by
increasing fairness and transparency in dispute resolution. The planned project activities
focused on 1) reforming the tax dispute resolution process; and 2) creating a feedback loop
between dispute resolution personnel and audit staff.
Table 9 summarizes the findings for Component 7, and a detailed discussion follows the
summary.
Table 9: Component 7—Activities and findings
Expected Outcome 7.1
RRA understands the gaps between their current tax dispute resolution process and good practices and
takes steps towards improvement. RRA creates an action plan to improve audit quality, and, over time,
reduce the number of audit disputes.
Activities
Findings
7.1.1: Review and recommend
improvements to the current tax
dispute resolution process,
• OTA recommended improvements to the tax dispute process,
including changing the interest rate and penalty rates. OTA
provided a template for reference, which RRA ultimately
adopted.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
33
including the objections and
appeals process
• OTA also advised on the legislative changes required to enable
the new process, which RRA ultimately secured approval
through Parliament. Regulations still need to be put into place
to operationalize the change.
• Once RRA gained approval, OTA assisted them to create an
action plan to improve audit quality with the aim of reducing
the number of audit disputes. The evaluation team was not able
to quantify fiscal gain or reduction in the number of appeals as
no baselines were previously established.
• OTA also suggested setting up an appeals tribunal (an
international best practice); however, this was not adopted and
RRA continues to have the administrative review process.
7.1.2: Deliver a workshop on
international practices in tax
disputes
• Provided a 1-day training on audit procedures and tax law to
the audit staff.
Expected Outcome 7.2
RRA forms and regularly holds working group meetings to provide feedback to auditors regarding the
outcome of the dispute resolution process. Auditors use feedback from dispute resolution personnel to
improve audit quality, and, over time, reduce the number of audit disputes.
Activities
Findings
7.2.1: Develop a working group
composed of dispute resolution
staff and auditors
• OTA conducted a workshop on understanding the gaps between
their current tax dispute resolution process and good practices.
• A working group was set up to discuss the outcomes of appeals
to enhance auditors’ skills.
Expected Outcome 7.3:
Tax dispute resolution staff attend auditor training and understand the audit division's policies and
procedures and use the information to achieve fairer and more accurate results in dispute resolution.
Activities
Findings
7.3.1: Recommend inclusion of
dispute resolution staff in auditor
training
• Dispute resolution staff did not receive any audit training from
the OTA although RRA had some internal training for dispute
resolution staff on audit procedures and would discuss appeal
outcomes with the auditors.
• The dispute resolution staff provided feedback to the auditors
through the working group.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
34
Expected Outcome 7.4:
RRA creates and publishes taxpayer audit dispute resolution information on their website. Taxpayers
have access to accurate information regarding their rights in disputing the results of an audit with the tax
administration.
Activities
Findings
7.4.1: Assist RRA to develop tax
dispute resources for inclusion on
the RRA website
• OTA did not provide any support for taxpayer education in
reference to the appeals process.
• The RRA website does not have any new resources on dispute
resolution other than a small section on legislative changes that
is difficult to navigate for an average taxpayer. Taxpayers
continue to receive information about next steps through an
assessment notice in the mail. Mailing information to
taxpayers was also the procedure prior to OTA’s program.
COMPONENT 7 FINDINGS
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Partially achieved
The outcomes under this component were partially achieved. OTA advised RRA on
improvements to the tax dispute process, including changing the interest rate and penalty rates,
which were adopted. Additionally, OTA assisted RRA to propose and ultimately pass the
needed legislative changes. Policy changes are needed to fully operationalize this
recommendation, but the legislation paves the way for this to happen. OTA established a
feedback loop between the audit staff and dispute resolution personnel through a cross-agency
working group; however, they were unable to reform the dispute resolution process.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Outcomes Sustained: Feedback mechanism, policy changes
The legislation changes solidified modifications for the tax dispute process, which makes it
sustainable. RRA will still need to translate this legislative change into a policy that is more
operational, but that is in process. The internal process changes, such as the working group, are
still in place. There is continued use of feedback loops between the audit staff and dispute
resolution staff.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
35
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Hindering Factors: Readiness for dispute resolution process, limited scope of training,
The evaluation team observed that RRA was not at the point to begin instituting a dispute
resolution process. This is a complex process that requires changes at multiple levels and more
intensive capacity building. Multiple KII participants made observations on the mode and
quantity of training provided by OTA—under this component, a one-day (or shorter) training on
audit, tax law, and procedures was provided by the advisor—and many at RRA felt that this
training was insufficient or was not provided to enough staff to affect change. The general
perception was that additional staff training on all aspects is needed.
COMPONENT 7 CONCLUSIONS
A focus on developing the tools for taxpayer education on dispute resolution would have been
more beneficial for RRA.
While the internal process changes were mostly achieved, a major component was not addressed.
Taxpayer education and facilitation is critical to increasing tax compliance. Tax codes and
procedures are complicated, and any effort spent to simplify tax procedures must be clearly
communicated to the public to increase compliance. Auditors should be trained to discuss any
proposed assessment with taxpayers, give them an opportunity to submit new evidence, and then
work with the taxpayer to evaluate the new evidence. An investment by tax administration in
developing and establishing a dispute resolution process, which includes an initial screening (to
identify a frivolous dispute) and offers opportunities to resolve cases before going to a court, can
reduce the number of tax disputes. RRA could have benefited from more support from OTA in
developing such materials to make the operation of the legal framework fairer for taxpayers.
Information provided in a format that is easily understandable by the taxpayers on their
obligations like the assessment amount, timeframe to pay, and mode of payments will have a
direct impact on disputes. Tax disputes often arise due to several factors—including limited
taxpayer’s understanding of the tax legislation or a lack of communication between the auditor
and taxpayer, a difference of opinion on taxability issues, a strategy by the taxpayer to delay
payment of assessment by disputing it, or avoidance of payment due to a simple disagreement
with assessment. No such dissemination of information or facilitation to taxpayers was
supported by OTA.
COMPONENT 7 RECOMMENDATIONS
Assist organizations to identify the root causes of problems/issues to determine the appropriate
solutions to address the issues.
OTA should assist RRA, or other organizations in the future, to understand the root causes of the
problem to ensure the solution is appropriate. In this case, educating the taxpayer would have
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
36
had a more significant impact in dropping the number of appeals than changing internal
processes. If taxpayers understand the process and the reason for the assessment, they are more
likely to comply, resulting in a reduced number of appeals. Both OTA and RRA should have
seen this as an opportunity to change taxpayer behavior by mounting an education and awareness
program first. Once the root causes of the issue were addressed, then the training of staff would
have been more effective.
Pr og ram Compon e nt 8: Interna l Audit
COMPONENT 8 OBJECTIVE
The objective of this component was to improve RRA’s performance through enhancing
the effectiveness of its internal audit (IA) function. The planned project activities focused on
1) revising and implementing an internal audit manual, 2) training staff in internal audit
methodology, and 3) improving RRA departments' responses to internal audit recommendations
to improve internal and managerial controls.
An internal audit unit is a quality control mechanism for all functional aspects of the revenue
administration, starting from the initial registration of the taxpayers to their tax assessment and
exercise of enforcement functions and leading to the final phase of debt write-off of the
unrecoverable debts and taxpayer accounts. Information learned about effectiveness and
correctness of every step of the process should be fed back into the organization to create an
environment that seeks to improve processes and efficiency.
Table 10 summarizes the findings for Component 8, and a detailed discussion follows the
summary.
COMPONENT 8 FINDINGS
Table 10: Component 8—Activities and findings
Expected Outcome 8.1:
RRA leadership approves and implements standardized internal audit forms and updated IA
procedures. The revised procedures and standardized internal audit reports reflect findings and actions
that improve consistent application of laws, regulations, and procedures. RRA management monitors
the progress of departments in utilizing IA findings to mitigate risks, optimize operations, and reduce
revenue leakages in a timely fashion.
Activities
Findings
8.1.1: Review existing IA
procedures, reports, and
• OTA reviewed RRA’s IA procedures, reports, and inventory of
existing cases and identified gaps in IA procedures.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
37
inventory, and identify gaps in
audit procedures
8.1.2: Review RRA departments’
processes for acknowledging and
implementing recommendations
resulting from IA
• OTA reviewed departmental processes for acknowledging and
implementing recommendations resulting from IA.
• RRA was already using standardized forms; OTA assisted in
updating and improving the process.
8.1.3: Revise IA procedures
• OTA reviewed procedures and made recommendations on
revising or updating some of the IA procedures and
standardized internal audit reports. This improved the
consistent application of laws, regulations, and procedures.
• RRA continues to use the revised forms and procedures.
8.1.4: Develop processes and
implement a plan to address
RRA’s inventory of aged IA
recommendations
• RRA IA department had a backlog of departmental functions
recommendations, which were not yet implemented. OTA
assisted RRA to develop processes and implement a plan to
address RRA’s inventory of aged IA recommendations. OTA
suggested that RRA develop a tracking system for the
inventory of aged IA cases. RRA has not yet implemented
that recommendation.
Expected Outcome 8.2:
IA liaisons are assigned from each department, bridging the knowledge gap between IA and the various
departmental functions. Liaisons communicate as needed to collaborate to identify and resolve issues
across functional areas.
Activities
Findings
8.2.1: Brief RRA leadership on
the role and value of establishing
Audit Liaisons and create an
Audit Liaison selection process
• OTA briefed RRA leadership on the role and value of
establishing IA liaisons. RRA initiated a more formal
approach to embedding an IA liaison in each functional
department.
• IA liaison (a role previously filled by informal focal persons)
continued to be assigned from each department, bridging the
knowledge gap between IA and the various departmental
functions. They communicated as needed and collaborated to
identify and resolve issues across functional areas.
8.2.2: Provide training to audit
liaisons on their role
• There was no formal training, training materials, or guidance
on communicating with other departments provided by the
OTA, because the IA liaisons were experienced in the role.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
38
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Partially achieved
The outcomes under this component were partially achieved. OTA’s recommendations on
revising or updating some internal audit procedures and standardized internal audit reports were
put into use by RRA. Additional recommendations for revisions of internal audit procedures to
assist with the backlog of cases are not yet fully implemented. An internal audit manual was
developed with the assistance of OTA. While RRA already had informal focal persons from
various departments for internal audit purposes, OTA helped formalize the IA liaison positions.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Component 8 Outcomes Sustained: Updated audit manual in use, IA liaison roles
formalized
Gains in Component 8 have largely been sustained. RRA continued to use the audit manual with
updated procedures and put a process in place to update the manual to maintain its relevance.
Although some divisions have had turnover of individuals, the IA liaisons have remained a
position within RRA. Given this turnover, there will be a need for RRA to continue training to
maintain quality and consistency across divisions.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Component 8 Outcomes Hindering Factors: Training not needed for IA liaisons
Prior to the program, RRA already had staff acting as informal focal persons for the IA. While
OTA assisted with formalizing the role of IA liaisons, training was not needed for this
experienced cohort, which already held meetings to identify issues, collaborate, and address
problems across their functional areas. Similarly, OTA did recommend working group meetings
but did not develop an internal communications plan or supported RRA to set up an internal
communication process.
COMPONENT 8 CONCLUSIONS
Sustained progress was made in updating forms, tools, and procedures for the IA and in
formalizing the IA liaison role. To support these gains, a formal training program is needed,
particularly considering the high staff turnover rate in RRA. Establishing a tracking system for
the audit recommendations backlog system and uptake of new recommendations is similarly
important for future progress.
OTA made significant progress with the proposed activities in Component 8. They were able to
update many of the forms and tools needed to perform IAs and to standardize the process for
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
39
their implementation. Additionally, they formalized the IA liaisons positions within the
divisions as the main points of contact. These were all positive changes and largely sustained by
RRA. While progress was made, there was no formal training program initiated as part of the IA
effort given the experience of current staff. This training will need to be established to sustain
change over time, given the high turnover within RRA. RRA also was not able to implement a
system to track action on the backlog of audit recommendations. Establishing a backlog tracking
system will also be important in terms of ensuring the IA function serves its purpose. The
administration of RRA needs to better embrace the value of the internal audit function and see
the efforts as a measurement tool for the performance of the revenue function. More effort could
have been proposed by OTA in the areas of backlog reduction, all-staff training, and tracking IA
recommendations after they have been delivered to the operations team.
COMPONENT 8 RECOMMENDATIONS
For future programs, invest in understanding and scoping the organizational needs in order to
develop a sequential plan for based on organizational capacity for implementing and sustaining
changes in the IA process.
The workplan may have been overly ambitious given the limited time and resources. As stated
in other areas, it is important for OTA to look at organizational needs more holistically and
ensure RRA, or other organizations in the future, are capacitated not only to implement the
change but also to maintain it. OTA may need to be more targeted in future projects to ensure
activities/functions can be implemented, rather than trying to address a broad number of changes
within an organization at one time.
Pr og ram Compon e nt 9: Ris k M a nagement (Informati o n
Te c hnology Risk Man agem ent )
COMPONENT 9 OBJECTIVE
The objective of this component was to assist RRA to implement a risk management
process to support the operations and strategic objectives of RRA and to enhance the
physical security of RRA headquarters. The planned project activities included reviewing the
risk management process, training management on risk management, developing a methodology
for risk registers, developing a physical security plan, and mitigating physical and IT risks.
Table 11 summarizes the findings for Component 9, and a detailed discussion follows the
summary.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
40
COMPONENT 9 FINDINGS
Table 11: Component 9—Activities and findings
Expected Outcome 9.1
RRA develops and implements a corporate risk register for identifying and mitigating compliance risks.
RRA effectively identifies corporate risks and prioritizes resources according to their magnitude.
Activities
Findings
9.1.1: Review the risk
management process and
framework at RRA for
identification of gaps in corporate
risk registry
• OTA supported the risk management department to identify a
significant number of potential corporate risks and to prioritize
resources to mitigate according to their magnitude.
• RRA continues to update the corporate risks in the register to
capture emergent risks and develop mitigation plans.
• While RRA has made significant progress in systematizing its
risk management process and mitigation, some areas remain to
be addressed, such as the safety of taxpayer information.
9.1.2: Provide training to middle
and senior management on risk
management practices
• OTA provided training on risk management to both middle
and senior management and communicated with the risk
management staff on a regular basis to reinforce the training.
9.1.3: Assist RRA in developing a
methodology for identifying and
rating corporate risks in risk
registers
• OTA reviewed the risk registers and worked with RRA on a
process to prioritize resource allocation for mitigation.
Expected Outcome 9.2
RRA implements safety and security improvements, reducing risk to employee safety and RRA
facilities and property, and protecting taxpayer information.
Activities
Findings
9.2.1: Develop a physical security
improvement plan based on risks
of physical security at RRA and
their potential impact on
personnel, equipment, and
operations
• For the physical security improvement plan, OTA developed
processes based on identified risks that enhanced RRA’s
physical security through:
o Improving physical security by implementing a color-
coded card system for staff to limit access to specific
floors.
o Revising staff code of conduct, which includes anti-
harassment policies.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
41
o Implementing a background check requirement for new
staff recruitment.
o Sensitizing staff on security protocols.
o Requiring visitors to always wear a visitor’s pass while
inside RRA.
o Limiting taxpayers’ entry/access to the first floor of the
RRA building unless accompanied by an RRA employee.
o Placing security guards in key access areas.
o Developing fire drills and evacuation procedures.
9.2.2: Provide guidance on
mitigating risks outlined in
security improvement plan
• RRA developed a risk mitigation plan and a business
continuity plan.
Expected Outcome 9.3
RRA adopts an IT plan allowing them to mitigate urgent threats to its information security and puts in
place a longer-term plan to allocate resources to meet its IT needs.
Activities
Findings
9.3.1: Assess and identify priority
IT security needs, including IT
risks and mitigations
• RRA reinstituted its IT strategy and a steering committee that
currently meets regularly to monitor the performance of IT
projects and to make informed decisions on future IT
investments in alignment with RRA’s strategic plans.
• OTA supported RRA to develop and adopt an IT plan allowing
RRA to mitigate urgent threats to its information security
department. The IT plan included:
o Revised IT policy, which included care and control of
official IT equipment.
o User policies that included password protections and
network protection to control security breaches, data
loss, fraud, and errors in RRA operations.
o USB ports disabled on the official computers and
laptops.
• RRA implemented a longer-term plan to allocate resources to
meet its current IT needs.
• RRA adopted an IT training plan ensuring that IT staff receive
training commensurate with their responsibilities, as well as in
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
42
critical areas (such as cybersecurity). The additional training
allowed RRA IT staff to carry out their duties more effectively
and respond to ongoing and changing threats.
• While significant progress has been made in securing IT
equipment and infrastructure, gaps remain in information
security that will need to be prioritized.
EQ 1: Which of the expected project outcomes were achieved?
Outcomes Status: Key outcomes achieved
The evaluation team found that OTA and RRA successfully achieved key outcomes in
Component 9. RRA was able to develop robust risk mitigation procedures, including
identifying, recording, and mitigating risk. They were able to prioritize resource allocation using
this information and minimize the risks identified. RRA was able to take concrete actions with
the support of OTA to specifically identify risks related to physical and IT security. As noted in
the table above, changes were made to ensure the physical safety of the staff, including badging
protocols, limiting access of visitors within the building, and posting security guards at key
access points. Similarly, RRA was able to reinstitute an organization-wide IT strategy that had
not been fully implemented. This allowed RRA to monitor the IT security risk more proactively
through a comprehensive IT plan and a steering committee, which met regularly to assess and
prioritize risks and resources.
EQ 2: Which of the achieved outcomes have been sustained up to the present?
Component 9 Outcomes Sustained: Risk management, strengthening security
RRA has largely been able to maintain outcomes achieved in Component 9 related to risk
management, including the changes instituted to strengthen physical and IT security. The
evaluation team noted that the changes RRA made were done in a holistic manner, which
included both the changes to policy and procedures and the concomitant training and
sensitization of staff on these changes. This also involved a process to prioritize and reallocate
resources as necessary.
EQ 4: For any expected outcomes that were not achieved, which factors hindered
success?
Component 9 Outcomes Hindering Factors
OTA advisors recommended the establishment of an emergency command center to better
coordinate and respond to risks efficiently. This would be additional to the measures already put
into place mentioned above in Table 11, under Outcome 9.3. The resources for the emergency
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
43
command center were made available at the time of the evaluation, but the unit has not yet been
established.
While RRA made significant progress on physical and IT security, cyber or information security,
including protecting confidential taxpayer’s information, remains weak. The RRA prioritized
risk areas where they could make progress quickly—“the low hanging fruit.” Information
security is more complex and will take time, capacity, and funding to address.
COMPONENT 9 CONCLUSIONS
The main contributions of OTA in risk management centered on the physical security of RRA
equipment and building. Due to time and resource constraints, OTA did not take on
development of a complete risk management program outside of addressing physical security. A
robust risk management program that considers incorporating risk analysis, hazard mitigation,
and disaster recovery planning across the agency, is a significant need for RRA.
The purpose of a gap analysis is to flag potential risks and constraints to every administrative
process within a tax authority. The risk management group is a key component of a compliance
improvement plan that identifies factors that impede or otherwise can impair the operations and
effectiveness of the revenue administration.
Risk management often requires the intake of significant amounts of data to identify potential
strengths and weaknesses and the outcome of this exercise involves entering relevant, clean, and
accurate data into the analysis for accurate forecasting of risks. OTA primarily focused on the
physical security aspects of the risk program for RRA, which included the physical security of
the building, requiring employees to wear badges, and defining protocols around when and
where taxpayers can access the tax authority. OTA did not have the time or sufficient resources
to support a complete risk management program for the RRA. It appears that OTA did not work
on, or help with, strategizing and planning for other risk areas relating to taxpayer information or
other tax functions, for example, tax compliance. Confidentiality of taxpayer documents was
improved to an extent—more in terms of equipment theft rather than regarding confidential
taxpayer information. Hence, OTA’s core support around risk management remained central to
only the physical security of the equipment and building.
COMPONENT 9 RECOMMENDATIONS
An organization-wide risk analysis, particularly for IT, and the development of a robust
management program across all departments is a pressing need for RRA.
Risk management has links to every other function in RRA and while the physical security issues
were important, they overshadowed the importance of an effective disaster recovery plan for
RRA’s records and data. The Risk Management project by OTA was very effective in mitigating
risk in physical space and developing risk registers and employee safety measures. The review
team was left with the impression that RRA accounting software was still very much at risk from
hacking threats and from general disasters, yet the protection and recovery plans were vague.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
44
The risk management team needs to complete a full analysis of risk in the IT function and
develop a mitigation plan as soon as possible.
Ev al uati on Q ue stion 3
The focus of EQ 3, regarding a new approach to program design that involved counterparts in
operational manual development, was a high interest activity for OTA at the time this evaluation
was designed. Although prior sections discuss aspects of OTA’s support for developing
operational manuals, it is worth highlighting the positive effects of this effort.
EQ 3. OTA took a new approach to program design in Rwanda by involving
counterparts in the design and writing of key operational manuals. Did OTA's
new approach contribute to the sustainability of the outcomes, and if so, how?
Previously, OTA used a predetermined set of manuals to apply to each project. The manuals
were adapted by OTA advisors for each new project, with only minimal involvement of their
counterparts. OTA’s new approach, which included close collaboration with their Rwandan
counterparts for operational manuals development, had several positive effects. The evaluation
team found that by employing this approach, OTA obtained buy-in from their counterparts,
promoted ownership of the product, and created a positive environment for updates and
revisions.
OTA worked with RRA counterparts to develop the RRA audit manual. Component 3 highlights
some quality issues with the audit manual, nevertheless, involving RRA counterparts in both the
design and writing of the manuals contributed not only to their relevance but also to the
counterparts’ acceptance of end product. This approach also laid the foundation for RRA to
continue to design and write additional manuals.
IV. PROGRAM LEVEL RECOMMENDATIONS
While Section III includes specific recommendations for each of OTA’s nine program
components at the activity level, this section provides a discussion of recommendations focused
at a higher, overall program level. The recommendations focus on two overarching areas:
program design and ensuring appropriate staffing and training of OTA advisors.
Pr og ram Desi gn Co nsi de rati on s
Based on the evaluation findings and conclusions, the evaluation team recommends that OTA
consider the following three suggested changes in project approach and design.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
45
THEORY OF CHANGE
The evaluation found that both the project logframe document, work plans, regular reports, and
the End of Program Review (May 11, 2022) did not consistently follow a common framework in
terms of outcomes, activities, and outputs—and sometimes not even in the overall project
components and objectives. Consequently, the alignment of tasks to strategic plans and the
creation of feedback loops were weak. While work plans were developed with some ownership
from the technical senior managers at RRA, the evaluation team concluded that this ownership
should have been built-in at a much earlier stage of the project and should also have included a
wider range of RRA staff cadre.
A theory of change (or similar) should be created at an early stage of the project. The
process to create the theory of change would have helped the participants better identify the
development needs and to obtain a common understanding of a feasible way forward. The
theory of change development should follow a common process, using standard formats and
terminology that will be further used throughout the project planning and monitoring process.
BENCHMARKING AT PROGRAM INCEPTION
In addition, if the theory of change had been created, the basis for establishing a monitoring
framework for measuring and maintaining accurate, factual records of outputs and outcomes
over the course of the project implementation would have been in place. Baseline data and
progress indicators would have been part of that framework.
Collecting benchmarking data from the counterparts should take place at the inception/
start of a new component. This will a) build capacity at the counterparts to collect, synthesize,
and report data and b) help OTA quantify its impact. We also suggest articulating some of the
outputs and outcomes with more measurable/quantifiable indicators. Using outputs such as
“review the policy” or “provide training” lack the specific information needed to assess
usefulness or specific activities undertaken.
The completion of tasks at the output and outcome level needs to be evidenced by facts and data
in RRA reporting. The evaluation team therefore recommends that OTA should advise and
build capacity of RRA to ensure proper data collection, including establishing baselines for
any production-related component. Collection and comparison of data are vital for
determining progress.
SYSTEMS APPROACH
The evaluation team recommends that OTA advisors invest more time to understand the
counterpart’s organization, business processes, and priorities prior to developing a
workplan. By developing a robust understanding of the organization’s context, OTA advisors
can understand the organization’s readiness for the activity, ability to implement recommended
changes, and potential to sustain changes after the OTA engagement—factors critical to the
short- and long-term success of the program.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
46
As noted in the components section, a number of outcomes were not achieved or not sustained
because OTA advisors did not understand the interdependencies. For example, OTA introduced
a series of internal changes to increase taxpayer compliance but did not address the most critical
aspect of taxpayer education. If taxpayers are not aware of or do not understand the changes
meant to streamline the process, ultimately there will not be an increase in taxpayer compliance.
In another case, during the development of the KPIs, the department managers and staff received
some training, but senior managers were not included in KPI training, and they do not use KPIs
to inform higher-level decision-making.
The evaluation team recommends that OTA advisors develop an enhanced task plan for
each component they are responsible for implementing, including who needs to be trained,
what reports need to be generated, and what actions need to be taken to maintain the
sustainability of the operation. KPI work was further delayed by the lack of a management
information system to collect data. Using a systems approach and working from a more robust
understanding of the organization could have allowed OTA to focus on eliminating these barriers
to progress and sequence activities to maximize impact.
To that point, the evaluation team observed that OTA helped RRA make progress in collecting
outstanding arrears, but focused on process, including reducing the time to begin collection and
appropriately staffing to resolve each case in a timely manner. However, the evaluation team
believes that the project should have suggested that RRA develop a more comprehensive method
to resolve the wider issues. For example, RRA arrears balance included a large number of
nonactive registrants who discontinued business years ago.
OTA should work with RRA, through its Single Project Implementation Unit that enables
follow-up activities to ensure the continuity of technical support and consistent interaction,
especially when providing intermittent or virtual support.
St affi ng and Onbo arding Consi derations
SELECTION OF ADVISORS
Many OTA advisors came from a background in the Internal Revenue Service (IRS) at the
executive level and have since retired from the IRS. Candidates’ seniority and years of
experience make them attractive to OTA, but they often lack practical day-to-day operations
expertise to translate recommendations to the frontline workers who implement the reforms. In
addition, the field continues to evolve and the practices used during their tenure at IRS may no
longer be appropriate.
For example, much of OTA’s work was focused on the senior management level of RRA, but
these capacity building efforts did not sufficiently trickle down to the required levels. For
example, OTA suggested that RRA introduce a timesheet to capture staff time and effort spent
on audits. An outdated template of a timesheet from the Internal Revenue Service was
introduced with very little adaptations or consultations. The process to introduce any complex
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
47
system change requires a significant organizational communication effort and collaboration with
other relevant departments (such as Payroll, Human Resources, and Risk Management). The
timesheet program was put together for RRA without these necessary considerations and
consequently, the timesheet had limited use and was not trusted by employees who saw it as a
control measure.
For future OTA projects, the evaluation team therefore recommends OTA consider the
composition of the teams of advisors to encompass a wider range of experiences and
backgrounds.
Several counterparts and advisors reported that the program did not establish a primary, local
technical contact for the major activities under the various components at RRA and
communications between the advisors and counterparts were not always clear, which led to some
gaps in activity. While OTA engaged a local project coordinator to provide communications,
logistics, and administrative support for the program, the evaluation team recommends that
OTA provide, in addition to or instead of the local project coordinator, a local person
embedded among the counterparts with a higher-level technical role capable of working
across multiple program component areas.
ORIENTATION PACKAGE
The evaluation team recommends that all OTA advisors receive a thorough orientation package.
The orientation package should include information from RRA staff on organizational structure,
identified issues, business processes, and other information to get the OTA advisors prepared for
the work. This insight provided by employees would have been a valuable timesaving tool.
Orientation on Rwandan cultural and their political and legislative processes should have been
provided or accommodated within the team using a local advisor. Several respondents suggested
that technical assistance would be more effective if the local culture were understood.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
48
V. ANNEXES
Annex A . List of Docum e nts Reviewe d
Terms of reference—Rwanda Revenue Authority and U.S. Department of the Treasury—Office
of Technical Assistance (OTA)
Project Logframe, OTA, 2019
Workplans for 2015–2016, 2017, 2018, 2019–2020, OTA
Monthly reports, OTA (59 documents)
Statements of work, OTA (66 documents)
Trip reports, OTA (82 documents)
Project narratives, OTA (6 documents)
Project performance reports, OTA (5 documents)
Closing memo, OTA
Evaluation coversheet, OTA
Establish an RRA Future State Vision, OTA, March 2016
OTA’s Summary of Future State Vision Offsite, OTA, March 19, 2016
OTA—Revenue Administration Program, Rwanda End-of-Project Review, George Dodd and
Leslie Lehrkinder, May 11, 2022
Debt Management Manual, RRA, Domestic Taxes Department
Taxpayer Perceptions on the RRA Appeal System, Strategy and Risk Analysis Department,
Research and Policy Analysis Section, Theonille Mukamana, August 2020
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
49
Annex B . K e y Informants Interviewe d
NAMES REDACTED
RWANDA REVENUE AUTHORITY—PHYSICAL INTERVIEWS
Assistant Commissioner for Charge of Registration, Filing and Payment
Compliance Improvement Planning Officer
Director of Effective Management of RRA Risk Registers
Professional in Charge DTD Performance Management
Single Project Implementation Unit Coordinator
Assistant Commissioner in Charge of Research, Planning, M&E Division
Tax Appeals Management and Dispute Resolution Officer
Assistant Commissioner in Charge of Human Resource Division
Institutional Review Officer/ Implementation of RRA New Operating Model
Director of Change Management and Development
Professional in Charge of Security and Safety
Audit Quality Review Officer
Deputy Commissioner General
Audit Support Officer
Assistant Commissioner in Charge of Debt Management Division
RWANDA REVENUE AUTHORITY—VIRTUAL INTERVIEWS
Commissioner General
Commissioner for Domestic Taxes
Commissioner for Internal Audit and Integrity
OTA—VIRTUAL INTERVIEWS
Advisor
Advisor
Program Manager and Advisor
Advisor
AMERICAN CHAMBER OF COMMERCE IN RWANDA
Board President
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
50
Annex C . Ev alu ati o n Tea m
Steve Crout spent 25 years with the Canadian Government in Employment Insurance, Goods
and Service Tax, Investigations, and Income Tax Administrations. He started international work
in 2001 and has worked in developing and post-conflict countries, covering all functions of the
revenue administration. Steve resides in Western Canada.
Uzma Ashraf Barton is a Public Financial Management and Economic Policy expert with more
than fifteen years of experience in more than 12 countries working on public finance, tax policy
and administration, legal and financial market development, governance reform implementation,
and capacity building and research. Her academic research is widely published, and her book,
Rethinking Finance, (Kluwer, 2018) explores issues of systemic financial risks and suggests
sustainable financing options. Ms. Barton has a PhD and an LLM in Financial Law.
Joseph Kotun is a Director of Monitoring, Evaluation, and Learning at Bixal. He has more than
25 years of technical experience in international development, focused on evaluation. Before his
current role at Bixal, he worked extensively as an evaluation Team Leader for contractors with
USAID, MCC, USDA, U.S. Department of State, World Bank, DFID, Asian Development Bank,
and other private sector and NGO clients. He has a master’s degree in international development
and management from the School for International Training Graduate Institute and a Bachelor of
Science degree in Earth and Biological Sciences from McGill University.
Michael Tatone is a Senior International Evaluation Project Manager at Bixal. He has 13 years
of technical experience in evaluation and research, managing and providing technical support for
evaluations and impact studies in conflict and violence in Colombia, Brazil, and Chicago.
Previous roles have included Research Manager at the University of Chicago and various Project
Manager and Researcher roles in academia and NGOs. He has a Master of Political Science
degree from Universidad de Los Andes in Colombia and a Bachelor of Arts degree in Political
Science from University of Wisconsin-Eau Claire.
Mats Alentun is a senior evaluation, program planning, and good governance expert with a
strong international background. He has 15 years of technical experience in evaluation and
evaluability and 20 years of technical experience in international development. His professional
experience includes six long-term assignments in four developing countries (Mozambique,
Zimbabwe, Tanzania, and Bangladesh), various short-term missions in Africa, Southeast Asia,
and the Balkans, as well as top positions in seven Swedish central government agencies. He has
a Master of Economic Sciences degree from the Swedish School of Economics and Business
Administration in Helsinki, Finland. He currently resides in Kigali, Rwanda.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
51
Annex D . Ev alu ati on team’s re sp onse to t he O TA Revenu e
Te am ’s Com ments
The evaluation team received the OTA Revenue Team’s written response to the evaluation report
on July 17, 2023. Below is the evaluation team’s response:
Thank you for taking the time to review and respond to the Evaluation Report for the Rwanda
RRA project. Bixal and its partners strive to ensure accurate and objective findings and
conclusions when completing any evaluation and appreciate the feedback provided to ensure this
is achieved. While the evaluation team appreciates OTA’s perspective, the evaluation also
gathered input from a variety of people and resource documents to support its conclusions. The
purpose of the evaluation is to provide OTA with in-depth, third-party, retrospective insights of
its project activities.
It is generally acknowledged that some of the findings presented to the client may not be of a
favorable nature but are meant to provide learnings and advance the process of technical
assistance in Rwanda and to other countries.
The comments from the OTA Revenue Team focused on three areas:
1. Inaccuracies: the evaluation team have gone through all the points presented by the OTA
Revenue Team and updated the report as appropriate. Not all the points were
incorporated because the report may reflect a different point of view than that of the OTA
Revenue Team based on the evaluation team’s access to a larger body of data.
2. Missing facts: many of these points in the OTA Revenue Team’s comments were
providing new information about the project in Rwanda which the evaluation team did
not have when formulating its findings. The evaluation team did request all relevant
information be provided to ensure the evaluation team would be able to conduct a
comprehensive review. The new information could not be incorporated into the findings
of this report.
3. Contradictory statements: As with category number one, the evaluation team has gone
through each point and updated the report if required. In many cases, the evaluation team
feels that the report is accurate, and any contradictory statements may be a result of a
difference of definition or viewpoint.
As an evaluation team, we take the concerns of the OTA advisors seriously. The evaluation team
did meet with the OTA Revenue Team on Aug 26, 2023, to discuss the issues further in a session
separate from the presentation to the Management Team of OTA. Our goal is to make the
evaluation process a learning event for everyone.
Office of Treasury Assistance: Rwanda Revenue Authority Program Evaluation Draft Final Report
52
Annex E . O TA Res po nse to “Offi ce of Tec hnical Assist ance:
Rw an da Revenue Program Evaluation Repor t ”
OTA confirms receipt of the Program Evaluation Report by Bixal (dated August 1, 2023) related
to OTA’s technical assistance engagement with the Rwanda Revenue Authority (RRA) from
2015 to 2020. OTA provided a written response to the evaluation report on July 17, 2023, and
OTA management met with Bixal representatives on July 28, 2023, to discuss the report’s
findings and recommendations. The dialogue was both informative and insightful and OTA
appreciates that Bixal incorporated technical corrections provided by the program.
OTA recognizes the importance of independent, third-party project evaluations in compliance
with the Foreign Aid Transparency and Accountability Act (FATAA). OTA also appreciates the
opportunity for program learning presented by the evaluation and its findings and
recommendations. Since the completion of the project with the Rwanda Revenue Authority in
2020, OTA has taken significant steps to strengthen important aspects of the technical assistance
project lifecycle, particularly as it relates to project design and monitoring. These changes
address several of the recommendations included in the evaluation report, including development
of logical frameworks, improved project scoping and problem definition, and systematized
monitoring of project progress against defined outputs and outcomes. The program has also
started to design outcome indicators for each project. Taken together, OTA believes that these
actions will improve the program’s ability to achieve impactful and sustained reforms and
counterpart capacity. Again, OTA appreciates the results and lessons gained from this
evaluation and will ensure that they will be integrated, as appropriate, into future technical
assistance projects.