Consider the need for educational change facing the Grand City community and how members of the task force have begun working toward meeting that need. You have reviewed the data reflecting demographics for the community and reflected on how the data should inform decisions regarding changes to your specialization. While changes may be welcomed by some members in the community, as a leader of change, you must also take into account why some members resist change. Perhaps the issues facing Grand City are similar or reflective of the needs of your own community. How might you support your community in a change process? How might you encourage change-resistant community members to work toward necessary change?
For this Discussion, you will use data to develop a specialization-specific scenario representing a need for change in your educational or community setting.
Note: You will develop the scenario in this Discussion for your colleagues to respond to and address the scenario from your perspective in Module 4 Discussion 2.
To prepare:
Review the assigned Fullan (2016) chapters for this module. Consider the processes and concepts regarding change and how educational leaders can support both.
In the City Hall location in Grand City, revisit the media of the task force’s opening meeting. Think about the issues addressed by the members represented and how communities strive to meet the educational requirements of their children/students and the needs of their members. Think about how the issues raised in the video and those by your colleagues thus far in the course resonate with your specialization in your own educational or community setting.
Select an issue that represents a need for change in your educational or community setting and involves, or has the potential to involve, your specialization area. Locate and review the existing data related to this issue. What is the data telling you about the issue and potential educational and/or community changes?
Note: If you are unable to determine an issue from your own educational setting or find data related to your issue, you can select an issue and review the data provided in Grand City.
Based on the data, develop a more detailed scenario of the issue for your Walden colleagues to respond to with evidence-based strategies for leading the change process.
Note: You may wish to base your scenario on a personal experience or on the information provided in Grand City.
A scenario description that explains:
The details of the issue, including the history of the problem, who is involved, and why the problem has occurred. Be sure to include specific reference to data to support your explanation.
How the issue presents a need for change within your specialization in your educational or community setting.
For this Discussion, and all scholarly writing in this course and throughout your program, you will be required to use APA style and provide reference citations.
Learning Resources
Note: To access this module’s required library resources, please click on the link to the Course Readings List, found in the Course Materials section of your Syllabus.
Required Readings
Fullan, M. (2016). The new meaning of educational change (5th ed.). New York, NY: Teachers College Press.
· Chapter 3, “Insights into the Change Process” (pp. 39–53)
· Chapter 5, “Planning, Doing, and Coping with Change” (pp. 82–96)
Giancola, S. (2014). Evaluation matters: Getting the information you need from your evaluation. U.S. Department of Education. Retrieved from https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf
Jayaratne, K. S. U. (2016). Tools for formative evaluation: Gathering the information necessary for program improvement. Journal of Extension, 54(1), 28. Retrieved from https://www.joe.org/joe/2016february/tt2.php
Henson, H. (2016). Data quality evaluation for program evaluators. Canadian Journal of Program Evaluation, 21(1), 99-108. doi:10.3138/cjpe.261
Document: Action Plan Template 1 (Word document)
Document: Action Plan Template 2 (Word document)
Required Media
Grand City Community
Go to the Grand City Community and click into City Hall to review the following for this module:
Laureate Education (Producer). (2017a). Grand City opening task force meeting [Video file]. Baltimore, MD: Author.
Laureate Education (Producer). (2016b). Grand City education and demographic data files [PDF]. Baltimore, MD: Author.
Research and Practice Notes/Notes sur la recherche et les méthodes
Data Quality Evaluation for Program Evaluators
Harold Henson
Hensky Consulting
Abstract: Problems with data quality are an ongoing challenge in the fi eld of program
evaluation. In this article the author argues that the same basic process and methodol-
ogy used in program evaluation in general could be applied to the assessment of data
quality. It is argued that standardized evaluation questions and lines of evidence can
be modifi ed to assess quality of data generated by programs for evaluation.
Keywords: Big Data, data quality, measurement error, program evaluation
Résumé : Un défi chronique auquel les évaluateurs doivent faire face concerne la
piètre qualité des données dans le domaine de l’évaluation de programmes. Dans cet
article, l’auteur soutient que le processus et la méthodologie utilisés dans le cadre de
l’évaluation de programmes en général peuvent être appliqués à l’évaluation de la
qualité des données. Notamment, les questions et les sources de données standardi-
sées peuvent être adaptées à l’évaluation de la qualité des données générées par les
programmes à des fi ns d’évaluation de ces derniers.
Mots clés : mégadonnées, qualité des données, erreurs de mesure, évaluation de
programmes
INTRODUCTION
Th e administrative data used to operate the programs studied by evaluators do not
always live up to their potential. It is usual to fi nd in evaluation reports that the fi nd-
ings were to some degree compromised due to some issue related to data quality.
Th is was best summarized in the 2009 report by the Offi ce of the Auditor General,
which found that 17 of 23 evaluations examined did not have access to adequate
program performance information ( Offi ce of the Auditor General, 2009 ).
An initial reaction to this challenge is that program managers should simply
fi x data problems. However, the situation is more nuanced than may appear at
fi rst. In many cases, the solution to these issues is not easily resolvable and in
most cases is not clearly understood. In fact, the issue of data quality has become
a separate vein of research associated with the move toward the greater use of
administrative data as a source of business intelligence.
Corresponding author: Harold Henson; [email protected] ; http://
henskyconsulting.com/ .
© 2016 Canadian Journal of Program Evaluation / La Revue canadienne d’évaluation de programme
31.1 (Spring / printemps), 99–108 doi: 10.3138/cjpe.261
mailto:[email protected]
Hensky Consulting
Hensky Consulting
100 Henson
© 2016 CJPE 31.1, 99–108 doi: 10.3138/cjpe.261
Statisticians have long appreciated the possible importance of the issue. In
general, the statistician’s approach is to attempt to model the problems in data
with error terms that capture the diff erence between the values in the database and
their true values. Th ese eff orts have yielded useful theoretical results for program
evaluators. For example, if the measurement errors are purely concentrated in the
outcome variable, it may well be possible to resolve the problem through larger
sample sizes. However, if there is a diffi culty in the measurement of program par-
ticipation, then there may be a downward bias in the measured program impact
due to attenuation bias ( Wooldridge, 2002 ). Although these insights provided in
the statistical literature are useful, they have rarely resulted in substantial improve-
ments to program evaluations.
Th e computer science community has also recognized the importance of this
issue. A vein of literature dating as far back as the late 1990s focuses on data quality
issues (see Wang, 1998, and General Accounting Offi ce, 2009, for examples), and
many ideas brought forward may prove interesting to our fi eld. For example, Wang
(1998) considered consistency of data as an important dimension of data quality.
More recent work by Zhu ( Zhu, Madnick, Lee, & Wang, 2014 ) proposes the use of
15 techniques, including artifi cial intelligence, to resolve some data quality issues. In
addition, recent discussions on the value to organizations of Chief Data Offi cers may
provide a structure similar to a departmental evaluation committee to ensure gov-
ernance and a level of discipline in ensuring data quality throughout the organiza-
tion ( Lee, Chung, Madnick, Wang, & Zhang, 2012, for an introduction to this topic).
It is argued in this article that evaluators are positioned to provide useful as-
sessments on the quality of administrative data. In an ideal world, evaluators and
program managers would work toward the resolution of quality issues before an
evaluation begins. In many cases, the major benefi t of resolving data quality issues
early on will be felt during the evaluation planning phase: a thorough assessment
of potential problems with data will allow more precise estimates of the level of
resources necessary to produce evaluations that are of suffi cient quality.
Th is is on some level a bold proposition, as evaluation is typically used to gen-
erate information on program impacts. However, noted evaluation commentators
such as Stuffl ebeam ( Stuffl ebeam & Coryn, 2014 ) have suggested that data may be
considered a potential subject of an evaluation. Th e justifi cation for use of program
evaluation techniques rests in the capacity of evaluators to capture the experience
of the users of the data systems in a fashion similar to what they would use for any
other type of program. Th e goal is to identify problems with the data without any
specialized IT knowledge. For this reason, this proposed application of evaluative
thinking is not overstepping the competence of the evaluation community.
OVERVIEW OF APPROACH
Evaluators do many things well. Th e essence of this approach is to apply their
competencies that have worked well in the evaluation of programs to the issue
of the quality of administrative data. However, care is taken to avoid suggesting
Data Quality Evaluation 101
CJPE 31.1, 99–108 © 2016doi: 10.3138/cjpe.261
that program evaluators overreach their competence. Th ere will not be an attempt
to assess the merits of various computer systems or soft ware. Th e evaluation ap-
proach will be used to assess the quality of the data as experienced by the analyst.
Th is is a domain for which evaluators are well-equipped.
Th e proposed approach features a series of basic questions, which will be dif-
ferent than those commonly used to evaluate programs. Th eir source is the larger
“Big Data” ( Sebastian-Coleman, 2013 ) literature, which focuses on the user expe-
rience rather than the merits of various computer systems. In fact, it is anticipated
that the use of these questions will enable a broader acceptance of the evaluation
report outside of the narrowly defi ned evaluation community.
It is then suggested that several low-cost lines of evidence be collected to sup-
port this approach. In most cases, these activities will be familiar to evaluators.
Other suggested activities derive from informal discussions with experienced data
analysts and the computer science literature. A small team of technically skilled
evaluators should be able to complete the work in a few months before starting
an evaluation. Th roughout all of these activities, the questions will be posed from
a user perspective rather than a systems perspective.
ASSESSING DATA QUALITY: QUESTIONS
Successful evaluative exercises are structured around a set of questions that frame
the collection of evidence. Th e cumulative evidence forms the empirical basis for
the conclusions in the study. Th e evaluation of data quality is no diff erent.
Th e fi ve generic questions outlined below should form a good starting point
in the development of the evaluation questions. Th ey are derived from the seminal
work of Sebastian-Coleman (2013) in the data quality literature. It is important to
note that the fi ve questions lead to an assessment of the data from purely a user
standpoint and do not attempt to conceptualize the collection of data as if it were
a program. For this reason, there is no mention of the cost of the data. Th is, of
course, may be seen as a limitation.
Although these fi ve questions provide a basis to develop the specifi c questions
that guide the assessment, they need not be an end point. Each database is used in
slightly diff erent ways, and each evaluation has diff erent issues. As a result, the fi nal
set of questions in any data quality evaluation may well be diff erent than the fi ve
suggested here, which provide a good starting point. Th is is similar to the way that
Canadian federal government evaluators may use the fi ve core questions required
by the Treasury Board of Canada’s Directive on the Evaluation Function ( 2009 ).
Is the Database Complete?
Probably the most important question in assessing the database from the perspec-
tive of the evaluator is whether it is complete. Th e degree of completeness is not a
simple binary assessment, but involves an examination of the data from diff erent
perspectives. However, the most important perspectives are those of the program
participants and their key characteristics.
102 Henson
© 2016 CJPE 31.1, 99–108 doi: 10.3138/cjpe.261
Typically, an administrative database can be seen as a very large spreadsheet.
Each row will represent a program participant, and each column or fi eld will
represent a specifi c characteristic. It is more complex in practice, given the wide-
spread use of relational databases, but for this introductory discussion, viewing
the database as a spreadsheet is suffi cient.
A complete database will have one column for each characteristic of the
service provided to the client by the program. Unfortunately, this is not always
possible for various technical reasons. For example, it may be the case that some
aspects of a program are not automated, and information is stored in paper fi les.
Another possible reason is that some information, such as participation in jury
duty, may be suppressed, as it is too sensitive.
Some important fi elds may contain qualitative contextual data that may vary
substantially in their level of completeness. For example, a project description may
be exactly the same as the previous year, with only the year being changed. Other
text fi elds may contain a few random characters that are suffi cient to fool the data
entry soft ware into allowing the form to be considered complete.
Th e other important perspective is that of the individual participants. Certain
participants may have their data omitted from an electronic database. It may be
that their fi le contains unusual complexities that forced the processing on paper,
or that a small regional offi ce may not be automated. In either case, possible biases
may remain in the existing computerized database. In such a situation, the count
of records on the database will be less than the number of clients.
Finally, the lack of proper documentation, such as user guides or metadata,
is by far the most serious problem. Virtually all programs have some document
that they can refer to as “offi cial.” Unfortunately, there are large variations in the
actual quality, as the databases are generally unusable without contact with a
person in the program area who is familiar with the oral traditions that surround
the use of the data. In general, evaluators will have to assess the quality of this
documentation from two diff erent perspectives. First, the overview should give a
prospective analyst a good perspective of how the data fi t together and relate to
the program. Second, the detailed fi eld-by-fi eld documentation is crucial when
using the actual database.
Care should be taken to try to see beyond the offi cial documentation. A body
of informal documentation, such as tutorials, course notes, or online help fi les,
quite oft en supports the use of many databases. Many of the more sophisticated
data management systems such as STATA provide facilities to make the database
self-documenting. For example, within STATA, users can upload help fi les as well
as comments. Labels attached to the various values of qualitative variables can also
replace written documentation.
Are the Data Timely?
It is important to verify that the data available to evaluators are reasonably up-
to-date. However, it is important to note the volatility of the most recent obser-
vations: it is not unusual for data to be modifi ed frequently aft er initial entry.
Data Quality Evaluation 103
CJPE 31.1, 99–108 © 2016doi: 10.3138/cjpe.261
Unfortunately, this may render the most recent data unusable from the perspec-
tive of statistical analysis due to a lack of precision.
A more important feature than the currency of the most recent observations
may be the existence of historical data. For example, Canadian federal programs
are usually evaluated every fi ve years. Th erefore, it should be possible to go back
more than fi ve years to track the changes that have occurred in the program since
the last evaluation.
Are the Data a Valid Representation of the Program?
Th e question of the “validity” of the data can be the most abstract. Essentially, an
evaluator may ask if a particular fi eld is a valid measure of some aspect of program
delivery as represented in the program logic model. It is possible that a given fi eld,
or combination of fi elds, will be taken as representing a particular concept when,
in fact, it represents something else. Th is can occur even if the measures reported
in the data are accurate.
Application processing times provide a classic example. It may be possible
that a measure of processing time will start with a completed application and end
with the provision of a service. However, this measure may not be valid if the ap-
plication process requires the client to interact with program staff to answer ques-
tions. A more valid measure may have used the point in time when the applicant
fi rst started these discussions as it may have taken several attempts to complete
the form satisfactorily. Unfortunately, the data that are available are not a valid
representation of the client experience in this case.
How Consistent Are the Data?
Data can be consistent either through time or across organizational divisions at
any point in time. In some cases, a lack of consistency (i.e., inconsistencies in
codes used in the database from year to year) does not indicate a problem from
an administrative perspective, although it may render the data less useful from a
statistical or evaluative perspective.
Issues related to data consistency may be more prevalent in cases where the
analysis occurs over a longer time span, or in evaluations dealing with large pro-
grams or organizations (see Arrow, 1974 , chapter 2, for a theoretical discussion,
and Canbäck, Samouel, & Price, 2006 , for empirical work). In other words, in
large organizations, more authority is typically delegated to managers, which may
lead to diff erent interpretations of directives regarding defi nitions underlying the
data systems. Th ere may be cases where the actual words have diff erent meanings
in diff erent contexts. For example, “manufacturing sector” may mean something
diff erent in a part of the country dominated by the textile industry rather than the
pulp and paper industry. Th ese issues may be very relevant if matching techniques
are used as a statistical test of program causality.
In addition, organizations evolve through time, as both the internal and
external environment forces change. Evaluators have to anticipate that a lack of
consistency may render some statistical methods, such as the Interrupted Time
104 Henson
© 2016 CJPE 31.1, 99–108 doi: 10.3138/cjpe.261
Series technique that uses comparisons in time, less reliable in terms of the esti-
mation of program causality ( Stuffl ebeam & Coryn, 2014 ). Oft en, changes to data
standards occur at the same time as changes in the program, which render the
evaluation of policy changes less reliable.
Is There an Issue with Integrity?
Data can contain errors for several reasons, many of which can easily be recti-
fi ed. Much of the time, extreme outliers are easy to identify and manage. Smaller
errors may be more diffi cult to spot. It will also be more diffi cult to validate data
that originated further back in time as human memory may have faded or key
individuals have left the organization.
It should be noted that this is an area where it is advantageous for evaluators
to work with internal auditors. Evaluators tend to treat data measurement errors
as simple random events. To internal auditors, the reasons for the errors may be
highly signifi cant. Internal auditors may also have conducted studies that have
validated or can explain many of the observations that appear unusual.
In many cases, the magnitude of this type of error will not be suffi cient to
aff ect the overall evaluability of the program. However, there can be cases where
variations can be empirically important. For example, if a program for young
people defi nes youth as those who are 25 or younger at the date of application, any
changes or fl exible application of this criterion may render regression discontinu-
ity techniques less reliable.
THE LINES OF EVIDENCE
All good evaluations are based on multiple lines of evidence. Many of the pro-
posed lines of evidence are similar to what an evaluator uses for a traditional
program evaluation. Others have been known to be useful among applied statisti-
cians working with administrative data. As with any evaluation, the results stem-
ming from one of these lines of evidence should not be taken as decisive. Strong
conclusions can come only if these lines are used in combination with each other.
Data Profi les
A data profi ling exercise involves a systematic analysis of all, or least a sample,
of the fi elds in the database. Th is is usually the most labour intensive of the lines
of evidence. It involves someone who has never worked with the data before,
tabulating every fi eld, then comparing the results against the documentation.
Th is will capture the perspective of an inexperienced user. Th is will include but
not be limited to
• Examining the statistical characteristics of the data, such as means and
medians, in comparison with a reasonable interpretation of the descrip-
tion of the variable;
• Checking implausible outliers or unexpected negative values;
Data Quality Evaluation 105
CJPE 31.1, 99–108 © 2016doi: 10.3138/cjpe.261
• If the variable is an integer that refers to a category, such as gender or
province, verifying that all values are described in the documentation; and,
• Graphically examining the distribution for unexpected spikes and troughs.
Th e above procedures would simply be applied in a mechanical fashion one
fi eld at a time. If time permits, comparative analysis may be undertaken.
It should be noted that special challenges are posed by text fi elds. In this
case, an analyst could read a random sample and rate each fi eld individually. Au-
tomated indicators may include scanning for duplication of phrases or the use of
phrases that are nonsensical. It is not uncommon to see the same typos reappear
from one record to another.
Th e raw output from data profi ling will be highly repetitious and voluminous.
Rather than collating the individual reviews in a very large document and presenting
the material as a formal report, it may be more productive to store the results in an
environment that would permit rapid retrieval and analysis, such as Microsoft Access.
Another way of managing the volume of the data is to collate the profi les by
theme. Th is may make for better reading, and also allow more ready assessment of
the completeness of the database. It should be noted that if this synthesis is done
well, it can form a highly eff ective alternative documentation that will have value
for the organization outside of the evaluation itself.
Key Informants
Th e typical group of users of any database is small and highly varied. Th us it is
unlikely that surveys of the users would be useful. However, key informant inter-
views have enough fl exibility to ensure that the questions are relevant to the style
of each type of user. Diff erent interview protocols should be developed for each
user type. Not only will it be necessary to adjust the level of detail in the response,
but it will also be necessary to adjust for inherent biases. Th e three classes of users
suggested in the following list may be useful in many situations:
Program Managers: Program managers will have a strategic perspective on
the program and how the data can be used to answer questions from senior man-
agement. Evaluators may obtain a high degree of cooperation, if the managers
think that they will get better data as a result of the exercise.
Power Users: Th e power users (main analysts) are in most cases the easiest
to please and best informed to discuss the potential inherent in the database.
Interviews with them may be longer and more detailed in nature. Key pieces of
information that may arise from such an interview may be informal knowledge
about how a given fi eld should be interpreted or the history behind suspicious
inconsistencies in the data. Verifying and documenting this knowledge might
benefi t the entire organization.
Inexperienced Users: It is important to have the perspective of individuals who
have attempted to use the data without the benefi t of an oral tradition that may ex-
ist within the program. Th is will allow senior management to be able to gauge the
extent to which the database is able to support broader use within the organization.
106 Henson
© 2016 CJPE 31.1, 99–108 doi: 10.3138/cjpe.261
Examples of Success
In essence, one of the most convincing validations of a database is its fi nal product.
In fact, it would be very diffi cult to claim that a database was problematic if there
were a large number of successful reports based on the data. However, diff erent
kinds of products will highlight diff erent aspects or qualities of the data. Th ey can
be seen in terms of the extent to which they address various questions about the
data. Database products can be sorted into two categories:
• First, if the program is producing regular reports featuring detailed
statistical annexes, it is likely that it has very good control of the data.
Frequency is a key indicator. If a program is only able to publish reports
on an annual basis with signifi cant internal eff ort, then there are likely
problems with the data that must be resolved manually. However, more
frequent publications would indicate a high degree of control over the
data and confi dence that the numbers could be released with less review.
It is still useful to keep in mind, however, that low-quality data could still
be published on a regular basis in some cases.
• Irregular reports produced for special purposes also provide evidence of
data quality. Frequently, these studies will be conducted by individuals
outside of the program, who will put considerable thought into some
narrow aspect of program operations. Th ey will also study the documen-
tation with fresh eyes and provide feedback on its quality. Past evalua-
tions may provide evidence of good historical data.
Replication of Known Totals
Replicating known totals with the administrative data is a good fi rst test of data
quality. For one, it is a very good way to address questions of completeness of the
data. As well, the quality of the documentation is put to the test here. Th is will
also test the volatility of the data if the only explanation for the variation between
the results and the published totals are data revisions.
However, it should be pointed out that at times the published totals can be
very diffi cult to replicate without the full methodology as many detailed ad-
justments must be made to the data during the calculation. Unfortunately, the
methodology behind the “offi cial” totals may not be readily available. Th is may
represent a fault in the metadata (documentation) rather than the actual data.
Still, it is important for an evaluator to be aware of this, as it is generally essential
that an evaluator understand all the theoretical thinking that may be behind the
offi cial estimates of total program activity.
Case Studies
A fi nal line of evidence can be an in-depth analysis of particular fi elds. In a case
study, evaluators may examine how one specifi c variable is being generated and
whether it is suitable for use in an evaluation. Such an analysis may generate
Data Quality Evaluation 107
CJPE 31.1, 99–108 © 2016doi: 10.3138/cjpe.261
advance knowledge of the possible biases caused by measurement error. It also
may be the case that observations for a given fi eld are missing in such a way as to
possibly bias the analysis.
In general, there may be two ways the candidate fi elds are selected. First,
there may be variables that are crucial to the evaluation as a whole. Second, curi-
ous patterns may emerge during the above data profi ling that warrant further
investigation.
Where the data profi ling was done at a distance from the program area so
as to maintain objectivity, the case studies will require closer interactions with
program staff .
THE FINAL PRODUCT
Th e fi nal report can very much resemble a program evaluation, as it will be a
synthesis of technical reports. However, the format of the fi nal report should suit
the needs of the organization. As this work will not be done for accountability
reasons, or to satisfy policy requirements, the report should be tailored around
internal needs. In fact, a fi nal report may not be necessary and the technical re-
ports associated with each line of evidence may be suffi cient. Th e fi nal decisions
about the nature of the output may come from the senior management, which
may include a Chief Data Offi cer.
It is anticipated that these reports will have three immediate uses:
Support for Future Evaluations
Th e report should help program managers resolve problems with the databases
before the evaluations occur. Ideally they should be available to the program
manager one or two years before the actual evaluation. If possible, the report may
even include detailed recommendations, such as areas where the documentation
can be improved.
Support for Evaluation Planning
Evaluators will know well ahead of time what evaluation questions can be an-
swered with a given budget. Th is will allow for more precise calibration of evalua-
tion budgets, as it will be less necessary to set aside funds for special contingencies.
Th ey may be used as input to evaluability assessments.
Support for Broader Use of Data
Th ese reports can support the broader use of the data outside the management
of the individual program. Improvements in technology have removed many of
the roadblocks to the realization of the potential of administrative data, although
privacy issues are still important. However, data quality and the uncertainty sur-
rounding it are oft en the fi nal roadblock to incorporation of the use of the data
into the knowledge management strategies of the larger organizations.
108 Henson
© 2016 CJPE 31.1, 99–108 doi: 10.3138/cjpe.261
REFERENCES
Arrow , K. ( 1974 ). Th e limits of organization . New York, NY : Norton .
Canbäck , S. , Samouel , P. , & Price , D. ( 2006 ). Do diseconomies of scale impact fi rm size and
performance? A theoretical and empirical overview . Journal of Managerial Economics ,
4 ( 1 ), 27 – 70 .
General Accounting Offi ce. ( 2009 ). Assessing the reliability of computer-processed data.
Washington, DC : Author .
Lee , Y. , Chung , W. , Madnick , S. , Wang , R. , & Zhang , H. ( 2012 , December). On the rise of
the chief data offi cers in a world of big data . Paper presented at ICIS 2012 Sim Aca-
demic Workshop , Orlando, FL.
Offi ce of the Auditor General of Canada. ( 2009 ). Evaluating the eff ectiveness of programs.
Ottawa, ON : Author .
Sebastian-Coleman , L . ( 2013 ). Measuring data quality for ongoing improvement: A data
quality assessment . Waltharn, MD: Elsevier .
Stuffl ebeam , D. , & Coryn , C . ( 2014 ). Evaluation theory, models, & applications . San Fran-
cisco, CA : Jossey-Bass .
Treasury Board of Canada . ( 2009 ). Directive on the evaluation function . Ottawa, ON :
Author .
Wooldridge , J. M. ( 2002 ). Econometric analysis of cross section and panel data hardcover .
Cambridge, MA : MIT Press .
Wang , R. ( 1998 ). A product perspective on total data quality management . Communica-
tions of the ACM , 41 ( February ), 58 – 65 . http://dx.doi.org/10.1145/269012.269022
…