class: center, middle, inverse, title-slide # Research Transparency in Social Science ## Prof. Theodoridis Seminar, UMass ### Fernando Hoces de la Guardia, BITSS ### March 2021 |
slides
--- background-image: url("Images/BITSSlogo.png"), url(Images/cega.png) background-size: contain, 200px background-position: 50% 100% , 0% 100% count:true #[BITSS](https://bitss.org)   .pull-left[ The Berkeley Initiative for Transparency in the Social Sciences works to improve
the credibility of science by
 advancing transparency,
 reproducibility, rigor, and
 ethics in research. .font130[**Core Team**] Aleks Bogdanoski Fernando Hoces de la Guardia Katie Hoeberling Edward Miguel ] .pull-right[ .right[ We are part of the Center for Effective Global Action ([CEGA](https://cega.berkeley.edu/)).

 .font130[**Many Others**] CEGA staff Undergrad and Graduate RAs Catalysts Outside Collaborators (Researchers, Programmers) ]] --- background-image: url("Images/ecosystem.PNG"), url(Images/cega.png) background-size: contain, 200px background-position: 50% 100% , 0% 100% name: about-bitss count:true # Part of the much larger Open Science Community --- # Table of Contents
 .font130[ 1. [Problems](#motivation) 2. [Solutions](#bitss) 3. [Accelerating Computational Reproducibility](#acre) 4. [Open Policy Analysis](#opa) ] --- count:false # Table of Contents
 .font130[ 1. [**Problems**](#motivation) 2. [Solutions](#bitss) 3. [Accelerating Computational Reproducibility](#acre) 4. [Open Policy Analysis](#opa) ] --- name: motivation # What does it mean to do good science? .font140[ Scientific Norms (Merton, 1942, [1973](https://sciencepolicy.colorado.edu/students/envs_5110/merton_sociology_science.pdf)): - **Disinterestedness**: search for the truth - **Organized skepticism**: peer review and replication - **Communality**: researchers should share all the scientific output (papers, data, code and materials) - **Universalism**: validity comes from quality of a scientific finding and not the hierarchies of those making the claims. ] --- background-image: url("Images/AMdV2007_01.PNG") background-size: contain background-position: 50% 100% count:false # .font80[Researchers and Scientific Norms (Anderson et. al., 2007)] --- background-image: url("Images/AMdV2007_02.png") background-size: contain background-position: 50% 100% count:false # .font80[Researchers and Scientific Norms (Anderson et. al., 2007)] --- background-image: url("Images/AMdV2007.png") background-size: contain background-position: 50% 100% count:true # .font80[Researchers and Scientific Norms (Anderson et. al., 2007)] --- background-image: url("Images/Brodeur.PNG") background-size: contain background-position: 50% 100% # P-hacking (for Economics: Brodeur et. al 2016, 2020) --- background-image: url("Images/Tess.PNG") background-size: contain background-position: 50% 100% # Publication Bias (Franco et. al. 2014) --- # .font80[ Low Replicability and Reproducibility ("Reproducibility Crisis")]

 .font180[ | Replication in Social Sciences
(same method, different sample) | Reproduction in Economics
(same data and methods) | |------------------------------------------------------------------- |------------------------------------------------------ | | OSC ([2015](https://docs.google.com/document/d/1mm_4HZnEz_2Bh8XuiS2tpqCH08MFPyqUhi1baKPqR8Y/edit#heading=h.7vqf2cziid7z)): 30%-60% | Chang & LiRobustness ([2015](https://www.nowpublishers.com/article/Details/CFR-0053)): 43% | | Camerer et. al. ([2016](https://science.sciencemag.org/content/351/6280/1433)): ~60% | Gertler et. al. ([2017](https://www.nature.com/articles/d41586-018-02108-9)): 14% | | Nosek & Camerer et. al. ([2018](https://www.nature.com/articles/s41562-018-0399-z)): ~60% | Kingi et. al. ([2018](https://hautahi.com/static/docs/Replication_aejae.pdf)): 43% | | Klein et. al. ([2018](https://journals.sagepub.com/doi/10.1177/2515245918810225)): 50% | Wood et. al. ([2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0209416#abstract0)): 25% | ] --- count:false # Table of Contents
 .font130[ 1. [Problems](#motivation) 2. [**Solutions**](#bitss) 4. [Accelerating Computational Reproducibility](#acre) 5. [Open Policy Analysis](#opa) ] --- name: bitss # .font80[Main Solutions] .font140[ 1. Registrations 2. Pre Analysis Plans 3. Computational Reproducibility ] --- # Registrations
 .font130[ - A registration is a record that contains minimal information about a study: title, authors, study country, status, keywords, abstract, start and end dates, outcomes, intervention information, basic research design, whether or not treatments are clustered (when performing an RCT), and Institutional Review Board (IRB) information. - Preferably, it should be recorded before analyzing data - **The main goal:** track the complete body of knowledge in a topic of research, regardless of the direction and magnitude of the results. ] --- background-image: url("Images/plosone.PNG") background-size: contain background-position: 50% 100% # Why Register: Kaplan and Irvin (2015) --- # Pre-Analysis Plans
 .font140[ - PAPs are **extensive** methodological descriptions of the analysis to be performed before the endline data is collected - Helps to prevents p-hacking - Only way to guarantee accurate statistical testing - Distinguishes confirmatory from exploratory analysis ] --- # How to do a PAP .font130[ Christensen, Freese and Miguel (2020) suggest detailed information on: .pull-left[ 1 - Detail study design 2 - Sampling details 3 - Outcome Measures 4 - Index construction 5 - Multiple hypothesis adjustments 6 - Subgroup analysis ] .pull-right[ 7 - Direction of the effect (optional) 8 - Exact statistical specification 9 - Structural model (optional) 10 - Timestamp ] ] --- # Common Concerns About PAPs
 .font130[ | Critique | Response | |:----------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | PAPs take too much time and are
too difficult (Olken 2015) | A PAP changes the timing of the analytic
component, not clear that it increases it | | Scientific discovery often comes
from surprises. PAPs stifle
discovery (Olken 2015) | PAPs do not prevent researchers from doing
exploratory work; they only require
researchers to be clear about the objectives
of their analyses (Ofosu and Posner 2020). | | If replications are cheap they will
rule out false positives, making
PAPs irrelevant.
(Coffman and Niederle 2015) | Very few experiments are inexpensive as to perform
many replications. Moreover, most of the false
positives have been identified where experiments
are least expensive (lab experiments). | ] --- background-image: url("Images/GoBifo1.PNG") background-size: contain background-position: 50% 100% # Why do a PAP: Casey et. al. ([2012](https://academic.oup.com/qje/article-abstract/127/4/1755/1841616?redirectedFrom=fulltext)) --- background-image: url("Images/results_blind_review.png") background-size: contain background-position: 50% 100% # Another Reason: Register Reports --- background-image: url("Images/3s_change.png") background-size: contain background-position: 50% 100% # Change is Happenning: Christensen et. al., ([2020](https://osf.io/preprints/metaarxiv/5rksu/)) --- count:false # Table of Contents
 .font130[ 1. [Problems](#motivation) 2. [Solutions](#bitss) 4. [**Accelerating Computational Reproducibility**](#acre) 5. [Open Policy Analysis](#opa) ] --- name:acre # .font90[Accelerating Computational Reproducibility (ACRe)] .font120[ In addition to reproducibility problems, there are two additional motivations for this projects: ] ## Motivation 2: More Inclusive Concept Scholarly Output .font120[
 Clarebout Principle:

 “An article about computational science in a scientific publication is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.”

 .right[  Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download)) ]
 ] --- # .font90[Accelerating Computational Reproducibility (ACRe)] .font120[ In addition to reproducibility problems, there are two additional motivations for this projects: ] ## Motivation 2: More Inclusive Concept Scholarly Output .font120[
 Clarebout Principle:

 “An article about computational science in a scientific publication is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.”

 .right[  Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download)) ]
 ] --- # .font90[M2: More Inclusive Concept Scholarly Output] .font130[ Potential benefits of following the Clarebout Principle Well discussed potential positive effects on: - Pedagogy - Incremental generation of knowledge ] -- .font130[ Under discussed: - Possible positive effect on diversity, equity and inclusion: no connections or language skills ("appropriate politeness") required to obtain materials ] ??? An under discussed benefit of this principle is that increases access to knowledge for students who are outside of elite schools (no connections or "appropriate politeness" required to obtain materials). --- # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. -- | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | |

 | | |

 | | |

 | | |

 |
 | --- count:false # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | Scope
 (select and verify) | Data and code exist? | |

 | | |

 | | |

 |
 | --- count:false # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | Scope
 (select and verify) | Data and code exist? | | Assess | Degree of reproducibility for
specific part of the paper | |

 | | |

 |
 | --- count:false # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | Scope
 (select and verify) | Data and code exist? | | Assess | Degree of reproducibility for
specific part of the paper | | Improve | E.g. fixed paths, libraries,
added missing files, etc. | |

 |
 | --- count:false # M3: Prevent Loss of Knowledge Every semester, graduate students around **the world** take an Empirical/Applied [`...`] Economics course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. | Stage | New Knowledge | |:-----------------: |:------------------------------------------------------------: | | Scope
 (select and verify) | Data and code exist? | | Assess | Degree of reproducibility for
specific part of the paper | | Improve | E.g. fixed paths, libraries,
added missing files, etc. | | Test robustness | Results are robust to additional
specifications | --- # M3: Prevent Loss of Knowledge
 .font120[ - In each of the previous steps there was valuable new knowledge - This knowledge is not usually disseminated - End of semester report, or presentation that is not published. - Language across courses is not standard (reproduction? replication?) - Current attempts to record this are closer to a full paper format (see [Replication Wiki](https://replication.uni-goettingen.de/)) - As a result there is a large amount of missed knowledge. - Each new semester, new students repeat similar excesses and have to reinvent the wheel. - Lost opportunity for building on top of previous exercises. - Lost opportunity for aggregating the new knowledge. ] --- # Beyond Binary Judgments Reproductions can easily gravitate towards adversarial exchanges. - Early career researcher (ECR) have incentives to emphasize unsuccessful reproductions - Original authors have a more senior position and can use it to deter in-depth reproductions from ECRs. - The media also focuses on eye-catching headlines -- ### Our approach: We do not want to say > "Paper X is (ir)reproducible" -- We do want to say > "Result Y in paper X has a high/low level of reproducibility according to several reproduction attempts. Moreover, improvements have been made to the original reproduction package, increasing its reproducibility to a higher level" --- count:false # Beyond Binary Judgments Reproductions can easily gravitate towards adversarial exchanges. - Early career researcher (ECR) have incentives to emphasize unsuccessful reproductions - Original authors have a more senior position and can use it to deter in-depth reproductions from ECRs. - The media also focuses on eye-catching headlines ### Our approach: We do not want to say > "Paper X is (ir)reproducible" We do want to say > "Result Y in paper X has a high/low **level** of reproducibility according to **several** reproduction attempts. Moreover, **improvements** have been made to the original reproduction package, **increasing** its reproducibility to a higher level" --- count: true background-image: url(Images/paper-claims.svg) background-size: 610px background-position: 70% 0% # Our Framework .pull-left[ .font100[ Each **reproduction attempt**
 is centered around scientific
 **claims** One paper can contain several claims. Each claim may be
 supported by various
 **display items**: tables,
 figures & inline results. A reproduction attempt is
 at the claim level, and
 reproducers must record
 their **specifications** of
 interest. ] DI: Display Item S: Specificaiton ] .pull-right[ ] --- count: true # .font70[Large part of this exercise is about standardization] .pull-left[ - Computational Reproduction (or Reproduction) - Replication (will not mention this term again!) - Reproduction attempt - Reproduction package - Claim - Display item - Specification - Preferred specification - Raw data - Analysis data ] .pull-right[ - Candidate paper - Declared paper - Reproduction tree - Complete Workflow - Computationally Reproducible from Analytic data (CRA) - Computationally Reproducible from Raw data (CRR) - Reasonable test - Feasible test - Minimal effort ] --- count: false # .font70[Large part of this exercise is about standardization] .pull-left[ - **Computational Reproduction (or Reproduction)** - **Replication (will not mention this term again!) ** - Reproduction attempt - **Reproduction package** - Claim - Display item - Specification - Preferred specification - **Raw data** - **Analysis data** ] .pull-right[ - **Candidate paper** - **Declared paper** - Reproduction tree - Complete Workflow - Computationally Reproducible from Analytic data (CRA) - Computationally Reproducible from Raw data (CRR) - Reasonable test - Feasible test - Minimal effort ] --- background-image: url(Images/stages.svg) background-size: contain names: Stages # Stages --- background-image: url(Images/scope.svg) background-size: contain count:false names: Stages # Stages --- background-image: url(Images/scope.svg) background-size: 300px background-position: 0% 100% names: scoping # Scoping .pull-left[ 1. Select or be assigned a candidate paper 2. Check ACRE Platform for previous entries and verify availability of reproduction package (RP) 3. If not RP, leave a short record, and repeat with a different candidate paper 4. Once RP is found then candidate becomes declared paper 5. Only then: read the paper and select claim(s), display items and specification to reproduce ] .pull-right[ .font90[ >**Box 1:** Summary Report Card for ACRE Paper Entry > **Title:** Sample Title > **Authors:** Jane Doe & John Doe > **Original Reproduction Package Available:** URL/No > [If "No"] >**Contacted Authors?:** Yes/No > [If "Yes(contacted)"] > **Type of Response:** Categories (6). > **Additional Reproduction Packages:** Number (eg., 2) > **Authors Available for Further Questions for ACRE Reproductions:** Yes/No/Unknown ] ] --- background-image: url(Images/stages.svg) background-size: contain names: Stages # Stages --- background-image: url(Images/assess.svg) background-size: contain # Assessment ## Two main parts for assessment: .font130[ 1. Find all the elements behind a display item 2. Score the reproducibility of that display item ] --- background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # .font70[Identify All the Elements Behind a Display Item] Reproducers will be asked to draw a clear connection to the raw data sources mentioned in the paper and the display item under reproduction. -- ### Data sources Connect the data sources in the paper's text with specific raw data files. -- ### Analytic data sets Describe each analytic data file. -- ### Code files Inspect all code files and record all their inputs and outputs. With all the information recorded above, reproducers can use the **ACRE Diagram Builder** to generate a **reproduction tree**. --- background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # ReproducibiliTREE .font90[ ```md table1.tex |___[code] analysis.R |___analysis_data.dta |___[code] final_merge.do |___cleaned_1_2.dta | |___[code] clean_merged_1_2.do | |___merged_1_2.dta | |___[code] merge_1_2.do | |___cleaned_1.dta | | |___[code] clean_raw_1.py | | |___raw_1.dta | |___cleaned_2.dta | |___[code] clean_raw_2.py | |___raw_2.dta |___cleaned_3_4.dta |___[code] clean_merged_3_4.do |___merged_3_4.dta |___[code] merge_3_4.do |___cleaned_3.dta | |___[code] clean_raw_3.py | |___raw_3.dta |___cleaned_4.dta |___[code] clean_raw_4.py |___raw_4.dta ``` ] --- count:true background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font100[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font100[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") * | Availability of materials, and reproducibility | * |------------------------------------------------| * |Analysis| Analysis| | Cleaning| Raw | | * |Code | Data | CRA | Code | Data | CRR | * | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font100[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| * L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font100[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| * L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font100[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | * L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font100[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | * L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font100[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | * L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels .font100[ ```md Levels of Computational Reproducibility (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3: Partial analysis data & code.| ✔ ✔ | ✔ - | - | - - | - - | - | L4: All analysis data & code.....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5: Reproducible from analysis...| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8: Some raw data................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9: All raw data.................| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | * L10:Reproducible from raw data...| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% count: false # Levels: Proprietary/Confidential Data .font100[ ```md Levels of Computational Reproducibility with Proprietary/Confidential Data (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| * | | Instr. | | | Instr.| | |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3*: Partial analysis data & code| ✔ ✔ | ✔ - | - | - - | - - | - | L4*: All analysis data & code....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5*: Proof of third party CRA....| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8*: Some instr. for raw data....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9*: All instr. for raw data.....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10*:Proof of third party CRR....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels: Proprietary/Confidential Data .font100[ ```md Levels of Computational Reproducibility with Proprietary/Confidential Data (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| | | Instr. | | | Instr.| | |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | * L3*: Partial analysis data & code| ✔ ✔ | ✔ - | - | - - | - - | - | * L4*: All analysis data & code....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | L5*: Proof of third party CRA....| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | * L8*: Some instr. for raw data....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | * L9*: All instr. for raw data.....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | L10*:Proof of third party CRR....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- count:false background-image: url(Images/assess.svg) background-size: 300px background-position: 0% 100% # Levels: Proprietary/Confidential Data .font100[ ```md Levels of Computational Reproducibility with Proprietary/Confidential Data (P denotes "partial", C denotes "complete") | Availability of materials, and reproducibility | |------------------------------------------------| | | Instr. | | | Instr.| | |Analysis| Analysis| | Cleaning| Raw | | |Code | Data | CRA | Code | Data | CRR | | P | C | P | C | | P | C | P | C | | ---------|---------|-----|---------|-------|-----| L1: No materials.................| - - | - - | - | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L2: Only code ...................| ✔ ✔ | - - | - | - - | - - | - | L3*: Partial analysis data & code| ✔ ✔ | ✔ - | - | - - | - - | - | L4*: All analysis data & code....| ✔ ✔ | ✔ ✔ | - | - - | - - | - | * L5*: Proof of third party CRA....| ✔ ✔ | ✔ ✔ | ✔ | - - | - - | - | ---------------------------------|--------|---------|-----|---------|-------|-----| L6: Some cleaning code...........| ✔ ✔ | ✔ ✔ | ✔ | ✔ - | - - | - | L7: All cleaning code............| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | - - | - | L8*: Some instr. for raw data....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ - | - | L9*: All instr. for raw data.....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | - | * L10*:Proof of third party CRR....| ✔ ✔ | ✔ ✔ | ✔ | ✔ ✔ | ✔ ✔ | ✔ | ``` ] --- background-image: url(Images/improve.svg) background-size: 700px background-position: 50% 20% # Improvements

 ## Three types of improvements: .font130[ 1. Improvements at the paper level 2. Improvements at the display-item level 3. Specific future improvements ] --- background-image: url(Images/improve.svg) background-size: 300px background-position: 0% 100% # Improvements: Paper-level .font140[ - Use version control software (Git/Github). - Improve documentation: comments, indentations, object names, etc. - Re-organize the reproduction package into a set of folders and sub-folders that follow [standardized best practices](https://www.projecttier.org/tier-protocol/specifications-3-0/#overview-of-the-documentation), and add a master script that executes all the code in order, with no further modifications. [See AEA's reproduction template](https://github.com/AEADataEditor/replication-template). - Literate programming environment (e.g., Jupyter notebooks, RMarkdown) - Re-write code using a differenet statistical software (ideally open source, like R, Python, or Julia). - Set up a computing capsule (e.g., [Binder](https://mybinder.org/) and [Code Ocean](https://codeocean.com/)). ] --- background-image: url("Images/version_control_diagram.svg") background-size: contain background-position: 50% 50% # Version Control Software: Comparison of Workflows --- background-image: url(Images/improve.svg) background-size: 300px background-position: 0% 100% # Improvements: Display item-level
 .font120[ - Adding missing raw data: files or meta-data - Example: "Add raw temperature and relative humidity data" - Adding missing analytic data files - Example: "Copy the row files from Data folder into new `Analysis\trade cost\Input`" - Adding missing analysis or cleaning code - Example: "Replaced broken Wald bootstrap code with updated code/command" - Debugging code - Example: "was counting each group 4 times in round 1, so fixed that" Key challenge: constructive exchanges with original authors and other reproducers ] --- background-image: url(Images/improve.svg) background-size: 300px background-position: 0% 100% # Improvements: future possible We ask reproducer to leave concise and actionable tasks for other reproducers in the future. Example 1: >"Revise the .aml and .bat code scripts to reflect reorganized structure" Example 2: > "Provide data and codes generating the other two figures in the paper, which are not given in the replication file." Example 3: > "Table 3 can be reproduced identically from the [...] analytic data files. I was not able to reproduce the analytic data files due to lack of access to ArcGIS software, but the code scripts and raw data files [...] are included in the reproduction package." --- background-image: url(Images/robust.svg) background-size: 700px background-position: 50% 20% # Robustness Checks

 ## Two main parts for robustness: .font130[ 1. Increase the number of robustness checks 2. Justify the appropriateness of a specific test ] --- background-image: url(Images/robust.svg), url("Images/robustness checks.svg") background-size: 300px, 500px background-position: 0% 100%, 100% 60% # Robustness **Robustness checks:** any possible change in a computational choice, both in data analysis and data cleaning -- .pull-left[ **Reasonable specifications** ([Simonsohn et. al., 2018](https://urisohn.com/sohn_files/wp/wordpress/wp-content/uploads/Paper-Specification-curve-2018-11-02.pdf)): 1. Sensible tests of the research question 2. Expected to be statistically valid, and 3. Not redundant with other specifications in the set. Reproducers will be able to record two types of contributions: - Mapping the universe of robustness checks - Proposing a specific robustness check ] --- background-image: url(Images/robust.svg) background-size: 300px background-position: 0% 100% # Robustness & Reproducibility .pull-left[ Robustness with level 1  Robustness with levels 5-9  ] .pull-left[ Robustness with levels 2-4  Robustness with level 10  ] --- # Promoting a Constructive Exchange ### For reproducers addressing original authors #### 1 - Contacting the original author(s) when there is no reproduction package #### 2 - Contacting the original author(s) to request specific missing items of a reproduction package #### 3 - Asking for additional guidance when some materials have been shared #### 4 - Response when the original author has refused to share due to *undisclosed reasons* #### 5 - Response when the original author has refused to share due to legal or ethical restrictions of the data #### 6 - Contacting the original author to share the results of your reproduction exercise ### For original authors responding to requests from reproducers #### 1 - Responding to a repeated request that has been addressed in an earlier interaction #### 2 - Acknowledging that the author no longer has access to certain part(s) of the reproduction package #### 3 - Acknowledging that some material is still embargoed for future research #### 4 - Responding to incomplete/unclear requests #### 5 - Responding to hostile responses from reproducers --- # .font80[Example 1: There is no reproduction package] .font80[ >**Subject:** Reproduction package for `[“Title of the paper”]` >Dear Dr. `[Lastname of Corresponding Author]`, > > I am contacting you to request a reproduction package for your paper titled `[Title]` which was published in `[Reference]`. A reproduction package may contain (raw and/or analytic) data, code, and other documentation that makes it possible to reproduce paper. Would you be able to share any of these items? > >I am a `[position]` at `[Institution]`, and I would like to reproduce the results, tables, and other figures using the reproduction materials mentioned above. I have chosen this paper because `[add context ...]`. Unfortunately, I was not able to locate any of these materials on the journal website, Dataverse `[or other data and code repositories]`, or in your website. > >I will record the result of my reproduction attempt on ACRE [...]. With your permission, I will also record the materials you share with me, which would allow access for other reproducers and avoid repeated requests directed to you. Please let me know if there are any legal or ethical restrictions that apply to all or parts of the reproduction materials so that I can take that into consideration during this exercise. > >In addition to your response above, would you be available to respond to future (non-repetitive) inquiries from me or other reproducers conducting an ACRE excercise? Though your cooperation with my and/or any future request would be extremely helpful, please note that you are *not required to comply*. >Since I am required to complete this project by `[date]`, I would appreciate your response by `[deadline]`. > >Let me know if you have any questions. Please also feel free to contact my supervisor/instructor `[Name (email)]` for further details on this exercise. Thank you in advance for your help! > >Best regards, >`[Reproducer]` ] --- count:false # .font80[Example 1: There is no reproduction package] .font80[ >**Subject:** Reproduction package for `[“Title of the paper”]` >Dear Dr. `[Lastname of Corresponding Author]`, > > I am contacting you to request a reproduction package for your paper titled `[Title]` which was published in `[Reference]`. A reproduction package may contain (raw and/or analytic) data, code, and other documentation that makes it possible to reproduce paper. Would you be able to share any of these items? > >I am a `[position]` at `[Institution]`, and I would like to reproduce the results, tables, and other figures using the reproduction materials mentioned above. I have chosen this paper because `[add context ...]`. **Unfortunately, I was not able to locate any of these materials on the journal website, Dataverse `[or other data and code repositories]`, or in your website**. > >**I will record the result of my reproduction attempt on ACRE [...]. With your permission, I will also record the materials you share with me, which would allow access for other reproducers and avoid repeated requests directed to you. Please let me know if there are any legal or ethical restrictions that apply to all or parts of the reproduction materials so that I can take that into consideration during this exercise.** > >In addition to your response above, would you be available to respond to future (non-repetitive) inquiries from me or other reproducers conducting an ACRE excercise? **Though your cooperation with my and/or any future request would be extremely helpful, please note that you are *not required to comply*.** >Since I am required to complete this project by `[date]`, I would appreciate your response by `[deadline]`. > >Let me know if you have any questions. Please also feel free to contact my supervisor/instructor `[Name (email)]` for further details on this exercise. Thank you in advance for your help! > >Best regards, >`[Reproducer]` ] --- # .font80[Example 1: Following up on additional materials] **Template email:** >**Subject:** Clarification for reproduction materials for `[“Title of the paper”]` >Dear Dr. `[Lastname of Corresponding Author]`, > >Thank you for sharing the materials. They have been immensely helpful for my work. > >Unfortunately, I ran into a few issues as I delved into the reproduction exercise, and I think your guidance would be helpful in resolving them. `[Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details]`. > >Thank you in advance for your help. > >Best regards, >`[Reproducer]` --- count:false # .font80[Example 1: Following up on additional materials] **Template email:** >**Subject:** Clarification for reproduction materials for `[“Title of the paper”]` >Dear Dr. `[Lastname of Corresponding Author]`, > >Thank you for sharing the materials. They have been immensely helpful for my work. > >Unfortunately, I ran into a few issues as I delved into the reproduction exercise, and I think your guidance would be helpful in resolving them. **`[Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details]`**. > >Thank you in advance for your help. > >Best regards, >`[Reproducer]` --- # An example of well described issues: .font80[ >Specifically, I am attempting to reproduce OUTPUT X (e.g., table 1, figure 3). I found that the following components are required to reproduce to reproduce OUTPUT X: ```md OUTPUT X └───[code] formatting_table1.R ├───output1_part1.txt | └───[code] output_table1.do | └───[data] analysis_data01.csv | └───[code] data_cleaning01.R* | └───[data] UNKNOWN └───output1_part2.txt └───[code] output_table2.do └───[data] analysis_data02.csv └───[code] data_cleaning02.R └───[data] admin_01raw.csv* ``` >I have marked with an asterisk (\*) the items that I could not find in the reproduction materials: data_cleaning01.R and admin_01raw.csv. After accessing these files, I will also be able to identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as UNKNOWN above) in the reproduction materials. > >I understand that this request will require some work for you or somebody in your research group, but I want to assure you that I will add these missing files to the reproduction package for your paper on the ACRE platform. Doing this will ensure that you will not be asked twice for the same missing file. ] --- count:false # An example of well described issues: .font80[ >Specifically, I am attempting to reproduce OUTPUT X (e.g., table 1, figure 3). I found that the following components are required to reproduce to reproduce OUTPUT X: ```md OUTPUT X └───[code] formatting_table1.R ├───output1_part1.txt | └───[code] output_table1.do | └───[data] analysis_data01.csv * | └───[code] data_cleaning01.R* * | └───[data] UNKNOWN └───output1_part2.txt └───[code] output_table2.do └───[data] analysis_data02.csv └───[code] data_cleaning02.R * └───[data] admin_01raw.csv* ``` >I have marked with an asterisk (\*) the items that I could not find in the reproduction materials: **data_cleaning01.R** and **admin_01raw.csv**. After accessing these files, I will also be able to identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as **UNKNOWN** above) in the reproduction materials. > >I understand that this request will require some work for you or somebody in your research group, but I want to assure you that I will add these missing files to the reproduction package for your paper on the ACRE platform. **Doing this will ensure that you will not be asked twice for the same missing file.** ] --- # Ok, I get it. But what is in for me? .font120[ - Standardized homework/project: everything is set up in terms of structure and deliverables. - Easy to grade (homework format). - Easy to guide and oversee (undergraduate dissertation format). - Easy to setup as an independent study. - Reduces duplication of requests to authors. - Facilitates a constructive exchange of ideas. - When emailing authors. - When discussion reproduction attempts. - Personal satisfaction that you’re contributing a public good to the profession! ] --- # .font80[Suggested timelines for different formats]

 	 2 weeks
 (~10 days)	 1 month
 (~20 days)	 1 semester
 (~100 days)

 	 analysis data	 raw data	 analysis data	 raw data	 analysis data	 raw data

 Scoping	 10% (1 day)	 5% (1 day)	 5% (5 days)

 Assessment	 35%	 25%	 15%

 Improvement	 25%	 0%	 40%	 20%	 30%

 Robustness	 25%	 5%	 25%	 25%

 --- # Easy to grade: report 1 <object data="Images/03_report_card_PS1.pdf" type="application/pdf" width="1000px" height="600px"> <embed src="Images/03_report_card_PS1.pdf">
This browser does not support PDFs. Please download the PDF to view it: Download PDF.
 </embed> </object> --- # Easy to grade: report 1 .pull-left[ <object data="Images/03_report_card_PS1.pdf" type="application/pdf" width="500px" height="600px"> <embed src="Images/03_report_card_PS1.pdf">
This browser does not support PDFs. Please download the PDF to view it: Download PDF.
 </embed> </object> ] .pull-left[ <object data="Images/Donaldson_RRRaj_AER.pdf" type="application/pdf" width="500px" height="600px"> <embed src="Images/Donaldson_RRRaj_AER.pdf">
This browser does not support PDFs. Please download the PDF to view it: Download PDF.
 </embed> </object> ] --- # .font80[Suggested timelines for different formats]

 	 2 weeks
 (~10 days)	 1 month
 (~20 days)	 1 semester
 (~100 days)

 	 analysis data	 raw data	 analysis data	 raw data	 analysis data	 raw data

 Scoping	 10% (1 day)	 5% (1 day)	 5% (5 days)

 Assessment	 35%	 25%	 15%

 Improvement	 25%	 0%	 40%	 20%	 30%

 Robustness	 25%	 5%	 25%	 25%

 --- background-image: url(Images/Homepage.png) background-size: contain name: platform # [socialsciencereproduction.org](https://www.socialsciencereproduction.org) --- # Ok, I Am Interested. What's Next? You can check the ACRE Guidelines, and also contribute if you want: https://bitss.github.io/ACRE/  .center[ A beta version of the platform will be available for the this semester for small scale testing .font120[ [**Sign up here**](https://forms.gle/yZivWcwijCzEhrBU6) ] if you would like to be part of our large-scale beta in the fall. ] --- count:false # Table of Contents 1. [Problems](#motivation) 2. [Solutions](#bitss) 4. [Accelerating Computational Reproducibility](#acre) 5. [**Open Policy Analysis**](#opa) --- background-image: url("Images/BITSSlogo.png"), url(Images/cega.png) background-size: 600px, 200px background-position: 50% 10% , 100% 95% name: opa # Open Policy Analysis (OPA) .pull-left[

 **Core BITSS Team** Aleks Bogdanoski Fernando Hoces de la Guardia Katie Hoeberling Teng Aleksandra Ma Edward Miguel **Undergraduate RAs** Sophia Bai Orianna Jia ] .pull-right[ .right[

 Keanu Lim Charles Love Emma Ng Ryan Scholes Jiayu Shi **Evidence Action** Grace Hollister Grace Morgan [BITSS](https://bitss.org) is part of the Center for Effective Global Action ([CEGA](https://cega.berkeley.edu/)). ]] --- count: true background-image: url("Images/alternative-facts-crowd.png"), url("Images/temp.png") background-size: 800px, 300px background-position: 0% 0%, 80% 10% # Motivation: Rise of Alternative Facts  .pull-right[

 *Senator [discussing facts]("https://www.youtube.com/embed/rV_YxKOE90g?start=188") on unemployment insurance* *Interviewer:* “We I have looked at what economist are saying, and [...] there is no measurable evidence that people are staying at home because of [$600 unemp. insurance]” *Congressman:* “[scoffs] I don’t know which economist you are talking about, but ...” ] --- count: true # Possible Mechanism: Opaqueness of Policy Analysis .font130[ - Incredible Certitudes (Manski, [2011](https://academic.oup.com/ej/article-abstract/121/554/F261/5079436)) - Report wars (Wesselink et al, [2013](https://www.sciencedirect.com/science/article/abs/pii/S1462901112002365)) - Low overall credibility of PA → credibility based on reputation ("serious") → assignment of reputation varies across political positions ] -- .font130[ - Parallels to "Reproducibility Crisis" and Open Science reponse (Hoces de la Guardia, Grant, Miguel [2020](https://academic.oup.com/spp/advance-article/doi/10.1093/scipol/scaa067/6018524?guestAccessKey=ac30d681-5391-4f91-b4d8-44c6aed11161)). Propose: - Core principles for Open Policy Analysis (OPA) in outputs, analysis and materials (reproducibility) - Agenda to implement and document how OPA can be applied into several policy issues - Growing adoption of open science methods into policy analysis ] --- count: true # Deworming Interventions - Parasitic worm infections are endemic in many countries, disproportionately affecting the poor - They interfere with regular bodily processes by decreasing nutrient uptake and can thus lead to serious consequences on human health, education outcomes, and long-term economic well being - Mass deworming interventions, at school level, have been propose as a cost-effective approach to tackle this problem -- ## Different settings for deworming - Context of original study (Kenya, 1998-99) had very high prevalence rates of worm infections - Implementation costs where very low ($0.42 per round of treatment) - Length of treatment was relatively short (2.4 years) - Current deworming settings have lower prevalnce rates, varying implementation costs and length of treatments -- ## Strong debate around initial results - Ozier ([2020](https://academic.oup.com/wbro/advance-article/doi/10.1093/wbro/lkaa005/6009048)) summarizes differences between original findings (Miguel and Kremer 2004) and a re-analysis (Aiken et al, 2015). Emphazises the role of communication of results in a reanalysis. - This type of debate (result/re-analysis) can be seen in several other topics. For example: minimum wage, inmigration, taxation. - OPA need not guarante agreement on key research finding, but should help avoid multiple policy reports --- background-image: url(Images/opa_framework.svg) background-size: 800px background-position: 60% 0% count:true # A Framework for Open Policy Analysis .pull-left[ Hoces de la Guardia, Grant, Miguel ([2020](https://academic.oup.com/spp/advance-article/doi/10.1093/scipol/scaa067/6018524?guestAccessKey=ac30d681-5391-4f91-b4d8-44c6aed11161)) ]

 ## Contributions of OPA to deworming: 1. Selected one policy estimate among several alternatives and establish a clear link between it and underlying assumptions 2. Added documentation to increase reproducibility 3. Created a public repository with all materials for one-click reproducibility --- background-image: url(Images/dw-open-out1.png), url(Images/dw-open-out2.png), url(Images/dw-open-out3.png), url(Images/open_output_all.svg) background-size: 500px, 500px, 500px, 100px background-position: 100% 80%, 70% 40%, 40% 0%, 15% 5% count: true # Open Output .pull-left[ ## [Demo](https://bitss-opa.shinyapps.io/dw-app/)
 ## Main features - One clear output previously agreed in consultation with policy partner - Two additional tabs to modify assumptions (key assumptions and all assumptions) - Each source is classified into research, data, or guesswork - High level equations added to illustrate location of components - Added feature to modify standard deviations - Track values of each component ] --- background-image: url(Images/dw-open-an2.png), url(Images/dw-open-an1.png), url(Images/open_analysis.svg) background-size: 400px, 400px, 100px background-position: 100% 100%, 70% 0%, 15% 5% count: true # Open Analysis .pull-left[ ## [Demo](https://bitss-opa.github.io/opa-deworming/)
 ## Main features - Complete narrative description of the methodology - Translation of each narrative step into an equation - Implementation of each equation into code - Combine all of the above into using a dynamic document (RMarkdown) - Presentation of narrative, equations, and code in layered fashion to avoid overwhelming the reader Icon figure ] --- background-image: url(Images/dw-open-mat2.png), url(Images/dw-open-mat1.png), url(Images/open_materials.svg) background-size: 600px, 600px, 100px background-position: 100% 100%, 70% 0%, 15% 5% count: true # Open Materials .pull-left[ ## [Demo](https://github.com/BITSS-OPA/opa-deworming)
 ## Main features - One-click reproducible documentation and app - Extensive readme files - Clear folder structure - Version controlled - Open data - Acknowledgment to all contributors ] --- background-image: url(Images/dw-nopen-out2.png), url(Images/dw-nopen-out3.png) background-size: 800px, 800px background-position: 90% 00%, 90% 100% count: true # What happens if one principle fails: Opaque Output .pull-left[ - Policy estimates of approach 2 of deworming (Baird et al, 2016) - Assume Open Analysis and Open Materials - Good for researchers - Unclear for policy analysts - Allows policy makers to cherry pick results - Does not disclose uncertainty - Unclear separation of roles between policy analyst and policy maker (Truman's request for a "one-handed" economist) ] --- count: true # Opaque Analysis/Materials
 .font140[ - Assume open output and open materials - In this scenario we can have two policy analyses: one from advocates and one from opponents - Each analysis can claim to be open source. Code is available, reproducible, and each as an interactive app. - But each analysis can bury in their code analytical choices that move the final policy estimate in their favor - Same though exercise can be done with opaque materials ] --- count: true # Lessons for future OPA projects: Timeline .font110[ 1. Reviewed existing documentation 2. Review code/spreadsheets and write down missing documentation. 3. Add equations to reflect any additional clarifying steps. 4. Translate analysis to code scripts (ideally open source software) 5. Check for computational reproducibility and consult with original analysts for discrepancies/further questions 6. Discuss with policy partner on target policy estimate 7. Draft doodle of app 8. Incorporate any potential new analysis (repeat steps 1 - 3) 9. Present draft of OPA (DD and doodle of app) to policy partner 10. Incorporate comments into documentation of the OPA 11. Build app based on analysis of dynamic document 12. Incorporate any additional features requested by policy partner 13. Verify that all three components produce the same output 14. Publish v1.0 ] --- count: true # Useful practices developed during the project
 .font120[ ## Documentation of analysis and app are connected - Each analytic step is declared in the documentation and wrapped in a function - All this steps are used to produce the results in the documentation - The steps are also stored in a file that is later sourced in the app. - If something changes in the report, it will update the app. ## Style guide - Group objects according to use: only within the documentation, for simulations, and for the app - Use diagram trees to depict nested relationships ] --- count: true # Lessons for future OPA projects: Costs - Costs (approx bandwidth over a year at full time): - Principal Investigator 30-50% - Research assistant/programmer 100-150% - Program Manager 20%-30% - Original researcher: 1-5% - Not all policy analysis justify this level of effort - Characteristics that might justify an OPA: - Topics with strong disagreement on the facts among analysts - Recurrent reports (eg. ex-ante economic analysis from development banks/agencies) - Topics that have large expected welfare effects (eg. tax reform, social cost of carbon) - With each new OPA project, templates will emerge and costs will likely fall --- count: true # Additional Benefits of OPA ## Easy to update and reuse After deworming OPA is released, anybody can modify and improve into a newer version ## Clearer connection of how evidence from research is used in policy analysis Researchers can see clearly where their estimates are being used in a policy analysis. For example, the OPA can be used to justify power calculations of potential new studies. ## Connection with forecasting When there is little information for a parameter used in an OPA, a forecasting exercise can be carried out to elicit expert knowledge (DellaVigna, Pope, Vivalt [2019](https://science.sciencemag.org/content/366/6464/428.summary)).


Home
Manage Drivers
Select Driver:
Driver License
Auto Insurance
Liability Insurance
Certification
Update Driver information
Update Name:
Update email address:
Update availability:
Submit


We allow ourselves to clutter our lives with our immediate to-dos and goals, often neglecting to remember the moments of happiness that we can be grateful for. Keeping track of these moments can help us appreciate them, grow in self-awareness and be happier. Everyone being happier makes the world a happier place.
Home page for Clarity app.
Profile picture and current happiness level of user. A higher level indicates more happiness.
The "Log Activity" screen allows a user to add a recent activity.
Drop down menus help the user choose an activity and rate how it changed the user's happiness level.
The "Recommendations" screen keeps track of the activities that made the user happy, and uses the user's happiness level to find similar users and make recommendations using the activities that made the other users happy.
A connection to social media allows the user to filter friends that the recommender system uses in order to make suggestions.
Welcome page to Clarity app.
Menu page.
"Share a Story" leads to a page of 4 lenses.
A pop up appears when a user's finger clicks over "journal", as quick-display.
Update screen helps the user self-log heuristics of well-being and goals.
A wizard can "scroll" up and down.
"Activity" takes us to boxes of categories and actions.

"Log Your Feelings" brings user to a gradated mood wheel.


HTML Basics
Lists
Ordered list
- first item
- second item
- third item
- forth item
- fifth item
Unordered list
Description List
- Name
- Godzilla
- Born
- 1952
- Birthplace
- Japan
- Color
- Green
Nested list
- Lorem Ipsum is simply dummy text
- Lorem Ipsum is simply dummy text
- Lorem Ipsum is simply dummy text
- Lorem Ipsum is simply dummy text
- Lorem Ipsum is simply dummy text
- Lorem Ipsum is simply dummy text
- Lorem Ipsum is simply dummy text
List with links
Paragraphs
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the
industry's
standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to
make a type specimen book. It has survived not only five centuries, but also the leap into electronic
t
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has
survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop
publishing software like Aldus PageMaker including versions of Lorem Ipsum.
Tables
| Ячейка номер один..... |
Ячейка 2 |
| Ячейка 4 |
Ячейка 5 |
| Ячейка 4 |
Ячейка 5 |
Image
Pre Tag
if (time==20) {
greeting="Добрый день ";
}
else {
greeting="Добрый вечер ";
}


Power to the Pinkies
By Casey Gum
As you may have gathered from my last post, I am a big fan of vi/vim. However, one thing that has always annoyed me about vi is that to get back into normal mode, you have to hit the escape key. It is probably one of the most difficult keys to hit on the whole keyboard and many laptop keyboard layouts have made it about the size of peanut.
People who have been doing command-line linux for a long time don’t have a problem with this since environmental adaptation has elongated their left pinkies [citation needed]. People who use Windows and OSX, on the other hand, need help.
So, the natural thing to do is to commandeer the enormous and conveniently located Caps Lock key and make it act like the escape key. (Note: If you need to type OMG LOL ROFL, you can just hold down the shift key and type – you don’t actually need to use Caps Lock – just saying…)
Here’s how you do it…
In Windows:
Note: If you are running Windows in Parallels on a Mac, this won’t work – go to the section on how to do it in OSX.
Take the block of text below and copy it into notepad. Then, save the file as EscapeKeyRemap.reg and double-click it to run it. (You can also Start > Run > regedit and then choose File > Import and import the file.) As you can easily see, it sets the “Scancode Map” key to a bunch of random numbers. This causes the Caps Lock key to behave like an escape key, which is all that really matters.
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
“Scancode Map”=hex:00,00,00,00,00,00,00,00,02,00,00,00,01,00,3a,00,00,00,00,00
EDIT: The previous registry key entry was wrong - I fixed it.
After you have modified the registry, you will need to reboot the system. Once you are done rebooting, you will be good to go.
In OSX:
In OSX, you will need to install PCKeyboardHack. You can download it from here: http://download.cnet.com/PCKeyboardHack/3000-2110_4-75363152.html.
After you download and run the disk image (.dmg file) you will see an icon on your desktop called PCKeyboardHack-7.4.0. Double-click it and it will open a new window with a file in it called PCKeyboardHack.reg. Double-click that to install it using all the default options. After it is done installing, you will need to reboot.
After rebooting, go to your system preferences and double-click the PCKeyboardHack icon.
This will bring up a dialog that allows you to remap your Caps Lock key. Check the Change Caps Lock box and enter the keycode 53.
Close the dialog and you’ll be good to go. This will carry over to Windows running under Parallels as well.
Happy Vimming!


{% extends 'lab-side-content.html' %} {% set AppParentId = "getting-started" %} {% set AppCurrentId = "introduction" %} {% block content %}
An open source toolkit for developing with HTML, CSS and JS.
Requires at least version 6 of Node. We reccomend using nvm to install and manage your Node versions.


by
Robin Johnson
(formerly Deputy-Director, Economics Division, Ministry of Agriculture)
The Institute of Policy Studies has performed a public service by publishing some of the papers on public administration that the late Peter Gorringe wrote while an official in the New Zealand Treasury2. The papers show a breadth and scope that is unusual for a public servant. They also provide a useful commentary on the public policy issues that engaged policy makers in the late 1980s and early 1990s. The book adds to the earlier book on the economic reforms by Boston, Martin, Pallot and Walsh3.
As is well known the author made considerable use of the new institutional economics in his diagnosis of New Zealand's economic problems of the period. The papers are full of references to Coase, Buchanan, North and Williamson. This literature makes common use of the particular insights of Ronald Coase with regard to transaction costs and property rights and the constitutional and institutional implications that follow when transaction costs are non-zero.
It appears to me that Gorringe applies the transaction cost model (to use this phrase as an overall descriptor) to the policy issues of the day rather than the policy process. This involves taking the constitutional structure as a given and recommending policy change within this paradigm. For example, if one takes his discussion of the State Sector Act 1988 and the Public Finance Act 1989 (pp 161-5), the discussion is limited to the devolution of the management of government departments to chief executives, and away from central agencies; and the need for a system of accountability for the goods and services core government agencies produce. While there is considerable discussion of the agency problem in this and other contexts, there is no reference (in the published papers) to the transaction costs of the government policy process itself.
In previous times this would have been discussed under the rubric of dead-weight losses in a welfare framework. More recent work by Dixit and Horn has brought policy implementation into an institutional framework4. Dixit asks what organisational forms will overcome the monopoly of information held by bureaucrats, the opportunistic behaviour of civil servants, the problem of divided principals, and government-owned asset specificity? Horn, independently of Dixit, bases his analysis on bounded rationality, opportunism, agency theory and transaction costs in a parliamentary system. He analyses the transaction costs involved in the political and administrative process including the cost of achieving durability or lasting impact (the latter turns out to be a coordination problem).
I believe these frameworks complement the work done by Gorringe (they appear to have been developed after most of his work was complete). In this article I set out what the transaction cost model can offer in this area and suggest some useful applications in the New Zealand case. I first discuss the interpretation of institutional economics made by Dixit and Horn and then follow with a discussion of some of the implications for public policy analysis.
The role of institutions
The transaction cost model of the political decision process sees politicians and interest groups resolving their differences by adopting policy measures that suit both parties. Politicians favour policies which extend the life of the party[ies] in power while interest groups favour policies which deliver benefits to them now and in the future. Interest groups are powerful precisely because they can influence voting patterns and party finances. Major transaction costs arise from the difficulties and uncertainties of maintaining control of the process over time.
Transaction cost economics derives from the work of Ronald Coase who first posited that when it is costless to transact, the efficient competitive solution of neoclassical economics obtains5.
It does so because the competitive structure of efficient markets leads the parties to arrive costlessly at the solution which maximises aggregate income regardless of the institutional arrangements. But with information deficiencies and imperfections in input and product markets further transaction costs are incurred by the firm in the conduct of its business. It is no longer costless to transact and the institutional arrangements have a bearing on the outcome as the base line is no longer the neoclassical solution.
Proponents of the institutional approach therefore focus on alternative governance structures in firms and ask what are the net gains after costs of information gathering and maintaining contracts have been taken into account6. The focus is on organisational alternatives rather than the marginal conditions. The marginal conditions are seen as some ideal benchmark which can never be achieved in the real world.
This approach is the basis of Oliver Williamson's analysis of hierarchical organisations and the economics of vertical and horizontal integration7. It stresses the role of a number of transaction costs that prevent the pure model of competition working in an imperfect world. In particular, costs of maintaining and policing contractual obligations and asset specificity are the determinants of the final outcome.
Can this reasoning be applied to government policy-making and organisation? North8, Horn and Dixit are among those who have applied transaction cost economics to political decision making in the realm of economic policy. There is a new recognition of the role of governance structures in policy making institutions. In turn, these constructs enable policy formation and implementation to be identified separately from the political process of which they are part.
Dixit's approach
Dixit develops an organisational approach following the writings of
Buchanan9,
North10,
and Williamson11.
In terms of the political process, the standard normative approach
to policy analysis views this whole process as a
social-welfare-maximising black box, exactly as the neoclassical
theory of production and supply viewed the firm as a
profit-maximising black box.....Economists studying business and
industrial organisation have long recognised the inadequacy of the
neoclassical view of the firm and developed richer paradigms and
models based on the concepts of various transaction costs
(op cit
p.9). Dixit therefore suggests that a better model of the policy
forming process should be based on an analysis of the relevant
transaction costs.
Dixit argues for the inclusion of the political parameters in the analysis
of the policy process. As a crude but effective caricature, one can
say that normative policy analysis began by supposing that the policy
was made by an omnipotent [having infinite power], omniscient
[infinite knowledge], and benevolent dictator. The work on second
best removed the omnipotence. That on information removed the
omniscience. However, the assumptions of benevolence and dictatorship
have remained unaffected by all these improvements in our
understanding of the limits on instruments and information....In
reality, a policy proposal is merely the beginning of a process that
is political at every stage - not merely the process of
legislation, but also the implementation, including the choice or
formation of an administrative agency and the subsequent operation of
this agency
(op cit pp 8-9) (italics in original).
Dixit asks what organisational forms will overcome the monopoly of information held by bureaucrats, the opportunistic behaviour of civil servants, the problem of divided principals, and government-owned asset specificity? He identifies transaction costs involved in overcoming the asymmetric distribution of information between parties (signaling and screening costs, costs of monitoring and incentives, auditing costs and costs of misrepresentation), costs involved in managing agents (monitoring, incentives, and contractual obligations), costs of agents responding to multiple principals (coordination of policies, playing off one principal against another), and costs related to asset specificity (irreversible investments and lack of durability).
Dixit takes the view that the election process creates a contract between the politicians (individuals or parties) or administrators (regulatory agencies etc), and the citizens (individuals or interest group organisations). The contract is a promise of a policy (or programme) in return for votes (or contributions). Unlike private contracts, such policy contracts are difficult to enforce12.
First, political contracts are rarely between two clearly identifiable contractors; they have multiple parties (voters or lobbyists) on at least one side of the relationship. Second, their terms are generally much more vague (unspoken) than those of economic contracts. They leave much room for interpretation, and many loopholes for escape and opportunities to blame third parties or force majeure for failure to deliver. Thus a promise to cut taxes rarely specifies by how much, and can be rescinded when the budget situation turns out to be much worse than expected. In the electoral contract, the citizens have granted the elected representatives certain powers to collect taxes and make decisions on their behalf, the only safeguards in this process remains the free vote and, if this goes, abuse of power is likely to follow.
Horn's approach
Horn puts more emphasis on the excessive cost of coordination in the political process. He spells out some of the details of the exchange politicians and citizens enter into13. His interpretation is also driven by bounded rationality, opportunism, agency theory and transaction costs. He introduces some of the specifics of the political process and the accompanying transaction costs, including the cost of durability or lasting impacts. This is a wider definition of transaction costs than described by Dixit above.
Horn's transactions approach examines the relationship among three sets of
actors with different roles and motivations. Legislators
are the
elected representatives who perform both legislative and executive
functions. Legislation can only be enacted by an enacting coalition
of individual legislators - that is, a group of legislators that is
large enough to guarantee the passage of a bill into law.
Constitutional differences between countries affect such groupings
and the solutions they might seek. Administrators
are appointed
rather than elected. They answer to the legislature. Horn assumes
that they do not bring policy preferences to their work!
Constituents
conveys the idea of particular groups in society that
have a [vested] interest in the policy making process. Constituents
enjoy the benefits - or suffer the costs - of legislation, offer
support or opposition to legislators, and, ultimately, elect
legislators to office. While these stereotypes tend to simplify the
complexity of the political/administrative process in any one
country, Horn maintains that they are sufficiently general to explain
decisions legislators make about the organisation of public
administration on a broad front.
The approach focuses on the difficulties political decision makers have
in securing continued electoral support on a longer term basis.
Legislators are regarded as self-seeking in their use of legislation
to increase their net political support and lasting power. Their
opportunities are limited by the transaction costs of
achieving agreement on their proposals. These costs are the time and
effort it takes to reach agreement on legislative refinements and any
time and effort that affected private interests subsequently have to
devote to participating in implementation and administration;
political uncertainty that the legislation will last; uncertainty
that the legislation will be administered as intended; and
uncertainty about the distribution of private benefits and costs. If
such relationships are regarded as the hazards of the exchange
between the respective actors, then they are generically similar to
Williamson's far-sighted but incomplete contracting
.
The elected/political appointees who are most likely to remain in power are those who are most successful in overcoming these transaction problems in the political process, such as those who are best able to reassure their supporters that the benefits of legislation will not be lost to administrators in implementation, or undone by subsequent legislatures. Successful implementation will depend on administrative agents who do not necessarily share the objectives of the enacting coalition and its constituents; these divergences create transaction costs to do with monitoring the performance of such agents and devising a system of rewards and sanctions that improve performance. There is no reference to asset specificity in this version of the model though government administrations are likely to depend on labour specificity in the administrative staff. Williamson14 notes that asset specificity is negligible in most public sector departments but human assets in many public bureaucracies involve considerable specificity (for example, nontransferable training and social conditioning).
Furthermore, the amount of net electoral support legislators receive from promoting a piece of legislation depends on the net flow of costs and benefits that private interests expect it to generate over time. Implementation bears on this calculus because private interests are sufficiently forward looking to anticipate how decisions on implementation will affect the flow of benefits and costs. Political support will cease unless they are well pleased (over time).
Horn's model supplements Dixit's model and brings back the broader concepts of public choice into main focus. The uncertainties identified by Dixit (information asymmetry, monitoring of agents and opportunism) create costs in the political process that have to be overcome if lasting control of the Treasury benches is to be achieved. The model posits that the endeavours of the legislators/politicians will continuously be directed to their own survival. This will be achieved by paying greater attention to the wishes of their supporters in a longer term framework. One term of office is never sufficient.
Now in a certain sense, these endeavours can be regarded as an optimal arrangement of the nation's affairs. The model posits that effective public administration requires that the transaction costs be minimised in determining and pursuing society's goals. There will be pressure from constituents to find least cost solutions to the problems of the day. There will be limits on the revenue and challenges from alternative expenditure items including government programmes. Debate and bargaining will tend toward least cost solutions.
The insight from the transaction cost model of the political market is
that it mimicks how policy decisions are reached in government
polities rather better than previous constructions. The black box
is opened out. Outcomes are based on the balancing of interests and
the respective power bases of the participants. The balancing of
interests extends to policy implementation and performance.
Influential participants have a vested interest in who is to
implement a policy and on what terms and will influence the political
decision accordingly.
The model recognises that power is unequally distributed and that silent majorities tend to be forgotten. The balancing process assumes the interested participants can anticipate future benefits and costs that affect their welfare and that they will act in this light. The model assumes that self-interest is dominant to the national interest in most circumstances.
The model does not require any deep analysis of political motives but rather sets out a useful framework that explains the process in which economic policy is conceived and executed. It clearly posits that decision making has to be a political process subject to electoral demands and interest group pressure on the one hand, and national interest concerns contained in rational advice from civil servants (who may or may not be disinterested) on the other.
It can be utilised to examine the structure of government bureaucratic organisations. Williamson offers a detailed analysis of the US Department of State15. He asks: could the organisation of foreign affairs by the Department of State be replaced by a private bureau? Not surprisingly, perhaps, he comes to the conclusion that it cannot and that there are specific tasks for which a government bureau is more suited. These tasks include situations where hazards of probity (lack of honesty) are to be avoided and organisational matters such as dependence on labour asset specificity (social conditioning and reliance).
The model indicates that blackboard exercises involving producer and consumer surpluses are of limited use in policy making. The presence of transaction costs will suggest different solutions to a policy problem depending where they fall. The case for government intervention in the market failure case is not clear-cut if the costs of intervention outweigh the possible benefits. There may be little or no relationship between the costs and benefits estimated by the outside observer and the evaluations that individuals place on alternatives in actual choice situations16.
Another application is to ask whether the policy advice function in a government bureau could be contracted out? The Canadian department of agriculture, Agriculture Canada, explored this option and sponsored a study of a similar proposal in New Zealand17.
While the study recognised that alternative sources of advice should be sought, for informational reasons, it came to the conclusion that confidentiality and probity concerns made the function more suitable for a public bureau.
More widely, the international flavour of the analysis overcomes the dominance of US writing on the subject dependent as it is on interpretations of the US constitution18>. Horn, in particular, while developing these ideas at Harvard, has incorporated the Westminster tradition into his version of the model and broadened its application to include state-owned profit-making organisations as well as government-financed bureaus.
While there is an emphasis on democratic governmental decision making, particularly with regard to maintaining electoral support, the focus on institutional economics broadens the application to non-democratic regimes and offers a fruitful avenue of further research. It also provides an entry point to the analysis of malfeasance and corruption in government administrations19.
Another application is in the distribution and administration of foreign aid. Many IMF and World Bank programmes are focused on particular national policy programmes or projects which depend on local initiatives and execution. The strengths and weaknesses of national administrations very soon come to the fore when visiting economists seek answers to poor response rates to massive aid packages. Institutional economics offers useful guidance to those seeking reform in E. Europe20.
I have argued elsewhere for a more realistic policy-making framework as a guide to better policy making21. National and international advisors in ministries would benefit from a better understanding of the institutional background. University training could benefit from the teaching of more institutional economics and its practical applications. More university people could with advantage work in the civil service. General economists could re-assess their welfare models and voluminous writings in the professional literature and pay greater regard to the conditions under which their policy recommendations will have to operate.
3572 words
1. Revised version of 'Murray Horn Revisited: Transaction Cost Models of Public Administration and Developing Country Experience' a paper presented to the NZ Association of Economists Conference, Rotorua, 1999.
2. Grimes, A., Jones, A., Procter, R., and Scobie, G. (2001), Economics for Policy: Expanding the Boundaries, Essays by Peter Gorringe, Institute of Policy Studies, Wellington.
3. Boston, J., Martin, J., Pallot, J., and Walsh, P. (1996), Public Management: the New Zealand Model, Oxford University Press.
4. Dixit, A. K. (1996), The Making of Economic Policy: A Transaction-Cost Politics Perspective, MIT Press, Cambridge, Massachusetts; Horn, M. (1995), The Political Economy of Public Administration, New York, Cambridge University Press.
5. Coase, R.H. (1960), The Problem of Social Cost, Journal of Law and Economics 3, 1-44.
6. This follows North, D.C. (1990a), Institutions, Institutional Change and Economic Performance, Cambridge.
7. See Williamson, O. E. (2001), Public Policy Analysis: Economics and Organisation Theory, IPS Policy Newsletter No 65, 7-24.
8. North, D.C. (1990b), A Transaction Cost Theory of Politics, Journal of Theoretical Politics 2, 355-67.
9. Buchanan, J.M. (1989), Explorations into Constitutional Economics, Tollison R.D. and Vanberg V.J. (eds), College Station, Texas A & M University Press.
10. According to Williamson, O.E. (1999), Public and Private Bureaucracies: A Transaction Cost Economics Perspective, Journal of Law, Economics, and Organisation 15, p.309, North suggests that the efficiency of politics is to be judged by examining how closely an actual political market approximates a zero transaction cost result.
11. Williamson (1999) does not offer an analysis, to my knowledge, of the economic-political system. He appears to be diverted by the possibly more interesting question, to him, of whether private bureaucracies replicate the attributes of public bureaucracies: `The discrete structural atributes that define and distinguish the public bureau and are responsible for its powers and limitations thus need to be identified and explicated' (Williamson, op cit, p.307).
12. Williamson (op cit, p.310) summarises the Dixit viewpoint; [he} works out of an incomplete contracting setup in which bounded rationality and opportunism are featured and holds that the object
is to understand how the combined economic-political system evolves
mechanisms to cope with the variey of transaction costs that it must
face
(Dixit 1996, p.xv).
13. Horn does not specifically refer to the relationship as a contract, nor to incomplete contracts. He states (1995, p.13) that legislators and their constituencies engage in exchange. Legislators want electoral support and constituents want the private benefits - or to reduce the private costs - of legislation. Failure to deliver is measured at the voting booth and through other means of political support.
14. Williamson op cit, p. 322,339.
15. See Williamson, op cit.
16. A case where such costs exceed benefits [soil erosion] is discussed by Pasour E.C. (1993), Economics and the Public Policy Process, New Zealand Economic Papers 27, 1-17.
17. See Storey, G.G. (1996), Investigation on the Implications of Government Reform in New Zealand for Obtaining Economic and Policy Analysis for the Agri-food Sector, Department of Agricultural Economics, University of Saskatchewan, for Agriculture Canada.
18. For a comparison of Westminster with the US government system, see Weaver, R.K. and Rockman, B.A. (1993), Do Institutions Matter? Government Capabilities in the United States and Abroad, The Brookings Institution, Washington.
19. See Johnson, R.W.M. (1999), The Role of Political and Economic Institutions in Rural Strategy Formulation and Implementation.
20. See Williamson, O.E. (1995), The Institutions and Governance of Economic Development and Reform, Proceedings of the World Bank Annual Conference on Development Economics 1994, Washington, pp 171-197.
21. See Johnson, R.W.M. (1994), The National Interest, Westminster and Public Choice, Australian Journal of Agricultural Economics 38, 1-30.


API Documentation
qx.Website
Error while loading!
Has the application been generated? (generate.py api)
You want a full text search? Use the browser search instead (
Ctrl
+ F).
Got it!


S3GPIO lets you control lights, motors, sensors, switches and HATs connected to the GPIO pins on a Raspberry Pi from Scratch3 via ScratchGPIO.
Enable S3GPIO
Host:
Reset to Default
Chrome extension by
David Ferguson


All pictures taken from recent Fine Decor projects.
2017 Fine Decor International Inc. Site by Dylan Tinianov


SofTail Deluxe.
The Deluxe 2018 comes with loads of new features such as the retro-modern LED lights, 107 v-twin engine and many more cool new features.
The 107 v-Twin Engine delivers power smoothness and a crisp throttle along with the satisfying rumble of a Harley Davidson.
The pull-back handlebars offer easy reach for the rider and has a chrome pullback riser that gets the grips even closer and gives the front end a aerodynamic look.
The chrome laced wheels with whitewall tires gives the bike the retro look it needs and deserves sending the bike back to the '50s.
The retro-modern style of LED lights allow the rider to go on night rides and be able to see and be seen with LED headlights, running lights, turn signals and brake lights.
The new Softail frame offers a lighter, stiffer frame compared to the older Softail and Dyna Deluxes.
The new high perfoprmance front suspension provides the performance of a racing style cartridge fork but with less weight.
The all new Mono rear shock suspension allows for dymanic cornering capability while keeping the classic look of a hardtail.
The 2018 Deluxe costs an excellent £14272 but remember the price may vary depending on where you live or your payment method.


Starring: a, b, c
{ “Title”: “Inception”, “Year”: “2010”, “Rated”: “PG-13”, “Released”: “16 Jul 2010”, “Runtime”: “148 min”, “Genre”: “Action, Mystery, Sci-Fi”, “Director”: “Christopher Nolan”, “Writer”: “Christopher Nolan”, “Actors”: “Leonardo DiCaprio, Joseph Gordon-Levitt, Ellen Page, Tom Hardy”, “Plot”: “Dom Cobb is a skilled thief, the absolute best in the dangerous art of extraction, stealing valuable secrets from deep within the subconscious during the dream state, when the mind is at its most vulnerable. Cobb’s rare ability has made him a coveted player in this treacherous new world of corporate espionage, but it has also made him an international fugitive and cost him everything he has ever loved. Now Cobb is being offered a chance at redemption. One last job could give him his life back but only if he can accomplish the impossible - inception. Instead of the perfect heist, Cobb and his team of specialists have to pull off the reverse: their task is not to steal an idea but to plant one. If they succeed, it could be the perfect crime. But no amount of careful planning or expertise can prepare the team for the dangerous enemy that seems to predict their every move. An enemy that only Cobb could have seen coming.“, “Language”: “English, French, Japanese”, “Plot”: “A thief, who steals corporate secrets through use of dream-sharing technology, is given the inverse task of planting an idea into the mind of a CEO.”, “Country”: “USA, UK”, “Awards”: “Won 4 Oscars. Another 139 wins & 192 nominations.“, “Poster”: “http://ia.media-imdb.com/images/M/MV5BMjAxMzY3NjcxNF5BMl5BanBnXkFtZTcwNTI5OTM0Mw@@._V1_SX300.jpg", “Metascore”: “74”, “imdbRating”: “8.8”, “imdbVotes”: “1,434,451”, “imdbID”: “tt1375666”, “Type”: “movie”, “tomatoMeter”: “86”, “tomatoImage”: “certified”, “tomatoRating”: “8.1”, “tomatoReviews”: “326”, “tomatoFresh”: “282”, “tomatoRotten”: “44”, “tomatoConsensus”: “Smart, innovative, and thrilling, Inception is that rare summer blockbuster that succeeds viscerally as well as intellectually.”, “tomatoUserMeter”: “91”, “tomatoUserRating”: “4.2”, “tomatoUserReviews”: “564261”, “tomatoURL”: “http://www.rottentomatoes.com/m/inception/", “DVD”: “07 Dec 2010”, “BoxOffice”: “$292.6M”, “Production”: “Warner Bros. Pictures”, “Website”: “http://inceptionmovie.warnerbros.com/", “Response”: “True” }


Hot-keys on this page
n s m x c change column sorting
| Module | statements | missing | excluded | coverage |
|---|---|---|---|---|
| Total | 233 | 4 | 0 | 98% |
| apps\core\admin.py | 1 | 0 | 0 | 100% |
| apps\core\models.py | 1 | 0 | 0 | 100% |
| apps\core\views.py | 5 | 0 | 0 | 100% |
| apps\favorites\admin.py | 1 | 0 | 0 | 100% |
| apps\favorites\models.py | 15 | 0 | 0 | 100% |
| apps\favorites\views.py | 21 | 0 | 0 | 100% |
| apps\off\admin.py | 9 | 0 | 0 | 100% |
| apps\off\management\commands\off_db.py | 62 | 1 | 0 | 98% |
| apps\off\models.py | 33 | 1 | 0 | 97% |
| apps\results\admin.py | 1 | 0 | 0 | 100% |
| apps\results\models.py | 15 | 1 | 0 | 93% |
| apps\results\views.py | 11 | 0 | 0 | 100% |
| apps\user\admin.py | 1 | 0 | 0 | 100% |
| apps\user\forms.py | 16 | 0 | 0 | 100% |
| apps\user\models.py | 1 | 0 | 0 | 100% |
| apps\user\views.py | 40 | 1 | 0 | 98% |
No items found using the specified filter.


@(errorId: String, throwable : Throwable) @stackTrace(t:Throwable) = @{ val sw = new java.io.StringWriter() val pw = new java.io.PrintWriter(sw) t.printStackTrace(pw) Html(sw.toString()) }
Something has caused the application to fail. Please try again.
Error id: @errorId
@throwable.getMessage
@if( play.api.Play.isDev(play.api.Play.current) ) {@stackTrace(throwable)}