class: center, middle, inverse, title-slide # Lecture 10 ## R and the tidyverse // randomization ### Ivan Rudik ### AEM 6510 --- exclude: true ```r if (!require("pacman")) install.packages("pacman") pacman::p_load( tidyverse, xaringanExtra, rlang, patchwork, nycflights13 ) options(htmltools.dir.version = FALSE) knitr::opts_hooks$set(fig.callout = function(options) { if (options$fig.callout) { options$echo <- FALSE } knitr::opts_chunk$set(echo = TRUE, fig.align="center") options }) ``` ``` ## Warning: 'xaringanExtra::style_panelset' is deprecated. ## Use 'style_panelset_tabs' instead. ## See help("Deprecated") ``` ``` ## Warning in style_panelset_tabs(...): The arguments to `syle_panelset()` changed in xaringanExtra 0.1.0. Please refer to the documentation to update your slides. ``` --- # Roadmap - What is R? - What is the tidyverse? - How do we import and manipulate data? Our goal is to take a hands on approach to learning how we do environmental economics research A good chunk of this lecture comes from Grant Mcdermott's [data science for economists](https://github.com/uo-ec607/lectures) notes, and [RStudio education](https://education.rstudio.com/) --- class: inverse, center, middle name: r # RStudio Cloud

 --- # Getting started We will be using [rstudio.cloud](https://rstudio.cloud) for our coding -- Why? -- You don't need to download/install anything -- I can prepare the packages and code and make it easy to download -- Let's get everything going... --- # Getting started: login ![](files/login.png) --- # Getting started: new project from github ![](files/project.png) --- # Getting started: new project from github ![](files/github_url.png) --- # Getting started: wait for deployment ![](files/deploying.png) --- # Click on class-code in bottom-right ![](files/workspace.png) --- # Click on class-code.Rproj ![](files/r_project.png) --- # Click yes  --- # Check package status (not required)  --- # Install packages (then press 'y')  --- class: inverse, center, middle name: r # Quick intro to R

 --- # Arithmetic operations R can do all the standard arithmetic operations ```r 1+2 ## add ``` ``` ## [1] 3 ``` ```r 6-7 ## subtract ``` ``` ## [1] -1 ``` ```r 5/2 ## divide ``` ``` ## [1] 2.5 ``` --- # Logical operations You also have logical operations ```r 1 > 2 ``` ``` ## [1] FALSE ``` ```r (1 > 2) | (1 > 0.5) # | is the or operator ``` ``` ## [1] TRUE ``` ```r (1 > 2) & (1 > 0.5) # & is the and operator ``` ``` ## [1] FALSE ``` --- # Logical operations We can negate expressions with: `!` This is helpful for filtering data ```r is.na(1:10) ``` ``` ## [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE ``` ```r !is.na(1:10) ``` ``` ## [1] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE ``` --- # Logical operators For value matching we use: `%in%` To see whether an object is contained within (i.e. matches one of) a list of items, use `%in%`. ```r 4 %in% 1:10 ``` ``` ## [1] TRUE ``` ```r 4 %in% 5:10 ``` ``` ## [1] FALSE ``` This is kind of like an `any` command in other languages --- # Logical operators Turns out there's no %notin%, but we can make one using `Negate` -- ```r `%notin%` <- Negate(`%in%`) ## The backticks (`) help to specify functions. 4 %notin% 5:10 ``` ``` ## [1] TRUE ``` --- # Logical operators To evaluate whether two expressions are equal, we need to use .hi-red[two] equal signs ```r 1 = 1 ## This doesn't work ``` ``` ## Error in 1 = 1: invalid (do_set) left-hand side to assignment ``` ```r 1 == 1 ## This does. ``` ``` ## [1] TRUE ``` ```r 1 != 2 ## Note the single equal sign when combined with a negation. ``` ``` ## [1] TRUE ``` --- # Assignment In R, we can use either `=` or `<-` to handle assignment.1 .footnote[ 1 The `<-` is really a `<` followed by a `-`. It just looks like an arrow because of the font on the slides. ] -- `<-` is normally read aloud as "gets" You can think of it as a (left-facing) arrow saying .hi[assign in this direction] --- # Assignment ```r a <- 10 + 5 a ``` ``` ## [1] 15 ``` -- An arrow can point in the other direction too (i.e. `->`), this is used less frequently though ```r 10 + 5 -> a a ``` ``` ## [1] 15 ``` --- # Assignment You can also use `=` for assignment, but the object must be on the left ```r b = 10 + 10 b ``` ``` ## [1] 20 ``` --- # Which assignment operator to use? Most R folks prefer `<-` for assignment .footnote[In RStudio you can assign `<-` to a hotkey to make using it as easy as `=`] -- `=` has a specific role for evaluation *within* functions, we don't want to mix them up -- It doesn't really matter though, other languages use `=` for both -- Use whatever you prefer, just be consistent --- # Help If you are struggling with a (named) function or object in R, simply type `?commandhere` ```R ?Negate ``` --- # Help Also try `vignette()` for a more detailed introduction to many packages ```R # Try this: vignette("dplyr") ``` -- Vignettes are a very easy way to learn how and when to use a package --- # Object-oriented programming In R: > "Everything is an object and everything has a name." --- # What are objects? We won't go into OOP details but here are some objects that we'll be working with regularly: - vectors - matrices - data frames - lists - functions - etc. -- A lot of these are probably familiar if you have coding experience -- But there are always language-specific features/subtleties --- # Global environment ```r ## Create a small data frame called "df". df <- data.frame(x = 1:2, y = 3:4) df ``` ``` ## x y ## 1 1 3 ## 2 2 4 ``` -- Now, let's try to run a regression1 on these "x" and "y" variables: .footnote[ 1 Yes, this is a dumb regression with perfectly co-linear variables. Just go with it. ] --- # Global environment ```r lm(y ~ x) ## The "lm" stands for linear model(s) ``` --- # Global environment ```r lm(y ~ x) ## The "lm" stands for linear model(s) ``` ``` ## Error in eval(predvars, data, env): object 'y' not found ``` -- Error? --- # Global environment The error message is ``` *## Error in eval(predvars, data, env): object 'y' not found ``` -- R can't find the variables that we've supplied in our [Global Environment](https://www.datamentor.io/r-programming/environment-scope/) -- Can you find x or y in the RStudio panel? --- # Global environment We have to tell R `x` and `y` are a part of the object `df` -- How? -- There are a various ways to solve this problem. One is to simply specify the datasource: ```r lm(y ~ x, data = df) ## Works when we add "data = df"! ``` ``` ## ## Call: ## lm(formula = y ~ x, data = df) ## ## Coefficients: ## (Intercept) x ## 2 1 ``` --- # Global environment: why it matters This matters largely for Stata users -- In Stata, the workspace is basically just a single data frame `\(\Rightarrow\)` all variables are in the global environment -- Big problem with this is you can't have multiple data frames / datasets in memory --- # Working with multiple objects We can create a second data frame in memory! ```r df2 <- data.frame(x = rnorm(10), y = runif(10)) df ``` ``` ## x y ## 1 1 3 ## 2 2 4 ``` ```r df2 ``` ``` ## x y ## 1 -1.43614571 0.8393892 ## 2 -0.62925965 0.2673705 ## 3 0.24352177 0.1951011 ## 4 1.05836223 0.2513002 ## 5 0.83134882 0.3891329 ## 6 0.10521182 0.8435996 ## 7 -1.74171309 0.9805974 ## 8 0.64524699 0.3063197 ## 9 0.09710422 0.2704168 ## 10 -0.07673369 0.6229376 ``` --- # Reserved words R has a bunch of key/reserved words that serve specific functions See [here](http://stat.ethz.ch/R-manual/R-devel/library/base/html/Reserved.html) for a full list, including (but not limited to): ```R if else while # looping function for # looping TRUE FALSE NULL # null/undefined Inf # infinity NaN # Not a number NA # Not available / missing ``` --- # Semi-reserved words There are other words that are sort of reserved, in that they have a particular meaning The most important one is `c()` which binds and concatenates objects together ```r my_vector <- c(1, 2, 5) my_vector ``` ``` ## [1] 1 2 5 ``` Other ones are `pi`, `e`, etc --- # Namespace conflicts Try loading up `dplyr` in RStudio ```r library(dplyr) ``` What warning gets reported? -- The warning *masked from 'package:X'* is about a .hi-red[namespace conflict] -- `dplyr` and the `stats` package (which gets loaded automatically when you start R) have functions named `filter` and `lag` --- # Namespace conflicts Whenever a namespace conflict arises, the most recently loaded package will gain preference -- The `filter()` function now refers specifically to the `dplyr` variant -- But what if we want the `stats` variant? -- 1. Temporarily use `stats::filter()` 2. Permanently assign `filter <- stats::filter` --- # Solving namespace conflicts Use `package::function()` -- Explicitly call a conflicted function from a package using the `package::function()` syntax: ```r stats::filter(1:10, rep(1, 2)) ``` ``` ## Time Series: ## Start = 1 ## End = 10 ## Frequency = 1 ## [1] 3 5 7 9 11 13 15 17 19 NA ``` --- # Solving namespace conflicts We can also use `::` for more than just conflicted cases. -- It can clarify where a function or dataset comes from and make the code clearer: ```r dplyr::starwars ## Print the starwars data frame from the dplyr package scales::comma(c(1000, 1000000)) ## Use the comma function, which comes from the scales package ``` -- The `::` syntax also means that we can call functions without loading package first. E.g. As long as `dplyr` is installed on our system, then `dplyr::filter(iris, Species=="virginica")` will work. --- # Solving namespace conflicts Assign `function <- package::function` -- You can permanently assign a conflicted function name to a particular package -- This will hold for the remainder of your current R session, or until you change it back: ```r filter <- stats::filter ## Note the lack of parentheses. filter <- dplyr::filter ## Change it back again. ``` --- # Indexing How do we index in R? -- We've already seen an example of indexing in the form of R console output: ```r 1+2 ``` ``` ## [1] 3 ``` The `[1]` above denotes the first (and, in this case, only) element of our output.1 -- In this case, a vector of length one equal to the value "3" --- # Indexing Try the following in your console to see a more explicit example of indexed output: ```r rnorm(n = 100, mean = 0, sd = 1) ``` ``` ## [1] 0.31561282 0.66029338 -1.72220241 -2.13462605 0.06894560 0.86782174 -2.29004418 -0.15019029 -0.26878179 1.79133204 0.67226804 -0.20930114 0.01218251 1.53411686 0.07729182 0.07843753 -0.77926107 0.16655967 0.26532457 0.89078071 -0.46788837 0.75837456 -0.64173636 ## [24] 0.62767182 0.24833013 -0.70007582 -0.56740159 -0.26139393 -1.06388504 -0.10636865 0.77110374 2.74740354 -0.08393476 0.54356763 0.75286121 -0.80867413 1.00111985 0.45605250 -1.43425031 -0.26530482 0.64176916 -0.41502103 -0.45957568 -0.79249402 -1.15853913 0.71089000 ## [47] 1.26760175 -0.14315106 -0.51502891 1.48289118 -0.16258891 0.04170917 0.48303990 -1.18012717 -0.66357374 -0.63464989 -0.70196303 0.57685038 -2.11308037 0.26090968 1.14712719 0.01479365 -0.31173924 -0.95619611 0.47341376 -1.51386408 0.16428100 -0.87086522 1.59332899 ## [70] 0.64659752 0.35736969 0.10239306 -0.67526683 0.97208502 0.75586993 -0.42828562 -0.71392476 -0.19038407 0.39986481 -0.97784491 0.18373691 -2.15031053 -0.62296653 -0.76543932 0.46430942 0.52228217 0.00979376 -0.44052620 1.19948953 -0.11746849 0.03820979 1.19480563 ## [93] 0.34395835 -0.32907297 1.67085792 -0.91805820 -0.08780733 1.32029372 1.73078613 2.16259608 ``` --- # Indexing: [] We can also use `[]` to index objects that we create in R. ```r a <- 11:20 a ``` ``` ## [1] 11 12 13 14 15 16 17 18 19 20 ``` ```r a[4] ## Get the 4th element of object "a" ``` ``` ## [1] 14 ``` ```r a[c(4, 6)] ## Get the 4th and 6th elements ``` ``` ## [1] 14 16 ``` --- # Indexing: [] It also works on larger arrays (vectors, matrices, data frames, and lists). For example: ```r starwars[1, 1] ## Show the cell corresponding to the 1st row & 1st column of the data frame. ``` ``` ## # A tibble: 1 x 1 ## name ## <chr> ## 1 Luke Skywalker ``` -- What does `starwars[1:3, 1]` give you? --- # Indexing: [] We haven't covered them properly yet (patience), but .hi-blue[lists] are a more complex type of array object in R -- They can contain a random assortment of objects that don't share the same characteristics -- - e.g. a list can contain a scalar, a string, and a data frame, or even another list --- # Indexing: [] The relevance to indexing is that lists require two square brackets `[[]]` to index the parent list item and then the standard `[]` within that parent item: ```r my_list <- list( a = "hello", b = c(1,2,3), c = data.frame(x = 1:5, y = 6:10) ) my_list[[1]] ## Return the 1st list object ``` ``` ## [1] "hello" ``` ```r my_list[[2]][3] ## Return the 3rd element of the 2nd list object ``` ``` ## [1] 3 ``` --- # Indexing: $ Lists provide a nice segue to our other indexing operator: `$` - Let's continue with the `my_list` example from the previous slide. ```r my_list ``` ``` ## $a ## [1] "hello" ## ## $b ## [1] 1 2 3 ## ## $c ## x y ## 1 1 6 ## 2 2 7 ## 3 3 8 ## 4 4 9 ## 5 5 10 ``` --- count: false # Indexing: $ Lists provide a nice segue to our other indexing operator: `$`. - Let's continue with the `my_list` example ```r my_list ``` ``` *## $a ## [1] "hello" ## *## $b ## [1] 1 2 3 ## *## $c ## x y ## 1 1 6 ## 2 2 7 ## 3 3 8 ## 4 4 9 ## 5 5 10 ``` Notice how our (named) parent list objects are demarcated: "$a", "$b" and "$c". --- # Indexing: $ We can call these objects directly by name using the dollar sign, e.g. ```r my_list$a ## Return list object "a" ``` ``` ## [1] "hello" ``` ```r my_list$b[3] ## Return the 3rd element of list object "b" ``` ``` ## [1] 3 ``` ```r my_list$c$x ## Return column "x" of list object "c" ``` ``` ## [1] 1 2 3 4 5 ``` --- # Indexing: $ The `$` form of indexing also works for other object types In some cases, you can also combine the two index options: ```r starwars$name[1] # first element of the name column of the starwars data frame ``` ``` ## [1] "Luke Skywalker" ``` --- # Indexing: $ However, note some key differences between the output from this example and that of our previous `starwars[1, 1]` example: ```r starwars$name[1] ``` ``` ## [1] "Luke Skywalker" ``` ```r starwars[1, 1] ``` ``` ## # A tibble: 1 x 1 ## name ## <chr> ## 1 Luke Skywalker ``` --- # Removing objects Use `rm()` to remove an object or objects from your working environment. ```r a <- "hello" b <- "world" rm(a, b) ``` You can also use `rm(list = ls())` to remove all objects in your working environment (except packages), but this is [frowned upon](https://www.tidyverse.org/articles/2017/12/workflow-vs-script/) -- Just start a new R session instead --- class: inverse, center, middle name: tidyverse # The tidyverse

 --- # What is "tidy" data? Resources: - [Vignette](https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html) (from the **tidyr** package) - [Original paper](https://vita.had.co.nz/papers/tidy-data.pdf) (Hadley Wickham, 2014 JSS) -- Key points: 1. Each variable forms a column. 2. Each observation forms a row. 3. Each type of observational unit forms a table. -- Basically, tidy data is more likely to be [long (i.e. narrow)](https://en.wikipedia.org/wiki/Wide_and_narrow_data) than wide --- # Checklist Install tidyverse: `install.packages('tidyverse')` Install nycflights13: `install.packages('nycflights13', repos = 'https://cran.rstudio.com')` --- # Tidyverse vs. base R Lots of debate over tidyverse vs base R -- The answer is [obvious](http://varianceexplained.org/r/teach-tidyverse/): We should teach the tidyverse first - Good documentation and support - Consistent philosophy and syntax - Nice front-end for big data tools - For data cleaning, plotting, the tidyverse is elite --- # Tidyverse vs. base R Base R is still great - Base R is extremely flexible and powerful - The tidyverse can't do everything - Using base R and the tidyverse together is often a good idea --- # Tidyverse vs. base R One point of convenience is that there is often a direct correspondence between a tidyverse command and its base R equivalent: | tidyverse | base | |---|---| | `?readr::read_csv` | `?utils::read.csv` | | `?dplyr::if_else` | `?base::ifelse` | | `?tibble::tibble` | `?base::data.frame` | Tidyverse functions typically have extra features on top of base R -- There are always many ways to achieve a single goal in R --- # Tidyverse packages Let's load the tidyverse meta-package and check the output. ```r library(tidyverse) ``` -- We have actually loaded a number of packages: **ggplot2**, **tibble**, **dplyr**, etc -- We can also see information about the package versions and some [namespace conflicts](https://raw.githack.com/uo-ec607/lectures/master/04-rlang/04-rlang.html#59) --- # Tidyverse packages The tidyverse actually comes with a lot more packages than those that are just loaded automatically ```r tidyverse_packages() ``` ``` ## [1] "broom" "cli" "crayon" "dbplyr" "dplyr" "forcats" "ggplot2" "haven" "hms" "httr" "jsonlite" "lubridate" "magrittr" "modelr" "pillar" "purrr" "readr" "readxl" "reprex" "rlang" "rstudioapi" ## [22] "rvest" "stringr" "tibble" "tidyr" "xml2" "tidyverse" ``` e.g. the **lubridate** package is for working with dates and the **rvest** package is for webscraping -- These packages have to be loaded separately --- # Tidyverse packages We're going to focus on two workhorse packages: 1. [**dplyr**](https://dplyr.tidyverse.org/) 2. [**tidyr**](https://tidyr.tidyverse.org/) These are the packages for cleaning and wrangling data -- They are thus the ones that you will likely make the most use of -- Data cleaning and wrangling occupies an inordinate amount of time, no matter where you are in your research career --- # Pipes: %>% The pipe operator `%>%` lets us perform a sequence of operations in a very nice and tidy way -- Suppose we wanted to figure out the average highway miles per gallon of Audi's in the `mpg` dataset: ```r mpg ``` ``` ## # A tibble: 234 x 11 ## manufacturer model displ year cyl trans drv cty hwy fl class ## <chr> <chr> <dbl> <int> <int> <chr> <chr> <int> <int> <chr> <chr> ## 1 audi a4 1.8 1999 4 auto(l5) f 18 29 p compact ## 2 audi a4 1.8 1999 4 manual(m5) f 21 29 p compact ## 3 audi a4 2 2008 4 manual(m6) f 20 31 p compact ## 4 audi a4 2 2008 4 auto(av) f 21 30 p compact ## 5 audi a4 2.8 1999 6 auto(l5) f 16 26 p compact ## 6 audi a4 2.8 1999 6 manual(m5) f 18 26 p compact ## 7 audi a4 3.1 2008 6 auto(av) f 18 27 p compact ## 8 audi a4 quattro 1.8 1999 4 manual(m5) 4 18 26 p compact ## 9 audi a4 quattro 1.8 1999 4 auto(l5) 4 16 25 p compact ## 10 audi a4 quattro 2 2008 4 manual(m6) 4 20 28 p compact ## # … with 224 more rows ``` --- # Pipes: %>% There's two ways you might do this without taking advantage of pipes: -- The first is to do it step-by-step, line-by-line which requires a lot of variable assignment ```r audis_mpg <- filter(mpg, manufacturer=="audi") audis_mpg_grouped <- group_by(filter(mpg, manufacturer=="audi"), model) summarise(audis_mpg_grouped, hwy_mean = mean(hwy)) ``` ``` ## # A tibble: 3 x 2 ## model hwy_mean ## <chr> <dbl> ## 1 a4 28.3 ## 2 a4 quattro 25.8 ## 3 a6 quattro 24 ``` --- # Pipes: %>% Next you could do it all in one line which is hard to read ```r summarise(group_by(filter(mpg, manufacturer=="audi"), model), hwy_mean = mean(hwy)) ``` ``` ## # A tibble: 3 x 2 ## model hwy_mean ## <chr> <dbl> ## 1 a4 28.3 ## 2 a4 quattro 25.8 ## 3 a6 quattro 24 ``` --- # Pipes: %>% Or, you could use .hi-blue[pipes] `%>%`: ```r mpg %>% filter(manufacturer=="audi") %>% group_by(model) %>% summarise(hwy_mean = mean(hwy)) ``` ``` ## # A tibble: 3 x 2 ## model hwy_mean ## <chr> <dbl> ## 1 a4 28.3 ## 2 a4 quattro 25.8 ## 3 a6 quattro 24 ``` -- It performs the operations from left to right, exactly like you'd think of them: take this object (mpg), do this (filter), then do this (group by car model), then do this (take the mean of highway miles) --- # Use vertical space Pipes are even more readable if we write it over several lines: ```r mpg %>% filter(manufacturer=="audi") %>% group_by(model) %>% summarise(hwy_mean = mean(hwy)) ``` ``` ## # A tibble: 3 x 2 ## model hwy_mean ## <chr> <dbl> ## 1 a4 28.3 ## 2 a4 quattro 25.8 ## 3 a6 quattro 24 ``` Using vertical space costs nothing and makes for much more readable code --- class: inverse, center, middle name: dplyr # dplyr

 --- # Aside: dplyr 1.0.0 release Please make sure that you are running at least **dplyr** 1.0.0 before continuing. ```r packageVersion('dplyr') ``` ``` ## [1] '1.0.2' ``` ```r # install.packages('dplyr') ## install updated version if < 1.0.0 ``` --- # The five key dplyr verbs 1. `filter`: Subset/filter rows based on their values 2. `arrange`: Reorder/arrange rows based on their values 3. `select`: Select columns/variables 4. `mutate`: Create new columns/variables 5. `summarise`: Collapse multiple rows into a single summary value, potentially by a grouping variable -- Let's practice these commands together using the `starwars` data frame that comes pre-packaged with dplyr --- # Starwars Here's the `starwars` dataset, it has 87 observations of 14 variables ```r starwars ``` ``` ## # A tibble: 87 x 14 ## name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles starships ## <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <list> <list> <list> ## 1 Luke Skywalker 172 77 blond fair blue 19 male masculine Tatooine Human <chr [5]> <chr [2]> <chr [2]> ## 2 C-3PO 167 75 <NA> gold yellow 112 none masculine Tatooine Droid <chr [6]> <chr [0]> <chr [0]> ## 3 R2-D2 96 32 <NA> white, blue red 33 none masculine Naboo Droid <chr [7]> <chr [0]> <chr [0]> ## 4 Darth Vader 202 136 none white yellow 41.9 male masculine Tatooine Human <chr [4]> <chr [0]> <chr [1]> ## 5 Leia Organa 150 49 brown light brown 19 female feminine Alderaan Human <chr [5]> <chr [1]> <chr [0]> ## 6 Owen Lars 178 120 brown, grey light blue 52 male masculine Tatooine Human <chr [3]> <chr [0]> <chr [0]> ## 7 Beru Whitesun lars 165 75 brown light blue 47 female feminine Tatooine Human <chr [3]> <chr [0]> <chr [0]> ## 8 R5-D4 97 32 <NA> white, red red NA none masculine Tatooine Droid <chr [1]> <chr [0]> <chr [0]> ## 9 Biggs Darklighter 183 84 black light brown 24 male masculine Tatooine Human <chr [1]> <chr [0]> <chr [1]> ## 10 Obi-Wan Kenobi 182 77 auburn, white fair blue-gray 57 male masculine Stewjon Human <chr [6]> <chr [1]> <chr [5]> ## # … with 77 more rows ``` --- # 1) dplyr::filter Here we are subsetting the observations of humans that are at least 190cm ```r starwars %>% filter( species == "Human", height >= 190 ) ``` ``` ## # A tibble: 4 x 14 ## name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles starships ## <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <list> <list> <list> ## 1 Darth Vader 202 136 none white yellow 41.9 male masculine Tatooine Human <chr [4]> <chr [0]> <chr [1]> ## 2 Qui-Gon Jinn 193 89 brown fair blue 92 male masculine <NA> Human <chr [1]> <chr [1]> <chr [0]> ## 3 Dooku 193 80 white fair brown 102 male masculine Serenno Human <chr [2]> <chr [1]> <chr [0]> ## 4 Bail Prestor Organa 191 NA black tan brown 67 male masculine Alderaan Human <chr [2]> <chr [0]> <chr [0]> ``` --- # 1) dplyr::filter You can filter using regular expressions with grep-type commands or the `stringr` package ```r starwars %>% filter(stringr::str_detect(name, "Skywalker")) ``` ``` ## # A tibble: 3 x 14 ## name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles starships ## <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <list> <list> <list> ## 1 Luke Skywalker 172 77 blond fair blue 19 male masculine Tatooine Human <chr [5]> <chr [2]> <chr [2]> ## 2 Anakin Skywalker 188 84 blond fair blue 41.9 male masculine Tatooine Human <chr [3]> <chr [2]> <chr [3]> ## 3 Shmi Skywalker 163 NA black fair brown 72 female feminine Tatooine Human <chr [2]> <chr [0]> <chr [0]> ``` This subsets the observations for individuals whose names contain "Skywalker" --- # 1) dplyr::filter A very common `filter` use case is identifying/removing missing data cases: ```r starwars %>% filter(is.na(height)) ``` ``` ## # A tibble: 6 x 14 ## name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles starships ## <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <list> <list> <list> ## 1 Arvel Crynyd NA NA brown fair brown NA male masculine <NA> Human <chr [1]> <chr [0]> <chr [1]> ## 2 Finn NA NA black dark dark NA male masculine <NA> Human <chr [1]> <chr [0]> <chr [0]> ## 3 Rey NA NA brown light hazel NA female feminine <NA> Human <chr [1]> <chr [0]> <chr [0]> ## 4 Poe Dameron NA NA brown light brown NA male masculine <NA> Human <chr [1]> <chr [0]> <chr [1]> ## 5 BB8 NA NA none none black NA none masculine <NA> Droid <chr [1]> <chr [0]> <chr [0]> ## 6 Captain Phasma NA NA unknown unknown unknown NA <NA> <NA> <NA> <NA> <chr [1]> <chr [0]> <chr [0]> ``` --- # 1) dplyr::filter To remove missing observations, use negation: ```r starwars %>% filter(!is.na(height)) ``` ``` ## # A tibble: 81 x 14 ## name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles starships ## <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <list> <list> <list> ## 1 Luke Skywalker 172 77 blond fair blue 19 male masculine Tatooine Human <chr [5]> <chr [2]> <chr [2]> ## 2 C-3PO 167 75 <NA> gold yellow 112 none masculine Tatooine Droid <chr [6]> <chr [0]> <chr [0]> ## 3 R2-D2 96 32 <NA> white, blue red 33 none masculine Naboo Droid <chr [7]> <chr [0]> <chr [0]> ## 4 Darth Vader 202 136 none white yellow 41.9 male masculine Tatooine Human <chr [4]> <chr [0]> <chr [1]> ## 5 Leia Organa 150 49 brown light brown 19 female feminine Alderaan Human <chr [5]> <chr [1]> <chr [0]> ## 6 Owen Lars 178 120 brown, grey light blue 52 male masculine Tatooine Human <chr [3]> <chr [0]> <chr [0]> ## 7 Beru Whitesun lars 165 75 brown light blue 47 female feminine Tatooine Human <chr [3]> <chr [0]> <chr [0]> ## 8 R5-D4 97 32 <NA> white, red red NA none masculine Tatooine Droid <chr [1]> <chr [0]> <chr [0]> ## 9 Biggs Darklighter 183 84 black light brown 24 male masculine Tatooine Human <chr [1]> <chr [0]> <chr [1]> ## 10 Obi-Wan Kenobi 182 77 auburn, white fair blue-gray 57 male masculine Stewjon Human <chr [6]> <chr [1]> <chr [5]> ## # … with 71 more rows ``` --- # 2) dplyr::arrange `arrange` sorts the data frame based on the variables you supply: ```r starwars %>% arrange(birth_year) ``` ``` ## # A tibble: 87 x 14 ## name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles starships ## <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <list> <list> <list> ## 1 Wicket Systri Warrick 88 20 brown brown brown 8 male masculine Endor Ewok <chr [1]> <chr [0]> <chr [0]> ## 2 IG-88 200 140 none metal red 15 none masculine <NA> Droid <chr [1]> <chr [0]> <chr [0]> ## 3 Luke Skywalker 172 77 blond fair blue 19 male masculine Tatooine Human <chr [5]> <chr [2]> <chr [2]> ## 4 Leia Organa 150 49 brown light brown 19 female feminine Alderaan Human <chr [5]> <chr [1]> <chr [0]> ## 5 Wedge Antilles 170 77 brown fair hazel 21 male masculine Corellia Human <chr [3]> <chr [1]> <chr [1]> ## 6 Plo Koon 188 80 none orange black 22 male masculine Dorin Kel Dor <chr [3]> <chr [0]> <chr [1]> ## 7 Biggs Darklighter 183 84 black light brown 24 male masculine Tatooine Human <chr [1]> <chr [0]> <chr [1]> ## 8 Han Solo 180 80 brown fair brown 29 male masculine Corellia Human <chr [4]> <chr [0]> <chr [2]> ## 9 Lando Calrissian 177 79 black dark brown 31 male masculine Socorro Human <chr [2]> <chr [0]> <chr [1]> ## 10 Boba Fett 183 78.2 black fair brown 31.5 male masculine Kamino Human <chr [3]> <chr [0]> <chr [1]> ## # … with 77 more rows ``` --- # 2) dplyr::arrange We can also arrange items in descending order using `arrange(desc())` ```r starwars %>% arrange(desc(birth_year)) ``` ``` ## # A tibble: 87 x 14 ## name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles starships ## <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <list> <list> <list> ## 1 Yoda 66 17 white green brown 896 male masculine <NA> Yoda's species <chr [5]> <chr [0]> <chr [0]> ## 2 Jabba Desilijic Tiure 175 1358 <NA> green-tan, brown orange 600 hermaphroditic masculine Nal Hutta Hutt <chr [3]> <chr [0]> <chr [0]> ## 3 Chewbacca 228 112 brown unknown blue 200 male masculine Kashyyyk Wookiee <chr [5]> <chr [1]> <chr [2]> ## 4 C-3PO 167 75 <NA> gold yellow 112 none masculine Tatooine Droid <chr [6]> <chr [0]> <chr [0]> ## 5 Dooku 193 80 white fair brown 102 male masculine Serenno Human <chr [2]> <chr [1]> <chr [0]> ## 6 Qui-Gon Jinn 193 89 brown fair blue 92 male masculine <NA> Human <chr [1]> <chr [1]> <chr [0]> ## 7 Ki-Adi-Mundi 198 82 white pale yellow 92 male masculine Cerea Cerean <chr [3]> <chr [0]> <chr [0]> ## 8 Finis Valorum 170 NA blond fair blue 91 male masculine Coruscant Human <chr [1]> <chr [0]> <chr [0]> ## 9 Palpatine 170 75 grey pale yellow 82 male masculine Naboo Human <chr [5]> <chr [0]> <chr [0]> ## 10 Cliegg Lars 183 NA brown fair blue 82 male masculine Tatooine Human <chr [1]> <chr [0]> <chr [0]> ## # … with 77 more rows ``` --- # 3) dplyr::select Use commas to select multiple columns out of a data frame, deselect a column with "-", select across multiple columns with "first:last": ```r starwars %>% select(name:skin_color, species, -height) ``` ``` ## # A tibble: 87 x 5 ## name mass hair_color skin_color species ## <chr> <dbl> <chr> <chr> <chr> ## 1 Luke Skywalker 77 blond fair Human ## 2 C-3PO 75 <NA> gold Droid ## 3 R2-D2 32 <NA> white, blue Droid ## 4 Darth Vader 136 none white Human ## 5 Leia Organa 49 brown light Human ## 6 Owen Lars 120 brown, grey light Human ## 7 Beru Whitesun lars 75 brown light Human ## 8 R5-D4 32 <NA> white, red Droid ## 9 Biggs Darklighter 84 black light Human ## 10 Obi-Wan Kenobi 77 auburn, white fair Human ## # … with 77 more rows ``` --- # 3) dplyr::select You can also rename your selected variables in place ```r starwars %>% select(alias = name, crib = homeworld) ``` ``` ## # A tibble: 87 x 2 ## alias crib ## <chr> <chr> ## 1 Luke Skywalker Tatooine ## 2 C-3PO Tatooine ## 3 R2-D2 Naboo ## 4 Darth Vader Tatooine ## 5 Leia Organa Alderaan ## 6 Owen Lars Tatooine ## 7 Beru Whitesun lars Tatooine ## 8 R5-D4 Tatooine ## 9 Biggs Darklighter Tatooine ## 10 Obi-Wan Kenobi Stewjon ## # … with 77 more rows ``` --- # 3) dplyr::select If you just want to rename columns without subsetting them, you can use `rename`: ```r starwars %>% rename(alias = name, crib = homeworld) ``` ``` ## # A tibble: 87 x 14 ## alias height mass hair_color skin_color eye_color birth_year sex gender crib species films vehicles starships ## <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <list> <list> <list> ## 1 Luke Skywalker 172 77 blond fair blue 19 male masculine Tatooine Human <chr [5]> <chr [2]> <chr [2]> ## 2 C-3PO 167 75 <NA> gold yellow 112 none masculine Tatooine Droid <chr [6]> <chr [0]> <chr [0]> ## 3 R2-D2 96 32 <NA> white, blue red 33 none masculine Naboo Droid <chr [7]> <chr [0]> <chr [0]> ## 4 Darth Vader 202 136 none white yellow 41.9 male masculine Tatooine Human <chr [4]> <chr [0]> <chr [1]> ## 5 Leia Organa 150 49 brown light brown 19 female feminine Alderaan Human <chr [5]> <chr [1]> <chr [0]> ## 6 Owen Lars 178 120 brown, grey light blue 52 male masculine Tatooine Human <chr [3]> <chr [0]> <chr [0]> ## 7 Beru Whitesun lars 165 75 brown light blue 47 female feminine Tatooine Human <chr [3]> <chr [0]> <chr [0]> ## 8 R5-D4 97 32 <NA> white, red red NA none masculine Tatooine Droid <chr [1]> <chr [0]> <chr [0]> ## 9 Biggs Darklighter 183 84 black light brown 24 male masculine Tatooine Human <chr [1]> <chr [0]> <chr [1]> ## 10 Obi-Wan Kenobi 182 77 auburn, white fair blue-gray 57 male masculine Stewjon Human <chr [6]> <chr [1]> <chr [5]> ## # … with 77 more rows ``` --- # 3) dplyr::select *cont.* The `select(contains(PATTERN))` option provides a nice shortcut in relevant cases. ```r starwars %>% select(name, contains("color")) ``` ``` ## # A tibble: 87 x 4 ## name hair_color skin_color eye_color ## <chr> <chr> <chr> <chr> ## 1 Luke Skywalker blond fair blue ## 2 C-3PO <NA> gold yellow ## 3 R2-D2 <NA> white, blue red ## 4 Darth Vader none white yellow ## 5 Leia Organa brown light brown ## 6 Owen Lars brown, grey light blue ## 7 Beru Whitesun lars brown light blue ## 8 R5-D4 <NA> white, red red ## 9 Biggs Darklighter black light brown ## 10 Obi-Wan Kenobi auburn, white fair blue-gray ## # … with 77 more rows ``` --- # 3) dplyr::select The `select(..., everything())` option is another useful shortcut if you only want to bring some variable(s) to the "front" of a data frame ```r starwars %>% select(species, homeworld, everything()) %>% head(5) ``` ``` ## # A tibble: 5 x 14 ## species homeworld name height mass hair_color skin_color eye_color birth_year sex gender films vehicles starships ## <chr> <chr> <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <list> <list> <list> ## 1 Human Tatooine Luke Skywalker 172 77 blond fair blue 19 male masculine <chr [5]> <chr [2]> <chr [2]> ## 2 Droid Tatooine C-3PO 167 75 <NA> gold yellow 112 none masculine <chr [6]> <chr [0]> <chr [0]> ## 3 Droid Naboo R2-D2 96 32 <NA> white, blue red 33 none masculine <chr [7]> <chr [0]> <chr [0]> ## 4 Human Tatooine Darth Vader 202 136 none white yellow 41.9 male masculine <chr [4]> <chr [0]> <chr [1]> ## 5 Human Alderaan Leia Organa 150 49 brown light brown 19 female feminine <chr [5]> <chr [1]> <chr [0]> ``` --- # 3) dplyr::select You can also use `relocate` to do the same thing ```r starwars %>% relocate(species, homeworld) %>% head(5) ``` ``` ## # A tibble: 5 x 14 ## species homeworld name height mass hair_color skin_color eye_color birth_year sex gender films vehicles starships ## <chr> <chr> <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <list> <list> <list> ## 1 Human Tatooine Luke Skywalker 172 77 blond fair blue 19 male masculine <chr [5]> <chr [2]> <chr [2]> ## 2 Droid Tatooine C-3PO 167 75 <NA> gold yellow 112 none masculine <chr [6]> <chr [0]> <chr [0]> ## 3 Droid Naboo R2-D2 96 32 <NA> white, blue red 33 none masculine <chr [7]> <chr [0]> <chr [0]> ## 4 Human Tatooine Darth Vader 202 136 none white yellow 41.9 male masculine <chr [4]> <chr [0]> <chr [1]> ## 5 Human Alderaan Leia Organa 150 49 brown light brown 19 female feminine <chr [5]> <chr [1]> <chr [0]> ``` --- # 4) dplyr::mutate You can create new columns from scratch as transformations of existing columns: ```r starwars %>% select(name, birth_year) %>% mutate(dog_years = birth_year * 7) %>% mutate(comment = paste0(name, " is ", dog_years, " in dog years.")) ``` ``` ## # A tibble: 87 x 4 ## name birth_year dog_years comment ## <chr> <dbl> <dbl> <chr> ## 1 Luke Skywalker 19 133 Luke Skywalker is 133 in dog years. ## 2 C-3PO 112 784 C-3PO is 784 in dog years. ## 3 R2-D2 33 231 R2-D2 is 231 in dog years. ## 4 Darth Vader 41.9 293.  Darth Vader is 293.3 in dog years. ## 5 Leia Organa 19 133 Leia Organa is 133 in dog years. ## 6 Owen Lars 52 364 Owen Lars is 364 in dog years. ## 7 Beru Whitesun lars 47 329 Beru Whitesun lars is 329 in dog years. ## 8 R5-D4 NA NA R5-D4 is NA in dog years. ## 9 Biggs Darklighter 24 168 Biggs Darklighter is 168 in dog years. ## 10 Obi-Wan Kenobi 57 399 Obi-Wan Kenobi is 399 in dog years. ## # … with 77 more rows ``` --- # 4) dplyr::mutate Boolean, logical and conditional operators all work well with `mutate` too: ```r starwars %>% select(name, height) %>% filter(name %in% c("Luke Skywalker", "Anakin Skywalker")) %>% mutate(tall1 = height > 180) %>% # TRUE or FALSE mutate(tall2 = ifelse(height > 180, "Tall", "Short")) ## Same effect, but can choose labels ``` ``` ## # A tibble: 2 x 4 ## name height tall1 tall2 ## <chr> <int> <lgl> <chr> ## 1 Luke Skywalker 172 FALSE Short ## 2 Anakin Skywalker 188 TRUE Tall ``` --- # 4) dplyr::mutate Lastly, combining `mutate` with `across` allows you to easily work on a subset of variables: ```r starwars %>% select(name:eye_color) %>% mutate(across(where(is.character), toupper)) %>% # Take all character variables, uppercase them head(5) ``` ``` ## # A tibble: 5 x 6 ## name height mass hair_color skin_color eye_color ## <chr> <int> <dbl> <chr> <chr> <chr> ## 1 LUKE SKYWALKER 172 77 BLOND FAIR BLUE ## 2 C-3PO 167 75 <NA> GOLD YELLOW ## 3 R2-D2 96 32 <NA> WHITE, BLUE RED ## 4 DARTH VADER 202 136 NONE WHITE YELLOW ## 5 LEIA ORGANA 150 49 BROWN LIGHT BROWN ``` --- # 5) dplyr::summarise Summarising useful in combination with the `group_by` command ```r starwars %>% group_by(species, gender) %>% # for each species-gender combo summarise(mean_height = mean(height, na.rm = TRUE)) # calculate the mean height ``` ``` ## # A tibble: 42 x 3 ## # Groups: species [38] ## species gender mean_height ## <chr> <chr> <dbl> ## 1 Aleena masculine 79 ## 2 Besalisk masculine 198 ## 3 Cerean masculine 198 ## 4 Chagrian masculine 196 ## 5 Clawdite feminine 168 ## 6 Droid feminine 96 ## 7 Droid masculine 140 ## 8 Dug masculine 112 ## 9 Ewok masculine 88 ## 10 Geonosian masculine 183 ## # … with 32 more rows ``` --- # 5) dplyr::summarise Note that including "na.rm = TRUE" is usually a good idea with summarise functions, it keeps NAs from propagating to the end result ```r ## Probably not what we want starwars %>% summarise(mean_height = mean(height)) ``` ``` ## # A tibble: 1 x 1 ## mean_height ## <dbl> ## 1 NA ``` --- # 5) dplyr::summarise We can also use `across` within summarise: ```r starwars %>% group_by(species) %>% # for each species summarise(across(where(is.numeric), mean, na.rm = T)) %>% # take the mean of all numeric variables head(5) ``` ``` ## # A tibble: 5 x 4 ## species height mass birth_year ## <chr> <dbl> <dbl> <dbl> ## 1 Aleena 79 15 NaN ## 2 Besalisk 198 102 NaN ## 3 Cerean 198 82 92 ## 4 Chagrian 196 NaN NaN ## 5 Clawdite 168 55 NaN ``` --- # Other dplyr goodies `group_by` and `ungroup`: For (un)grouping - Particularly useful with the `summarise` and `mutate` commands -- `slice`: Subset rows by position rather than filtering by values - E.g. `starwars %>% slice(c(1, 5))` --- # Other dplyr goodies `pull`: Extract a column from as a data frame as a vector or scalar - E.g. `starwars %>% filter(gender=="female") %>% pull(height)` -- `count` and `distinct`: Number and isolate unique observations - E.g. `starwars %>% count(species)`, or `starwars %>% distinct(species)` - You could also use a combination of `mutate`, `group_by`, and `n()`, e.g. `starwars %>% group_by(species) %>% mutate(num = n())`. --- # Other dplyr goodies There are also a whole class of [window functions](https://cran.r-project.org/web/packages/dplyr/vignettes/window-functions.html) for getting leads and lags, percentiles, cumulative sums, etc. - See `vignette("window-functions")`. --- # dplyr::xxxx_join The last set of commands we need are the `join` commands -- These are the same as `merge` in stata but with a bit more functionality --- # dplyr::xxxx_join We merge data with [join operations](https://cran.r-project.org/web/packages/dplyr/vignettes/two-table.html): - `inner_join(df1, df2)` - `left_join(df1, df2)` - `right_join(df1, df2)` - `full_join(df1, df2)` - `semi_join(df1, df2)` - `anti_join(df1, df2)` (You can visualize the operations [here](https://r4ds.had.co.nz/relational-data.html)) --- # dplyr::xxxx_join Lets use the data that comes with the the [nycflights13](http://github.com/hadley/nycflights13) package. ```r library(nycflights13) flights ``` ``` ## # A tibble: 336,776 x 19 ## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay carrier flight tailnum origin dest air_time distance hour minute time_hour ## <int> <int> <int> <int> <int> <dbl> <int> <int> <dbl> <chr> <int> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dttm> ## 1 2013 1 1 517 515 2 830 819 11 UA 1545 N14228 EWR IAH 227 1400 5 15 2013-01-01 05:00:00 ## 2 2013 1 1 533 529 4 850 830 20 UA 1714 N24211 LGA IAH 227 1416 5 29 2013-01-01 05:00:00 ## 3 2013 1 1 542 540 2 923 850 33 AA 1141 N619AA JFK MIA 160 1089 5 40 2013-01-01 05:00:00 ## 4 2013 1 1 544 545 -1 1004 1022 -18 B6 725 N804JB JFK BQN 183 1576 5 45 2013-01-01 05:00:00 ## 5 2013 1 1 554 600 -6 812 837 -25 DL 461 N668DN LGA ATL 116 762 6 0 2013-01-01 06:00:00 ## 6 2013 1 1 554 558 -4 740 728 12 UA 1696 N39463 EWR ORD 150 719 5 58 2013-01-01 05:00:00 ## 7 2013 1 1 555 600 -5 913 854 19 B6 507 N516JB EWR FLL 158 1065 6 0 2013-01-01 06:00:00 ## 8 2013 1 1 557 600 -3 709 723 -14 EV 5708 N829AS LGA IAD 53 229 6 0 2013-01-01 06:00:00 ## 9 2013 1 1 557 600 -3 838 846 -8 B6 79 N593JB JFK MCO 140 944 6 0 2013-01-01 06:00:00 ## 10 2013 1 1 558 600 -2 753 745 8 AA 301 N3ALAA LGA ORD 138 733 6 0 2013-01-01 06:00:00 ## # … with 336,766 more rows ``` --- # dplyr::xxxx_join ```r planes ``` ``` ## # A tibble: 3,322 x 9 ## tailnum year type manufacturer model engines seats speed engine ## <chr> <int> <chr> <chr> <chr> <int> <int> <int> <chr> ## 1 N10156 2004 Fixed wing multi engine EMBRAER EMB-145XR 2 55 NA Turbo-fan ## 2 N102UW 1998 Fixed wing multi engine AIRBUS INDUSTRIE A320-214 2 182 NA Turbo-fan ## 3 N103US 1999 Fixed wing multi engine AIRBUS INDUSTRIE A320-214 2 182 NA Turbo-fan ## 4 N104UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE A320-214 2 182 NA Turbo-fan ## 5 N10575 2002 Fixed wing multi engine EMBRAER EMB-145LR 2 55 NA Turbo-fan ## 6 N105UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE A320-214 2 182 NA Turbo-fan ## 7 N107US 1999 Fixed wing multi engine AIRBUS INDUSTRIE A320-214 2 182 NA Turbo-fan ## 8 N108UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE A320-214 2 182 NA Turbo-fan ## 9 N109UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE A320-214 2 182 NA Turbo-fan ## 10 N110UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE A320-214 2 182 NA Turbo-fan ## # … with 3,312 more rows ``` --- # Joining operations Let's perform a left join on the flights and planes datasets - *Note*: I'm going subset columns after the join, but only to keep text on the slide -- ```r left_join(flights, planes) %>% select(year, month, day, dep_time, arr_time, carrier, flight, tailnum, type, model) ``` ``` ## Joining, by = c("year", "tailnum") ``` ``` ## # A tibble: 336,776 x 10 ## year month day dep_time arr_time carrier flight tailnum type model ## <int> <int> <int> <int> <int> <chr> <int> <chr> <chr> <chr> ## 1 2013 1 1 517 830 UA 1545 N14228 <NA> <NA> ## 2 2013 1 1 533 850 UA 1714 N24211 <NA> <NA> ## 3 2013 1 1 542 923 AA 1141 N619AA <NA> <NA> ## 4 2013 1 1 544 1004 B6 725 N804JB <NA> <NA> ## 5 2013 1 1 554 812 DL 461 N668DN <NA> <NA> ## 6 2013 1 1 554 740 UA 1696 N39463 <NA> <NA> ## 7 2013 1 1 555 913 B6 507 N516JB <NA> <NA> ## 8 2013 1 1 557 709 EV 5708 N829AS <NA> <NA> ## 9 2013 1 1 557 838 B6 79 N593JB <NA> <NA> ## 10 2013 1 1 558 753 AA 301 N3ALAA <NA> <NA> ## # … with 336,766 more rows ``` --- # Joining operations Note that dplyr made a reasonable guess about which columns to join on (i.e. columns that share the same name), and told us what it chose ``` *## Joining, by = c("year", "tailnum") ``` There's an obvious problem here: the variable `year` does not have a consistent meaning across our joining datasets -- In one it refers to the *year of flight*, in the other it refers to *year of construction* Luckily, there's an easy way to avoid this problem: try `?dplyr::join` --- # Joining operations You just need to be more explicit in your join call by using the `by = ` argument ```r left_join( flights, planes %>% rename(year_built = year), ## Not necessary w/ below line, but helpful by = "tailnum" ## Be specific about the joining column ) %>% select(year, month, day, dep_time, arr_time, carrier, flight, tailnum, year_built, type, model) %>% head(3) ## Just to save vertical space on the slide ``` ``` ## # A tibble: 3 x 11 ## year month day dep_time arr_time carrier flight tailnum year_built type model ## <int> <int> <int> <int> <int> <chr> <int> <chr> <int> <chr> <chr> ## 1 2013 1 1 517 830 UA 1545 N14228 1999 Fixed wing multi engine 737-824 ## 2 2013 1 1 533 850 UA 1714 N24211 1998 Fixed wing multi engine 737-824 ## 3 2013 1 1 542 923 AA 1141 N619AA 1990 Fixed wing multi engine 757-223 ``` --- # Joining operations Note what happens if we again specify the join column but don't rename the ambiguous `year`: ```r left_join(flights, planes, ## Not renaming "year" to "year_built" this time by = "tailnum") %>% select(contains("year"), month, day, dep_time, arr_time, carrier, flight, tailnum, type, model) %>% head(3) ``` ``` ## # A tibble: 3 x 11 ## year.x year.y month day dep_time arr_time carrier flight tailnum type model ## <int> <int> <int> <int> <int> <int> <chr> <int> <chr> <chr> <chr> ## 1 2013 1999 1 1 517 830 UA 1545 N14228 Fixed wing multi engine 737-824 ## 2 2013 1998 1 1 533 850 UA 1714 N24211 Fixed wing multi engine 737-824 ## 3 2013 1990 1 1 542 923 AA 1141 N619AA Fixed wing multi engine 757-223 ``` -- Make sure you know what "year.x" and "year.y" are --- class: inverse, center, middle name: tidyr # tidyr

 --- # Key tidyr verbs 1. `pivot_longer`: Pivot wide data into long format (i.e. "melt", "reshape long") 2. `pivot_wider`: Pivot long data into wide format (i.e. "cast", "reshape wide") 3. `separate`: Split one column into multiple columns 4. `unite`: Combine multiple columns into one -- Let's practice these verbs together in class --- # 1) tidyr::pivot_longer ```r stocks <- data.frame( time = as.Date('2009-01-01') + 0:1, X = rnorm(2, 0, 1), Y = rnorm(2, 0, 2), Z = rnorm(2, 0, 4) ) stocks ``` ``` ## time X Y Z ## 1 2009-01-01 0.6163028 1.779204 -1.461802 ## 2 2009-01-02 -0.1024888 2.548539 -2.816719 ``` We have 4 variables, the date and the stocks How do we get this in tidy form? --- # 1) tidyr::pivot_longer ```r stocks %>% pivot_longer(-time, names_to = "stock", values_to = "price") ``` We need to pivot the stock name variables `X, Y, Z` longer 1. Choose non-time variables: `-time` 2. Decide what variable holds the names: `names_to = "stock"` 3. Decide what variable holds the values: `values_to = "price"` --- # 1) tidyr::pivot_longer ```r stocks %>% pivot_longer(-time, names_to = "stock", values_to = "price") ``` ``` ## # A tibble: 6 x 3 ## time stock price ## <date> <chr> <dbl> ## 1 2009-01-01 X 0.616 ## 2 2009-01-01 Y 1.78 ## 3 2009-01-01 Z -1.46 ## 4 2009-01-02 X -0.102 ## 5 2009-01-02 Y 2.55 ## 6 2009-01-02 Z -2.82 ``` --- # 1) tidyr::pivot_longer Let's quickly save the "tidy" (i.e. long) stocks data frame for use on the next slide ```r tidy_stocks <- stocks %>% pivot_longer(-time, names_to = "stock", values_to = "price") ``` --- # 2) tidyr::pivot_wider ```r tidy_stocks %>% pivot_wider(names_from = stock, values_from = price) ``` ``` ## # A tibble: 2 x 4 ## time X Y Z ## <date> <dbl> <dbl> <dbl> ## 1 2009-01-01 0.616 1.78 -1.46 ## 2 2009-01-02 -0.102 2.55 -2.82 ``` ```r tidy_stocks %>% pivot_wider(names_from = time, values_from = price) ``` ``` ## # A tibble: 3 x 3 ## stock `2009-01-01` `2009-01-02` ## <chr> <dbl> <dbl> ## 1 X 0.616 -0.102 ## 2 Y 1.78 2.55 ## 3 Z -1.46 -2.82 ``` -- Note that the second example has effectively transposed the data --- # 3) tidyr::separate ```r economists <- data.frame(name = c("Adam.Smith", "Paul.Samuelson", "Milton.Friedman")) economists ``` ``` ## name ## 1 Adam.Smith ## 2 Paul.Samuelson ## 3 Milton.Friedman ``` ```r economists %>% separate(name, c("first_name", "last_name")) ``` ``` ## first_name last_name ## 1 Adam Smith ## 2 Paul Samuelson ## 3 Milton Friedman ``` -- This command is pretty smart. But to avoid ambiguity, you can also specify the separation character with `separate(..., sep=".")` --- # 3) tidyr::separate A related function is `separate_rows`, for splitting up cells that contain multiple fields or observations (a frustratingly common occurence with survey data) ```r jobs <- data.frame( name = c("Jack", "Jill"), occupation = c("Homemaker", "Philosopher, Philanthropist, Troublemaker") ) jobs ``` ``` ## name occupation ## 1 Jack Homemaker ## 2 Jill Philosopher, Philanthropist, Troublemaker ``` --- # 3) tidyr::separate A related function is `separate_rows`, for splitting up cells that contain multiple fields or observations (a frustratingly common occurence with survey data) ```r ## Now split out Jill's various occupations into different rows jobs %>% separate_rows(occupation) ``` ``` ## # A tibble: 4 x 2 ## name occupation ## <chr> <chr> ## 1 Jack Homemaker ## 2 Jill Philosopher ## 3 Jill Philanthropist ## 4 Jill Troublemaker ``` --- # 4) tidyr::unite ```r gdp <- data.frame( yr = rep(2016, times = 4), mnth = rep(1, times = 4), dy = 1:4, gdp = rnorm(4, mean = 100, sd = 2) ) gdp ``` ``` ## yr mnth dy gdp ## 1 2016 1 1 104.73167 ## 2 2016 1 2 102.67609 ## 3 2016 1 3 97.35823 ## 4 2016 1 4 97.84164 ``` --- # 4) tidyr::unite ```r ## Combine "yr", "mnth", and "dy" into one "date" column gdp %>% unite(date, c("yr", "mnth", "dy"), sep = "-") ``` ``` ## date gdp ## 1 2016-1-1 104.73167 ## 2 2016-1-2 102.67609 ## 3 2016-1-3 97.35823 ## 4 2016-1-4 97.84164 ``` --- # 4) tidyr::unite Note that `unite` will automatically create a character variable: ```r gdp_u <- gdp %>% unite(date, c("yr", "mnth", "dy"), sep = "-") %>% as_tibble() gdp_u ``` ``` ## # A tibble: 4 x 2 ## date gdp ## <chr> <dbl> ## 1 2016-1-1 105. ## 2 2016-1-2 103. ## 3 2016-1-3 97.4 ## 4 2016-1-4 97.8 ``` -- If you want to convert it to something else (e.g. date or numeric) then you will need to modify it using `mutate` --- # 4) tidyr::unite ```r library(lubridate) gdp_u %>% mutate(date = ymd(date)) ``` ``` ## # A tibble: 4 x 2 ## date gdp ## <date> <dbl> ## 1 2016-01-01 105. ## 2 2016-01-02 103. ## 3 2016-01-03 97.4 ## 4 2016-01-04 97.8 ``` --- # Other tidyr goodies Use `crossing` to get the full combination of a group of variables ```r crossing(side=c("left", "right"), height=c("top", "bottom")) ``` ``` ## # A tibble: 4 x 2 ## side height ## <chr> <chr> ## 1 left bottom ## 2 left top ## 3 right bottom ## 4 right top ``` -- See `?expand` and `?complete` for more specialized functions that allow you to fill in (implicit) missing data or variable combinations in existing data frames --- class: inverse, center, middle name: tidyverse # Randomization and inference

 --- # The Rubin causal model .hi-blue[What is a causal effect?] -- A starting point is that a causal effect is a .hi[comparison between two potential outcomes] -- What the difference in some outcome in the presence vs the absence of a given treatment? -- e.g. what is the difference in health between high and low levels of air pollution? -- Let's begin formalizing this idea --- # The Rubin causal model: potential outcomes Suppose we have a set of .hi-blue[observational units] - These can be people, states, animals, air quality monitors, etc Each unit has two .hi[potential outcomes], but only one is observed -- The potential outcome is `\(Y^1_i\)` if unit `\(i\)` received some treatment, and `\(Y^0_i\)` if the unit did not -- `\(Y^0_i\)` corresponds to the control state of the world for `\(i\)` --- # The Rubin causal model: observable outcomes Note that these potential outcomes are not the same as observable outcomes -- Observable outcomes are outcomes that actually show up in the data (i.e. the *factual* outcome)1 .footnote[ 1The potential outcome that did not happen is called the *counterfactual outcome.* ] -- We can write the observable outcome `\(Y_i\)` as a simple equation: `$$Y_i = D_i Y^1_i + (1-D_i)Y^0_i$$` where `\(D_i = 1\)` if `\(i\)` received treatment and 0 otherwise --- # The Rubin causal model: treatment effects `$$Y_i = D_i Y^1_i + (1-D_i)Y^0_i$$` This is the basis of the *Rubin causal model* -- Rubin defines a treatment/causal effect `\(\delta_i\)` as the difference between the two potential outcomes: `$$\delta_i = Y^1_i - Y^0_i$$` -- This leads to an obvious problem: we only observe one of these two states for each unit `\(i\)` but we need to know both to recover `\(\delta_i\)` --- # The Rubin causal model: average treatment effect We can derive three parameters of interest from the definition of a treatment effect: .hi-blue[Average treatment effect (ATE):] `$$E[\delta_i] = E[Y^1_i - Y^0_i] = E[Y^1_i] - E[Y^0_i]$$` -- This is the average of the individual treatment effects -- It is also unknowable --- # Rubin: average treatment on the treated .hi-blue[Average treatment on the treated (ATT):] `$$E[\delta_i|D_i=1] = E[Y^1_i - Y^0_i |D_i=1] = E[Y^1_i|D_i=1] - E[Y^0_i|D_i=1]$$` -- This is the average of the individual treatment effects only for the `\(i\)` in the treated group -- It is also unknowable since we never observe `\(E[Y^0_i|D_i=1]\)` -- If the treatment effect differs across `\(i\)` then `\(ATE \neq ATT\)` --- # Rubin: average treatment on the untreated .hi-blue[Average treatment on the untreated (ATU):] `$$E[\delta_i|D_i=0] = E[Y^1_i - Y^0_i |D_i=0] = E[Y^1_i|D_i=0] - E[Y^0_i|D_i=0]$$` -- This is the average of the individual treatment effects only for the `\(i\)` in the untreated group -- It is also unknowable since we never observe `\(E[Y^1_i|D_i=0]\)` -- If the treatment effect differs across `\(i\)` then `\(ATE \neq ATU\)` --- # Hands on: understanding treatment effects We've got our definitions, now lets be a bit more clear as to what we are doing with some empirical examples putting our new R tools to work -- Suppose treatment `\(D\)` is whether a state has a conservation policy -- The outcome `\(Y\)` is the number of bird species in that state -- We want to understand the causal effect of conservation policy on the number of species --- # Hands on: understanding treatment effects Here's our dataset of both potential outcomes: ```r cons_df # data frame of conservation treatment, outcomes ``` ``` ## # A tibble: 8 x 4 ## state Y1 Y0 delta ## <int> <dbl> <dbl> <dbl> ## 1 1 4 7 -3 ## 2 2 7 9 -2 ## 3 3 7 0 7 ## 4 4 10 1 9 ## 5 5 7 7 0 ## 6 6 6 0 6 ## 7 7 9 3 6 ## 8 8 13 4 9 ``` Calculate the average treatment effect --- # Hands on: understanding treatment effects ATE = `\(E[\delta_i]\)` which we can compute with `dplyr::summarise`: ```r cons_df %>% dplyr::summarise(mean(delta)) ``` ``` ## # A tibble: 1 x 1 ## `mean(delta)` ## <dbl> ## 1 4 ``` The average treatment effect is 4: a conservation policy increases the number of bird species in a state by 4 --- # Hands on: understanding treatment effects Notice that not all states benefit from conservation policies, and it even backfires in one state The ATE is just the average over all the different treatment effects ```r cons_df ``` ``` ## # A tibble: 8 x 4 ## state Y1 Y0 delta ## <int> <dbl> <dbl> <dbl> ## 1 1 4 7 -3 ## 2 2 7 9 -2 ## 3 3 7 0 7 ## 4 4 10 1 9 ## 5 5 7 7 0 ## 6 6 6 0 6 ## 7 7 9 3 6 ## 8 8 13 4 9 ``` --- # Hands on: understanding treatment effects Suppose we have a perfect policymaker who knows each state's potential outcomes and can perfectly decide whether each state should have a conservation policy -- The policymaker assigns treatment and then observes the actual outcome according to `\(Y_i = D_i Y^1_i + (1-D_i)Y^0_i\)` -- What does the dataset look like for the observed outcomes? --- # Hands on: understanding treatment effects ```r observed_df <- cons_df %>% mutate( Y = ifelse(delta > 0, Y1, Y0), D = as.numeric(delta > 0) ) %>% select(state, Y, D) observed_df ``` ``` ## # A tibble: 8 x 3 ## state Y D ## <int> <dbl> <dbl> ## 1 1 7 0 ## 2 2 9 0 ## 3 3 7 1 ## 4 4 10 1 ## 5 5 7 0 ## 6 6 6 1 ## 7 7 9 1 ## 8 8 13 1 ``` --- # Hands on: estimating ATEs Given the .hi-blue[observed] data, what if we tried to *estimate* the ATE by comparing mean outcomes of treated `\((D_i=1)\)` vs untreated units `\((D_i=0)\)` -- This is the simple difference in mean outcomes (SDO): `\begin{align} SDO &= E[Y^1|D=1] - E[Y^0|D=0] \\ &= {1 \over N_T}\sum_{i=1}^{N_T} (y_i | d_i = 1) - {1 \over N_U}\sum_{i=1}^{N_U} (y_i | d_i = 0) \end{align}` where `\(N_T\)` is the number of treated units and `\(N_U\)` is the number of untreated units --- # Hands on: estimating ATEs We can compute the SDO using `dplyr::summarise` in conjunction with `dplyr::group_by` on treatment status `\(D\)`: ```r observed_df %>% dplyr::group_by(D) %>% dplyr::summarise(meanY = mean(Y)) ``` ``` ## # A tibble: 2 x 2 ## D meanY ## <dbl> <dbl> ## 1 0 7.67 ## 2 1 9 ``` The SDO is 9 - 7.67 = 1.33 < 4! --- # Hands on: estimating ATEs We know the ATE is 4, why is the SDO giving us a much smaller estimate of the effect of the conservation policy? -- Because the SDO is actually composed of three pieces, only one of which is the ATE:1 .footnote[ See Mixtape pages 89-91 for the derivation. ]  SDO = ATE + -- Selection Bias + -- Heterogeneous Treatment Effect Bias  -- What are these mathematically and intuitively? --- # Hands on: decomposing the SDO `\begin{align} \underbrace{E[Y^1|D=1] - E[Y^0|D=0]}_{\text{SDO}} =& \underbrace{{1 \over N_T}\sum_{i=1}^{N_T} (y_i | d_i = 1) + {1 \over N_U}\sum_{i=1}^{N_U} (y_i | d_i = 0)}_{\text{SDO}} \\ =&\underbrace{E[Y^1] - E[Y^0]}_{\text{ATE}} \\ &+\underbrace{E[Y^0 | D = 1] - E[Y^0 | D= 0]}_{\text{Selection bias}} \\ &+\underbrace{(1-\pi)(ATT - ATU)}_{\text{Het. Treat. Eff. Bias}} \end{align}` where `\(\pi\)` is the share of treated units --- # Hands on: decomposing the SDO The LHS we know sums to 1.33 so the RHS must as well -- The first term is the ATE, what we actually want to estimate -- We .hi-blue[know] ATE = 4 so the last two terms are why SDO < ATE -- Let's work through these two in more detail --- # Hands on: Selection bias The second term is .hi-blue[selection bias:] `$$E[Y^0 | D = 1] - E[Y^0 | D= 0]$$` It is the inherent differences between the two groups if they did not *actually* get the conservation policy -- .hi[how are the two groups different at baseline?] -- The problem is we don't observe `\(E[Y^0 | D = 1]\)`: what the treated group `\((D=1)\)` would have looked like without treatment `\((Y^0)\)` -- Calculate this with the `cons_df` data frame with both potential outcomes --- # Hands on: Selection bias ```r cons_df %>% mutate( D = as.numeric(delta > 0) ) %>% group_by(D) %>% summarise(mean(Y0)) # Difference in potential control outcomes across the two groups ``` ``` ## # A tibble: 2 x 2 ## D `mean(Y0)` ## <dbl> <dbl> ## 1 0 7.67 ## 2 1 1.6 ``` Selection bias is thus 1.6 - 7.67 = -6.07 --- # Hands on: het. treat. effect bias The third term is the bias from heterogeneous treatment effects across groups: `$$\underbrace{(1-\pi)}_{\text{Share w/o policy}} \times \underbrace{(ATT - ATU)}_{\text{Diff. in treat. effect}}$$` It is the difference in the effect of the conservation policy across the two groups multiplied by the share that did not get a conservation policy --- # Hands on: het. treat. effect bias `$$\underbrace{(1-\pi)}_{\text{Share w/o policy}} \times \underbrace{(ATT - ATU)}_{\text{Diff. in treat. effect}}$$` What's the intuition for this? -- Basically: - How much more of an effect did the policy have on the units that *happened* to get treatment versus the effect the policy would have had on the units that *happened* to be in the control group -- - Scale the size of this difference by fraction of control units -- Calculate this with the `cons_df` data frame with both potential outcomes --- # Hands on: het. treat. effect bias ```r cons_df %>% mutate( D = as.numeric(delta > 0) ) %>% group_by(D) %>% summarise(mean(delta), n()) # Difference in potential control and treatment outcomes across the two groups, count in treatment and control ``` ``` ## # A tibble: 2 x 3 ## D `mean(delta)` `n()` ## <dbl> <dbl> <int> ## 1 0 -1.67 3 ## 2 1 7.4 5 ``` Heterogeneous treatment effect bias is thus: (1 - 5/(5+3))*(7.4 - (-1.67)) = 3.40 -- In total we have: .hi[SDO (1.33) = ATE (4) + SB (-6.07) + HTEB (3.40)] --- # Recap: treatment effect estimates Taking a simple difference in means of outcomes between treatment and control groups .hi[does] contain what we want -- Unfortunately it is .hi-red[confounded] by two forms of bias: -- 1. .hi-blue[Selection bias:] The units in the treatment group different from the control group in the absence of treatment -- 2. .hi-blue[Heterogeneous treatment effect bias]: The units in the treatment group respond to treatment differently than units in the control group --- # Bias examples What are some examples of these forms of bias? -- .hi-blue[Selection bias:] -- If we select certain groups into treatment, for example, if we pass conservation policy in states with little biodiversity - This may lead to a negative bias on conservation policy impacts, even with a positive ATE -- .hi-blue[HTEB:] -- If we select units into treatment based on expected response, for example, if we pass policy in states where birds are **very** sensitive to conservation - This may lead to an overestimate of the size of the treatment effect --- # Empirical economics A huge chunk of economics is trying circumvent these forms of bias1 .footnote[ 1We're more heavily concerned with selection bias than HTEB. There are also other biases to worry about that we will get to later. ] -- The rest of class is largely dedicated to: 1. Understanding what tools we have to correctly estimate the ATE -- 2. Understanding what assumptions are required for these tools to be valid -- 3. Understanding when these ATE estimates are valid -- 4. Seeing how this all works (hands on) in modern environmental economics -- This should be a less-technical complement to Brian's class --- # Recovering the ATE There are many different ways to recover the ATE -- We will cover some subset of: 1. Randomized control trials 2. Regression discontinuity 3. Difference-in-differences 4. Cross-sectional regressions 5. Two way fixed effects -- All of these approaches have pluses and minuses --- # Understanding the source of bias What is the fundamental problem leading to selection bias and HTEB? -- Both stem from treatment being correlated with characteristics of the observational units -- Selection bias is because treatment is correlated with unit characteristics in the absence of treatment -- HTEB is because treatment is correlated with the size of units' responses to treatment --- # A potential solution What's a simple way to deal with this? --  .hi-blue[RANDOMIZATION]  -- If we randomize treatment across observational units then SDO = ATE -- Randomization drives selection bias and HTEB to zero -- Let's see why --- # Randomization Randomization means that treatment will be .hi-blue[independent] of the potential outcomes -- Mathematically: `$$Y^1,Y^0 \perp D$$` -- In our example we know independence is violated because we assigned the conservation policy to states that had `\(Y^1 > Y^0\)` -- What if we randomized conservation policy? --- # Randomization Randomization of treatment / policy means that: `$$E[Y^1 | D = 1] - E[Y^1 | D = 0] = 0 \qquad E[Y^0 | D = 1] - E[Y^0 | D = 0] = 0$$` -- If we randomized treatment across units, then on average, the difference in potential outcomes across groups should be zero -- This means that: `$$E[Y^1 | D = 1] = E[Y^1 | D = 0] \qquad E[Y^0 | D = 1] = E[Y^0 | D = 0]$$` -- The second equation `\(E[Y^0 | D = 1] - E[Y^0 | D = 0] = 0\)` directly gives us that selection bias is zero with the SDO --- # Randomization What does randomization do to HTEB? -- `\begin{align} &ATT - ATU \\ & = (E[Y^1 | D = 1] - E[Y^0 | D = 1]) - (E[Y^1 | D = 0] - E[Y^0 | D = 0]) \\ & = (E[Y^1 | D = 1] - E[Y^1 | D = 0]) - (E[Y^0 | D = 1] - E[Y^0 | D = 0]) \\ & = (0) - (0) = 0 \end{align}` -- HTEB goes to zero! --- # Randomization and SDO Randomization means that SDO = ATE: `$$\underbrace{{1 \over N_T} \sum_{i=1}^{N_T} (y_i | d_i = 1) - {1 \over N_C} \sum_{i=1}^{N_C} (y_i | d_i = 0)}_{\text{SDO}} = \underbrace{E[Y^1] - E[Y^0]}_{\text{ATE}}$$` -- All we need to estimate the average treatment effect of a policy is: -- 1. Data on treatment assignment -- 2. Data on observable outcomes -- 3. The independence assumption: `\(Y^1,Y^0 \perp D\)` --- # Hands on: randomization and SDO Let's see how this works in practice by re-constructing our dataset and then randomizing treatment -- ```r set.seed(12345) ate <- 4 # average treatment effect n_obs <- 100 # number of observations cons_rand_df <- tibble( state = seq(1, n_obs), # state identifier Y0 = floor(runif(n_obs)*10)) %>% # control/untreated potential outcome mutate( D = as.numeric(runif(n()) > 0.5), # randomized treatment Y1 = Y0 + ate + round(rnorm(n())), # generate treatment potential outcome Y = D*Y1 + (1-D)*Y0 # generate observed outcome ) %>% select( state, D, Y # keep only observable variables ) ``` --- # Hands on: randomization and SDO ```r cons_rand_df # data frame of randomized treatment, observable outcome ``` ``` ## # A tibble: 100 x 3 ## state D Y ## <int> <dbl> <dbl> ## 1 1 0 7 ## 2 2 1 11 ## 3 3 1 11 ## 4 4 1 11 ## 5 5 1 8 ## 6 6 1 4 ## 7 7 1 7 ## 8 8 1 11 ## 9 9 0 7 ## 10 10 0 9 ## # … with 90 more rows ``` --- # Hands on: randomization and SDO Now take the SDO: ```r cons_rand_df %>% dplyr::group_by(D) %>% dplyr::summarise(meanY = mean(Y)) ``` ``` ## # A tibble: 2 x 2 ## D meanY ## <dbl> <dbl> ## 1 0 4.42 ## 2 1 8.82 ``` The SDO is 8.82 - 4.42 = 4.40, a very close estimate of the ATE! -- As `n_obs` `\(\rightarrow \infty\)`, we will have that `\(SDO \rightarrow ATE\)` --- # Randomization: what it does and doesn't do What does independence imply? -- `$$E[Y^1|D = 1] - E[Y^1 | D = 0] = 0 \qquad E[Y^0|D = 1] - E[Y^0 | D = 0] = 0$$` The two groups have the same potential outcomes, .hi-blue[*on average*] -- What does independence .hi-red[not] imply? That: `$$E[Y^1 | D = 1] - E[Y^0 | D = 0] = 0 \qquad E[Y^1 | D = 1] - E[Y^0 | D = 1] = 0$$` It does not imply that the observed outcomes are the same across the two groups, nor does it imply that the two potential outcomes of a single group are the same --- # Independence Is independence is a reasonable assumption in .hi[observational data]? -- Observational data are just data that we collect from the real world - e.g. observed state policy choices and outcomes, observed pollution levels and outcomes -- Independence is .hi-red[unlikely] to hold in these scenarios -- So what can we do? --- # Conditional independence Often we will instead rely on .hi-red[conditional independence:] `$$(Y^1,Y^0 \perp D) | X$$` -- After controlling for some other variables `\(X\)`, the potential outcomes are independent of treatment `\((Y^1,Y^0 \perp D)\)` -- This is much weaker than (unconditional) independence: we only need independence to hold for units that share the same `\(X\)` values, not for all units -- Many of the estimation tools used in economics rely on (variants of) the conditional independence assumption


ATM Withdrawal:
accountNumber
AccountNumber
id
Balance
location
Amount to withdraw:
Submit
10
Reset
20
50
Return
100
We could not find any Account information for ID:
accountNumber


RSA
RSA Institute Of Technology
(Engineering College)
Affiliated to MUMBAI UNIVERSITY OF MUMBAI
GRADE CARD
Cer.No
Candidate Full Name
Candidate Mother Name
Examination Name
Year of Examination
Branch
Seat No
Course Name
Grade
Credit Points
Remark


class: center, middle, title background-image: url(splash.jpg) # **8 Things** That Make **Continuous Delivery** Go Nuts
Eduards Sizovs
@eduardsi
??? - 8 vs 10 - for bigger orgs - real world experience --- class: orgalign, middle, fuckup .left-column[ .no[1] ] .right-column[ # Lack of Organisational Alignment ## ] --- class: middle, fuckup background-image:url(shadow.jpg) .left-column[ .no[1] ] .right-column[ # Lack of Organisational Alignment ## AKA Shadow Continuous Delivery ] --- class: center, middle # Story time ## (Fintech Startup, Germany) ??? - Once devs decided to implement continuous delivery... - Leam lead - once in a day - Ops did give support (review changes at end of Sprint) - devs continue on their own. - devs build pipeline and start deploying to production every day. - they prepare a list of changes at the end of Sprint "because Ops wants so". - middle of week - production goes down (critical integration point) - Ops team goes though a list of changes and finds nothing - Recovery takes 6 hours because Devs are AFRAID to admit they deploy once a day --- # Consequences ## SLA missed - lost integration partner and ~20K customers ## .fade[Lost trust between management and IT] ## .fade[Lost trust between dev and ops] ## .fade[Team Lead fired] --- # Consequences ## .fade[SLA missed - lost integration partner and ~20K customers] ## Lost trust between management and IT ## .fade[Lost trust between dev and ops] ## .fade[Team Lead fired] --- # Consequences ## .fade[SLA missed - lost integration partner and ~20K customers] ## .fade[Lost trust between management and IT] ## Lost trust between dev and ops ## .fade[Team Lead fired] --- # Consequences ## .fade[SLA missed - lost integration partner and ~20K customers] ## .fade[Lost trust between management and IT] ## .fade[Lost trust between dev and ops] ## Team Lead fired --- # Lessons learned ## Since CD is primary a HUMAN system, ALIGNMENT is vital ## .fade[See the reason behind every denial] ## .fade[Emphatize (work as an Ops guy for a week, pair-ops)] ## .fade[Compromise, don't go to the extreme on the first jump] ## .fade[Apply Beer-Driven Diplomacy] --- # Lessons learned ## .fade[Since CD is primary a HUMAN system, ALIGNMENT is vital] ## See the reason behind every denial ## .fade[Emphatize (work as an Ops guy for a week, pair-ops)] ## .fade[Compromise, don't go to the extreme on the first jump] ## .fade[Apply Beer-Driven Diplomacy] --- # Lessons learned ## .fade[Since CD is primary a HUMAN system, ALIGNMENT is vital] ## .fade[See the reason behind every denial] ## Emphatize (work as an Ops guy for a week, pair-ops) ## .fade[Compromise, don't go to the extreme on the first jump] ## .fade[Apply Beer-Driven Diplomacy] --- # Lessons learned ## .fade[Since CD is primary a HUMAN system, ALIGNMENT is vital] ## .fade[See the reason behind every denial] ## .fade[Emphatize (work as an Ops guy for a week, pair-ops)] ## Compromise, don't go to the extreme on the first jump ## .fade[Apply Beer-Driven Diplomacy] --- # Lessons learned ## .fade[Since CD is primary a HUMAN system, ALIGNMENT is vital] ## .fade[See the reason behind every denial] ## .fade[Emphatize (work as an Ops guy for a week, pair-ops)] ## .fade[Compromise, don't go to the extreme on the first jump] ## Apply Beer-Driven Diplomacy --- class: center, middle # Move Ops to cross-functional team as you do with QA. ??? - Instead of delegating Ops duties (throwing over the wall) and building separate "Ops Team", give each team someone awesome at Ops. --- class: spotify, middle, center ![](spotify.png) ### https://dl.dropboxusercontent.com/u/1018963/Articles/SpotifyScaling.pdf --- class: middle, fuckup background-image:url(heroes.jpg) .left-column[ .no[2] ] .right-column[ # Relying on a Tiger Team ] --- class: center, middle # Story time ## (Fintech Enterprise, Latvia) ??? - A special Tiger Team comprising of coolest devs and "DevOps guys" has been formed...  - Tigers go to losers and ask to make next deployment via pipeline - Losers refuse since they have different goals and no time "for playing games" - Tigers ask for manager's help. Together, they try to convince losers by gently setting deadline. - Thanks to paid overtimes, losers give up. - Rumors in the kitchen that managers values tigers more than losers. - Losers make deployment after 10 hot fixes introduced by tigers. Losers are making fun of tigers. - Pipeline is green. Tigers are having a drink. - Losers are having a drink too - they celebrate the fact Tigers wasted 1 month of their life building this weird tool. - The next deployment is made in an old fashion, since team is more comfortable with it... Or just in spite of tigers. - The only team who maintains and loves pipeline is Tiger Team. --- # Lessons learned ## Adoption must go from the inside-out. ## .fade[External support is highly appreciated. But not more than that.] ## .fade[CD is just solution for a problem. Team that is in trouble should learn how to solve it.] ??? - team that directly benefit from CD (feels like tigers). - if you build a new team, "loosers" will probably reject initiave, because they will feel offended or unimportant. - Instead, find a team that has delivery troubles and agree that they should move faster and reliably. --- # Lessons learned ## .fade[Adoption must go from the inside-out. ] ## External support is highly appreciated. But not more than that. ## .fade[CD is just solution for a problem. Team that is in trouble should learn how to solve it.] ??? Sometimes when you ask team to go fast, they assume you understand that it means "cutting corners, such as tests". Bring external support. --- # Lessons learned ## .fade[Adoption must go from the inside-out.] ## .fade[External support is highly appreciated. But not more than that.] ## CD is just solution for a problem. Team that is in trouble should learn how to solve it. ??? - invest in teams ability to solve their problems and continuously improve. CD will follow. - why consultants can't implement continuous delivery (ok, solve problem, what's next?) --- class: middle, fuckup .left-column[ .no[3] ] .right-column[ # Newton Law's Enemies ] --- class: story, middle, center ## The system being produced will tend to have a structure that mirrors the structure of the group that is producing it, whether or not this was intended. ### (c) Conway's Law --- background-image:url(crack-lv.png) class: whitebg --- background-image:url(crack-lv-sys.png) class: whitebg --- background-image:url(crack-ca-sys.png) class: whitebg --- background-image:url(crack-ca-wall.png) class: whitebg ??? it's very hard communicate (time overlap, california team). Instead of fixing the wall first, managers said: BRING THE PAIN FORWARD! LET'S MOVE FASTER AND DEPLOY AS SOON AS FEATURE IS READY. CD IS SOLELY TECHNICAL CHALLENGE ANYWAY. --- background-image:url(crack.png) class: whitebg ??? System start to crack because: - In order to avoid communication penalty we have to find way to isolate!!! - Teams grow siloed. - Code grows, architecture grows siloed and knowledge islands appear (no PP, no CodeReviews, WHAT'S THAT???) - Teams started to blame each other (if there was a bug introduced by team A, team B doesn't care) - Team A promised to deliver today, but due to Team B they have to rollback. OOPS.  ### (c) Jeffrey Sologov --- class: whitebg background-image:url(all-teams-on-single-codebase.png) --- class: whitebg background-image:url(uimwdb.png) ??? -- this one is worse!!! -- can you ship features by solely doing changes in DB? --- class: whitebg background-image:url(bc.png) ??? -- it's not about doing micro services!!! -- Pursue MAXIMUM autonomy and team independence and org unit indendence. --- # Takeaways ## Continuous Delivery may require organisation to change (continuously, bi-di) ## .fade[Organise systems and teams around business capabilities] ## .fade[Never split a domain across geo locations / org units] ## .fade[Respect Newton's Laws of Software Development] ??? before going fast, clean up the basement - improve architecture so it enabled CD. And since you can't build architecture right from the first shot, org ability to adapt or change is vital. --- # Takeaways ## .fade[Continuous Delivery may require organisation to change (continuously, bi-di)] ## Organise systems and teams around business capabilities ## .fade[Never split a domain across geo locations / org units] ## .fade[Respect Newton's Laws of Software Development] --- # Takeaways ## .fade[Continuous Delivery may require organisation to change (continuously, bi-di)] ## .fade[Organise systems and teams around business capabilities] ## Never split a domain across geo locations / org units ## .fade[Respect Newton's Laws of Software Development] --- # Takeaways ## .fade[Continuous Delivery may require organisation to change (continuously, bi-di)] ## .fade[Organise systems and teams around business capabilities] ## .fade[Never split a domain across geo locations / org units] ## Respect Newton's Laws of Software Development ??? - There are many more rules, you can learn about them in MMM --- class: center, middle ## How does a project get to be a year late?... One day at a time ### (c) The Mythical Man-Month ??? мало помалу (понемножечку так) --- class: center, middle, whitebg, shadow ![](mmm.jpg) ![](orgpatterns.jpg) --- class: middle, fuckup .left-column[ .no[4] ] .right-column[ # Misunderstanding of Fundamental Principles ## Because blindly following practices is not enough ] --- ## Working in Mainline when branching is necessary ## .fade[Deployment pipeline is in place, but inconsistent] ## .fade[Buying expensive production-like environments for non-critical service testing] ## .fade[If somethings difficult or painful, do it more often (what if you're doing WRONG THING or doing THINGS WRONG?)] ??? - Experiments that never needs to be merged back. --- ## .fade[Working in Mainline when branching is necessary] ## Deployment pipeline is in place, but inconsistent ## .fade[Buying expensive production-like environments for non-critical service testing] ## .fade[If somethings difficult or painful, do it more often (what if you're doing WRONG THING or doing THINGS WRONG?)] --- ## .fade[Working in Mainline when branching is necessary] ## .fade[Deployment pipeline is in place, but inconsistent] ## Buying expensive production-like environments for non-critical service testing ## .fade[If somethings difficult or painful, do it more often (what if you're doing WRONG THING or doing THINGS WRONG?)] --- ## .fade[Working in Mainline when branching is necessary] ## .fade[Deployment pipeline is in place, but inconsistent] ## .fade[Buying expensive production-like environments for non-critical service testing] ## If somethings difficult or painful, do it more often (what if you're doing WRONG THING or doing THINGS WRONG?) --- background-image: url(condev.jpg) class: center, middle, condev ??? My personal statistics: people who didn't read this book do obvious mistakes during their CD journey. --- class: center, middle # Invite me for a coffee. ??? I am unemployed, use this opportunity! --- class: middle, fuckup .left-column[ .no[5] ] .right-column[ # Optimizing the wrong thing ] ??? - Deliver more!!! Optimize locally. Suboptimize the whole. --- background-image:url(push2.jpg) class: whitebg # Push ??? Example: more developers than QA people in the team. It doesnt make sense to produce more than QA can handle, because you don't want to overload people.  A lot of things to test? Yes! When at least something will be on prod? No clue! We have to test everything! --- background-image:url(pull.png) class: whitebg # Pull instead. ??? Hey, we can't test more than 1 item at a time. Period. --- background-image:url(balancedqa.png) class: center, middle, trello ??? - to pull the next item from dev, item under test must go to prod first. - In order to move forward, everyone tries to accelerate QA --- background-image:url(balanceddev.png) class: center, middle, trello ??? You reduced WIP! --- background-image:url(little.jpg) class: center, middle, whitebg ??? Linear relatinship between work-in-progress and average lead time. --- class: center, middle, little Team has 32 cards in process (total WIP) and a throughput of 2 cards/day, Average lead time = **???** days ??? 16 If the same team maintains the same throughput but increases its total WIP to 40 cards, the average cycle time becomes 32 days. **1 item on production is better than 10 in progress** --- background-image:url(trello.png) class: center, middle, whitebg, trello ??? - What do you normally do? GRAB THE NEXT FEATURE (OPTIMIZE FOR UTILIZATION). - AND YOU INCREASE BATCH SIZE. --- # What can you do instead? ## Unblock value stream ## .fade[Work on solution that will prevent blocking in future] ## .fade[Help your team finish existing work] ## .fade[Help other teams finish existing work] ## .fade[Improve things] --- # What can you do instead? ## .fade[Unblock value stream] ## Work on solution that will prevent blocking in future ## .fade[Help your team finish existing work] ## .fade[Help other teams finish existing work] ## .fade[Improve things] --- # What can you do instead? ## .fade[Unblock value stream] ## .fade[Work on solution that will prevent blocking in future] ## Help your team finish existing work ## .fade[Help other teams finish existing work] ## .fade[Improve things] --- # What can you do instead? ## .fade[Unblock value stream] ## .fade[Work on solution that will prevent blocking in future] ## .fade[Help your team finish existing work] ## Help other teams finish existing work ## .fade[Improve things] --- # What can you do instead? ## .fade[Unblock value stream] ## .fade[Work on solution that will prevent blocking in future] ## .fade[Help your team finish existing work] ## .fade[Help other teams finish existing work] ## Improve things --- # Takeaways ## Optimize the whole (In every system, exactly one constraint determines the system’s capacity) ## .fade[Reduce and limit WiP] ## .fade[Single item on production beats 10 under development] --- # Takeaways ## .fade[Optimize the whole (In every system, exactly one constraint determines the system’s capacity)] ## Reduce and limit WIP ## .fade[Single item on production beats 10 under development] ??? We know that reduced WIP makes us faster. And there are also impirical studies that show that large batches decrease quality. --- # Takeaways ## .fade[Optimize the whole (In every system, exactly one constraint determines the system’s capacity)] ## .fade[Reduce and limit WiP] ## Single item on production beats 10 under development --- class: center, middle, whitebg, shadow, kanban ![](kanban.jpg) --- class: middle, fuckup .left-column[ .no[6] ] .right-column[ # Delivering the wrong thing ] --- class: center, middle, story ## Continuous Delivery is a software engineering approach in which teams keep producing **valuable** software in short cycles and ensure that the software can be reliably released at any time ### (c) Wikipedia --- class: center, middle, whitebg, shit ![](shit.png) ??? Does it make sense to accelerate delivery of shit? Does it make sense to spend money on doing it realibly? --- class: middle, center # Before doing continuous delivery, find out what **value** really means, for whom and how to mine it.  Top secret: you don't have to do a lot to achieve a lot. - Vanity metrics (seems like we do a lot to delight customer) - Work smart, not hard. --- class: middle, fuckup background-image:url(woz.jpg) .left-column[ .no[7] ] .right-column[ # Technical Obsession ] --- class: middle ## Very famous product company, Australia ## Challenge: implement auto-deployment with auto-scaling ## Stack: AWS, Capistrano (too simple and boring) ??? Go and implement auto-deployment and auto-scaling with AWS and Capistrano, but no... --- background-image:url(deedee.jpg) ## it sure would be fun to use... --- background-image:url(deedee.jpg) ## it sure would be fun to use... ## Docker --- background-image:url(deedee.jpg) ## it sure would be fun to use... ## Docker ## because... --- background-image:url(deedee.jpg) ## it sure would be fun to use... ## Docker ## because... ## everyone is using it --- background-image:url(deedee.jpg) ## it sure would be fun to use... ## Docker ## because... ## everyone is using it ## and it opens new opportunities --- background-image: url(coreos.png) --- background-image: url(mesos.png) --- background-image: url(marathon.png) --- class: center, middle # 1 month later... --- background-image:url(hydra.jpg) # Hydra was born --- background-image:url(hydra.jpg) class: story # Hydra was born ## In Greek mythology, the Lernaean Hydra was an ancient serpent-like water monster with reptilian traits. **It possessed many heads – more heads than the vase-painters could paint – and for each head cut off it grew two more**. ??? - They felt so small and totally out of control. Team became slave of the system. --- background-image:url(ww.jpg) class: center, middle, story ## After 3 months long war, team suffered many losses and capitulated. It took them one more month to move back to good, old and simple stack and just 2 weeks to achieve the initial goal. --- class: center, middle, story background-image:url(lollypop.jpg) ## [Trendy solutions] are like teenage sex. Everyone is talking about it but few have actually done it. ## --- class: center, middle, story background-image:url(lollypop.jpg) ## [Trendy solutions] are like teenage sex. Everyone is talking about it but few have actually done it. ## Even if everyone is doing... doesn't mean you need to --- class: center, middle, story background-image:url(leaky.jpg) # The Law of Leaky Abstractions hurts --- class: center, middle, story background-image:url(complexity.jpg) ## **Complexitax** is the extra cost that you pay for accidental complexity introduced by a tool that does more than you need. ### (c) Neal Ford ??? --- class: center, middle, story ## Write down exactly what it is about the current stack that makes solving the problem prohibitively expensive and difficult. ??? Rule: before you or someone adds new tools or frameworks. - you will be surprised how little new things you really need and will avoid overproduction (important for CD) - Nothing opens more opportunities than simple, easy-to-understand solution (even if it's not fancy) --- class: middle, fuckup .left-column[ .no[8] ] .right-column[ # Giving up too early ] --- background-image:url(cry.jpg) ## I don't keep up with testing! --- background-image:url(cry.jpg) ## I don't keep up with testing!   ## QA engineers have different manager, he will not understand. --- background-image:url(cry.jpg) ## I don't keep up with testing!  ## QA engineers have different manager, he will not understand. ## Deployment is so cumbersome. --- background-image:url(cry.jpg) ## I don't keep up with testing!  ## QA engineers have different manager, he will not understand. ## Deployment is so cumbersome. ## You really want me to talk to security officer? --- background-image:url(cry.jpg) ## I don't keep up with testing!  ## QA engineers have different manager, he will not understand. ## Deployment is so cumbersome. ## You really want me to talk to security officer? ## It doesn't work for us! --- class: center, middle # Continuous Delivery causes so many problems! ???  The worst things that can happen is when your stop in face of obstacles. All companies have opportunities to change. --- class: center, shadow, whitebg ![](leanenterprise.jpg) ![](changetheworld.png) ??? - In fact, you don't have to change the world. You have to start changing just your tiny company. - And remember that inspiration comes during work not before it. --- class: nopadding, center, middle, thanks background-image: linear-gradient(rgba(0, 0, 0, 0.5), rgba(0, 0, 0, 0.5)), url(applause.gif) # Thank you


Schizophrenia, other diseases require much more than medicine
``It's marked by a feeling of: `I don't belong,''' says
psychologist Dennis O'Grady, who treats schizophrenics individually
and works with SA. ```And you're telling me by how you're treating
me that I am correct.'''
Therapy helps schizophrenics see that the world isn't as
threatening and dangerous as their instincts tell them, O'Grady
says. It also can teach them how to cope with everyday stresses,
says Bentley, ``how to think of alternatives rather than suicide.''
From cooking to people skills, she says, the mentally ill need
to learn life fundamentals that others take for granted. ``I don't
know why society thinks if you're mentally ill, you can't learn
anything. Nothing could be farther from the truth.''
In recent months, NAMI has tried to make support and knowledge
more available for those with psychoses by establishing what it
calls COPE, for Caring, Outreach, Partnership and Education. The
resource program, also directed toward family members, includes a
consumer guide about patient empowerment, a housing guide, a family
guide with productive ways of coping and a resource directory.


WITNESS LIST
Natural Resources
March 12, 2013 9:00 AM
ON:
Vaca, Scott (Texas Parks & Wildlife - L.E.), Austin, TX
Registering, but not testifying:
FOR:
Lewis, Gib (Exotic Wildlife Assoc), Austin, TX
ON:
Jensen, Michael J. (Mike) (Texas Parks and Wildlife Department), Austin, TX
FOR:
Tedder, Jo Karr (Central Texas Water Coalition), Spicewood, TX
Registering, but not testifying:
ON:
Motal, Becky (Lower Colorado River Authority), Austin, TX
Registering, but not testifying:
FOR:
Baum, Walt (AECT), Austin, TX
Gibbs, Gary (American Electric Power), Austin, TX
ON:
Urban, John Paul (PUC & TX), Austin, TX
Woodfin, Dan (ERCOT), Taylor, TX
ON:
Lott, Earl (TCEQ), Austin, TX
Registering, but not testifying:
FOR:
Wynn, Monty (Texas Municipal League), Austin, TX
FOR:
Felan, Chris (Atmos Energy), Dallas, TX
Registering, but not testifying:
FOR:
Billingsley, Anne (Oneok, Inc.), Austin, TX
Cannon, Thure (Texas Pipeline Association), Austin, TX
Nugent, Patrick (Texas Pipeline Assoc.), Austin, TX
ON:
Abrahamson, Ed (Railroad Commission of TX), Austin, TX
FOR:
Hinojos, Paul (Reeves County Commissioner), Pecos, TX
Zeman, Alan R. (Reeves Co. Ad. Hoc Groundwater Comm.), Pecos, TX
WITNESS LIST
Natural Resources
March 12, 2013 9:00 AM
Registering, but not testifying:
FOR:
Bang, W.J. County Judge (Reeves County), Pecos, TX
Reyes, Fred (City of Pecos), Pecos, TX
ON:
Mills, Kelly (TCEQ), Austin, TX
FOR:
Mann, James (Texas Pipeline Assoc.), Austin, TX
Registering, but not testifying:
FOR:
Cannon, Thure (Texas Pipeline Association), Austin, TX
Nugent, Patrick (Texas Pipeline Assoc.), Austin, TX
ON:
Ross McDonald, Mary ("Polly") (Railroad Commission of Texas, Pipeline Safety
Division), Austin, TX
Registering, but not testifying:
FOR:
Aus, Fred (Texas Rural Water Association), Austin, TX
Cohen, Howard (Utility District Advisory Corporation), Houston, TX
Cohen, Howard (Schwartz, Page, & Harding, LLP), Houston, TX
Gonzalez, Daniel (Texas Association of REALTORS), Austin, TX
Lary, Trey (Association of Water Board Directors), Houston, TX
Maxwell, David (AWBD), San Antonio, TX
Munoz, Ned (Texas Association of Builders), Austin, TX
Phenix, Billy (Self), Austin, TX
Robbins, Dean (Texas Water Conservation Assn.), Austin, TX
Thomas, Chelsey (Texas Association of Realtors), Austin, TX


Co-ordinates of places are used to give location. We express the co-ordinates in degrees (°), minutes (') and seconds (").
We read the latitude reference on the side margins of the map and the longitude reference on the top or bottom of the map.
As we zoom in and the scale of the map gets bigger, so we need to give more information for locating places on the map. Finding location of places on the political map of Africa requires only degrees. For a provincial map you will need to give degrees and minutes, and for a topographic map at a scale of 1 : 50 000, you will need to give degrees, minutes and seconds. This will be illustrated to you when you do the activities.
Note that the latitude and longitude (co-ordinates) of the buildings at Uitsig on the topographic map extract below are 25°44'18"S; 27°59'28"E.


Noah Knows Code
Week 7 @ DBC
March 3rd, 2016
JavaScript
One of the first things I read about JavaScript was that it did not have classes. This sounded strange to me, considering that JavaScript is also an object oriented language. My concern was that if I am constantly making numerous objects, many of them will have very similar characteristics. But, if I had to make each and every object instance by manually assembling the object properties, I am going to be spending a lot of time re-writing object traits.
Constructor Functions
As it turns out, JavaScript does have something extremely similar to Ruby classes. While completing exercises in JS, I was asked to use a constructor function. I noticed that a constructor function pre-loads the object properties when you call for a new instance of an object. This seemed strangely similar to the purpose of a Ruby class.
Syntax
The difference between Ruby classes and JavaScript constructor functions comes down to syntax. In fact, the most reassuring thing about learning programming so far has been that each and every language has completely unique syntax, but the logic is very much the same. This fact alone gives me the assurance that I am learning a way to communicate with the computer, and the placement of the symbols I use is just a small aspect of that communication.