Yet each week, the door of Josh Gruenke, associate director of student activities, creaks open as another former organization president’s eyes glisten with hope of reviving a piece of their college experience as they peek their head into the room.
When COVID-19 began its spread across the country, the Campus Involvement Center, for the first time, had a simpler goal than getting students acclimated and involved.
“We wanted to make sure that our students were physically safe,” Gruenke said. “We wanted to make sure that we were not increasing any risks.” Without the capability of organizations to function on campus, the message of involvement on campus shifted to checking on students’ mental well-being, so organization numbers plummeted, Gruenke said.
The pandemic left OU with an additional 236 frozen organizations awaiting reinstallment, according to Gruenke. OU currently has just 487 active student organizations on campus.
After reporting those numbers solemnly, Gruenke sits at his desk flipping through the past years of stagnancy amongst organizations represented on paper and finds another number: 914. The number of student groups on campus before March 2020.
Though some may not have been active, pre-pandemic organization numbers rarely seemed to dip below 700, Gruenke said.
Accordingly, the 2019 OU Women's Club Rugby Team charged the field each week with a full roster and a bench of eager subs awaiting an opportunity to get in on the action. Excitement on the field and from the sideline each practice, game and tournament manifested a culture of camaraderie and unity.
That is until news of the pandemic hit campus.
Suddenly, the excitement dwindled down to a tired Zoom call in the spring of 2020 hosted by Teagan Richman, a 2022 OU graduate. Then just a member of the team, her graduating predecessors dawned her the title of president of a six-player squad returning the following year, juxtaposing the roster of 20 Richman had once known.
“We had to come back in the fall, and everything was different,” Richman said. “Everything was online. We weren't allowed to practice. We didn't have a season anymore.”
The online format provided an insurmountable obstacle for recruiting and practice. With just a fourth of their previous roster size, even if restrictions were lifted, Richman would have needed to triple the roster to just fill the starting positions.
The lost year also accounted for an absence of team bonding and a maintenance of the culture that Richman and her former team had worked tirelessly to uphold in and outside of practice. This was one of the biggest losses for the team, Richman said.
Returning in the fall of 2021, after the past year of stagnancy, Richman surveyed her field and envisioned the team she had played with her freshman year. She knew it was her responsibility to make up for the recruitment effort that was lost over the past year and a half.
Richman was willing to do anything to restore the team back to its former glory, which included attending every first-year student organization event available on campus, she said. From the Campus Involvement Fair to the Party at Ping, Richman and what remained of her team could be found with their own booth. “We had like 30 people sign up,” Richman said. “We had about that many come out to the first practice and we were super excited. They didn’t realize their workload coming back to school in person; we ended up losing half of those people.”
All Richman wanted was to keep the team alive long enough to show the new generation of college women a physical, empowering sport, Richman said. She also said she just wanted the team to give more women what it had given her.
“There was something about the rugby team that just immediately invited me in,” Richman said. “It didn't matter what you look like, it didn't matter your skill level, it didn't matter who you were outside of the team, you were immediately a part of their family.”
Another 2022 graduate, Mark Russell, expressed a similar sentiment towards his extensive student organization experience.
Russell was the president of the Bobcats Spikeball Club: a club based around a 2 vs. 2 yard game that has grown in popularity over the past years.
“The best way to describe it is it’s volleyball, but on the ground,” Russell said. “Instead of hitting it over the net … you’re spiking on the net, hence the name ‘spikeball.’”
Russell described the game with a passion as he expressed how near the club is to his heart. In fact, Russell has referred to the club as his baby ever since he revived the club on campus this past summer.
“The previous president graduated last year,” Russell said. “He entrusted us, so I felt a responsibility … his final words as president to me were, ‘Take this club to the moon.’”
Though the club struggled to gain a following after OU students were sent home due to COVID-19, Russell was determined to relive the memories he had created in the club as an underclassman.
Every night during his sophomore year, Russell and his friends went outside and practiced against each other, but more importantly spending time with one another. His love for it grew just as it did for those around him.
“Spikeball was so much fun, and we have those memories with it,” Russell said. “I want other people to have that.”
Before being brought back in Russell’s hands, he remembered a friend of his from his junior year — someone he met in a spikeball club dwindling due to COVID-19. The student came to him and told him he was thinking about transferring because the spikeball club had been his only way of finding friends on campus.
“That really stuck with me,” Russell said. “This spikeball club has more to it than just spikeball.”
The story struck Russell and pushed him along his journey as president. He re-registered the club and put in the work to create an outreach team to assist him and the other executive members of the spikeball club in restoring the club on campus and restoring its former community-oriented priorities. He dug deeper into the connections that lay beneath the game and centered the club around that.
“I’ve always been a huge people person,” Russell said. “That's why it's so important for me to facilitate those types of bonds in my club between my members. I don't want them just showing up, having fun at spikeball and then going their separate ways and not talking outside of it.”
Each week, Gruenke gets former fraternity members, activists and low-level athletes sitting in front of his desk pleading their case for an organization with a majority following that graduated years before.
Despite the bleakness of the situation, Gruenke has seen it as an opportunity to improve the involvement system.
The Involvement Calculator was born from Gruenke and the Career Involvement Center’s optimism. In lieu of the cancelled 2020 Involvement Fair, the center presented students with a detailed questionnaire designed to mimic the fair on a personal level, providing students with a detailed list of five organizations catered directly towards each individual.
“That’s not a computer that does that,” Gruenke said. “Those are all individually read by a person … That's basically what our staff does all summer.”
The calculator is the newest installment in Bobcat Student Orientation, and it yielded a response from over 3,000 students eager for any way to get involved.
“I felt really badly for students that were missing out on the traditional college experience,” Grunke said. “I wanted to do whatever we could to make sure they got the best of whatever we could offer.”
Now, Gruenke said the university is reaching the pinnacle of its involvement recovery phase. Not only are more organizations popping up on campus, but students have shown interest in reinstating past groups that were victims of the pandemic.
Even so, the students interested in leading the organizations still have a long way to go.
“Many of these students have never even seen a fully functioning organization, and now they’re president of one,” Gruenke said. “We are trying to help them transition into those roles.”
Lily Wittman, a sophomore studying sports management and president of the sports management fraternity Sigma Alpha Sigma Mu, is one of those students Gruenke was talking about. Wittman is a sophomore learning from the remnants of the organization’s returning members about the process of the organization.
Though Wittman is an underclassman in just her first in-person semester with the fraternity, she knew her organization inside and out, which she would have to, considering the struggles the sports management fraternity has faced.
Before COVID-19, Sigma Alpha Sigma Mu was seen as a very selective and prestigious organization with GPA standards, an intensive interview process and a strict level of professionalism upheld throughout its 40-50 members. However, as numbers began to dwindle in the fall of 2020 and recruitment events were moved online, the fraternity was presented with a brand-new struggle.
The fraternity began to suffer from declining recruitment numbers despite attempts to keep the organization going in an online setting, Wittman said. However, it was more difficult than expected to maintain the same number of members, let alone foster the same culture, when each meeting was on a computer screen.
Even as an incoming freshman at the time, Wittman knew the organization was too special to give up on, despite not having met any of her fellow members in-person.
“They all made sure to know my name, my major, where I was from and I just felt special,” Wittman said. “So, I wanted to make sure that I put all my effort into bringing everything back.”
Unfortunately, some things were impossible to save, including the semesterly networking trips that serve as a core aspect of the fraternity, Wittman said. Losing the trips during COVID-19 decreased the professional and career-oriented value the organization had once sustained, making it even more difficult to convince new students of the fraternity's value.
To assist younger leaders, the Career Involvement Center created a written guide for executive leadership teams and hosted training days to enforce a strong leadership-based structure for the organizations to work from.
“(It’s) getting them back in that mindset of ‘Why did they join a student org?’” Gruenke said.
Though the effects of COVID-19 still linger in student organizations today, members’ passion for the organizations have begun the revival process.
OU Women’s Rugby has worked its way up to a 15-player roster, Richman said after a year’s worth of recruiting. Though this is the minimum number of players necessary, the team has been able to play through a successful fall season and attend the annual Nash Bash Rugby Festival this spring.
Though the team still has some recovery work to do, Richman said confidently that they have trained their members to maintain the inviting, yet tough, culture as they pass the organization on to younger generations of women in rugby. The team will undoubtedly reach and surpass their former roster size, but until then, Richman said she could not be happier with the hands she has left the program in.
Similarly, the Bobcats Spikeball Club has maintained 15-20 consistent members that meet each day to play the beloved game. As they make their way to the casual practices, Russell observes his community grow stronger and deeper.
“It’s been important to me because this is one of the best ways for me to be active, along with meeting a lot of new people,” Alex Graff, a 2022 graduate and former treasurer of the Spikeball Club, said.
Graff said though spikeball is fun, he values the friendships that the club fosters just as much, and Russell’s leadership has heightened that sense of community.
“(Mark’s) whole personality is just very energetic and welcoming,” Graff said. “He’s great at the whole public speaking thing, so he’s been great with making people feel involved.”
Russell, with the help of his fellow club leaders and outreach team, have set up many events over the past year alone, including formal tournaments that invite students from the University of Cincinnati and The Ohio State University, but also date parties to build the relationships within the club.
“I just want all of OU to know and be ready,” Russell said. “Spikeball club is going to be a large club here at OU. I can feel it.”
Though Sigma Alpha Sigma Mu has not quite reached the same amount of involvement as it once had before the pandemic, Wittman said the 35 current members they have recruited over the previous two years reflect the same values and standards that were expected before the campus shutdown.
Additionally, the organization has made up for its previous loss of the national networking trips by taking three this year to Columbus, Detroit and Pittsburgh.
“It's so different shaking hands with somebody instead of meeting them like on Zoom,” Wittman said. “That's why we did so many trips. Even if it is an hour to Columbus, it's still physically meeting that person, and they're going to remember your face.”
With plans already in the works for future trips to places such as New York City, Sigma Alpha Sigma Mu expects to have fully recovered from the effects of COVID-19 on involvement by the fall.
“This semester was, ‘We just need to get back on our feet,’” Wittman said. “We just wanted to do so many events at once instead of focusing on a big consulting project, but that will be our focus for the fall.”
As time goes on, OU organizations will improve at different rates. Whether it is the level of involvement or the frequency of activities, organizations undoubtedly need time to improve aspects of their organizations before restoring them to their most efficient and enjoyable state.
However, improvement has been a constant throughout nearly all student organizations after COVID-19, and this can only be boiled down to genuine interest and affinity for the organizations’ purposes and their members.


00-Instuctor-Guide.Rmd
treesiftr is an R package (R Core Team 2018) for visualizing the relationship between phylogenetic trees and phylogenetic data. Phylogenetic trees are crucial to the study of comparative biology, taxonomy, and evolution. However, understanding how to read a phylogenetic tree, and how a phylogenetic tree relates to underlying phylogenetic data, remains challenging.
treesiftr is intended to supplement classroom discussions of phylogeny. Sample slides that could be presented before the activity are included on the software’s GitHub repository. I recommend using treesiftr after a preliminary discussion of tree thinking (such as that found in Baum, Smith, and Donovan (2005)), phylogeny estimation, and how to read phylogenetic trees. After the activity, come back together as a class to talk through the activity and the answers. For more novice classes, it may even be beneficial to do the first five questions, then regroup to discuss the answers before proceding.
treesiftr has two modes. The first is a graphical user interface (GUI). Exercise 01-treesiftr is meant to be used with the graphical interface. The second is the RStudio interface. 02-treesiftr-advanced is meant to be used in RStudio, with executeable code blocks.
The first part of the instructor guide concerns the use of the GUI. The second part explains the RStudio interface, with worked examples.
Tip
Linked text goes to the glossary. If you see a term you don’t recognize, remember you can refresh your memory at the bottom of this worksheet!
Navigate to the treesiftr application.
treesiftr works by subsetting a phylogenetic matrix using the start
and step
arguments. These are found on the left-hand side of the screen. The start
argument controls where in the matrix you would like to begin visualizing characters. For example, a start value of 1 would indicate to begin visualizing characters from the first character in the matrix. The step
value indicates how many characters at once to visualize. A step value of three would indicate characters should be viewed in threes. For example, if start
= 1 and step = 3
, the visualization will show characters 1, 2, and 3. The start and end points of character visualizations are noted in the upper left-hand corner of the visualization. For example, in the below, visualization begins with the first character in the dataset. The first three characters (characters 1, 2, and 3) in the data matrix are visualized.
There are also two checkboxes. The parsimony score checkbox uses Phangorn’s (Schliep 2011, Schliep2017) parsimony scoring algorithm to print the parsimony score of the visualized characters on the tree. The likelihood score is also calculated in Phangorn under the Mk model (Lewis 2001). Leave these unchecked until otherwise noted.
treesiftr works by subsetting a phylogenetic matrix by user input. Then, a parsimony tree is constructed in Phangorn (Schliep 2011, Schliep2017) from the user-defined subset. The tree is scored under both parsimony and Lewis’ Mk model (Lewis 2001) for discrete character data. The data and tree are then visualized using ggtree (Yu et al. 2017), based upon the ggplot2 package (Wickham 2016). This application makes use of Shiny to provide a graphical interface, but there is a second included tutorial for more experienced users of the R statistical language.
Instructor’s Note: The answer is two, as viewed in the below screenshot.
Instructor’s Note: They should be larger. Parsimony estimation will favor the shortest tree while no such criteria is applied to a random tree. Sample output is below, though the exact numbers and distribution will vary.
Instructor’s Note: The answer is Ursavus brevirhinus, Ballusia elmensis, and Ursavus primaevus. Ursavus primaevus is no longer sister to Ursavus
brevirhinus and Ballusia elmensis on the tree of character 2-3. Sample output follows.
Instructor’s Note: The root of the tree likely blue, with a change to brown on the branch leading to Tremarctos, Melursus, Helarctos, and Ursus. Ursus thibetanus posesses the ancestral blue state, and is the best candidate for reversal.
Instructor’s Note: To make this decision, we would want to know if there are other bears that are not sampled on this tree that posess this character.
Instructor’s Notes: It is possible for characters with the same number of parsimony steps to have different likelihood scores.
Instructor’s Notes: Answers may vary. But generally, a character (character 46) that implies changes on a fairly short branch is replaced with one implying changes on longer branches.
Instructor’s Notes: The likelihood should become more negative (less likely)
Instructor’s Notes: 1
Instructor’s Notes: For some trees, we can see that there is no character data supporting the grouping. Parsimony software generally displays fully-resolved trees - this may be misleading if little to no data support the grouping!
Instructor’s Note: A sample slide deck for more advanced audiences is included in the folder
inst/slides
. This slide deck is meant to be viewed in RStudio, as it has executeable code blocks. Hoever, it is also included in HTML format, if this is desireable for presentation. This is simply included as a sample of how one might choose to deliver a lecture to cover some of the phylogenetic terms in this activity. It is not necessary to us this lecture to make use of this activity!
treesiftr works by subsetting a phylogenetic matrix using the start
, stop
and step
arguments. The start
argument controls where in the matrix you would like to begin visualizing characters. For example, a start value of 1 would indicate to begin visualizing characters from the first character in the matrix. The stop
value indicates what will be the first character in the last visualization. A stop value of 10 would indicate that the first character in the last visualization should be character 10. The step
value indicates how many characters at once to visualize. A step value of three would indicate characters should be viewed in threes. For example, if start
= 1, stop
= 10, and step = 3
, 10 visualizations will be produced. The first will visualize characters 1, 2, and 3. The final will be characters 10, 11, and 12.
A maximum parsimony tree is then estimated from each dataset. The tree is scored under both parsimony and Lewis’ Mk model (Lewis 2001) for discrete character data. The data and tree are then visualized using ggtree, based upon the ggplot2 package. This application makes use of Shiny to provide a graphical interface, but in this tutorial, we will use the R Studio interface to visualize our data.
Currently, treesiftr can be installed via the devtools install_github
function. treesiftr has a number of required packages. Install and load the below.
Instructor’s Note: This does not auto-execute in the instructor’s guide. phangorn, alignfigr, and ggplot2 are all available via CRAN. ggtree is available via bioconductor. For information on installing bioconductor packages, see here.
The first step to making a treesiftr visualization is to select the subset of the phylogenetic matrix that we would like to visualize. This is performed via a function called generate_sliding
. The below command will subset the
# Locate package data and read alignment
fdir <- system.file("extdata", package = "treesiftr")
aln_path <- file.path(fdir, "bears_fasta.fa")
bears <- read_alignment(aln_path)
# Read in a starting phylogenetic tree
tree <- read.tree(file.path(fdir, "starting_tree.tre"))
# Generate our list of dataframe subsets
sample_df <- generate_sliding(bears, start_char = 1, stop_char = 5, steps = 1)
The result of this is a dataframe, shown below:
## starting_val stop_val step_val
## 1 1 2 1
## 2 2 3 1
## 3 3 4 1
## 4 4 5 1
## 5 5 6 1
This dataframe dispays the start character (the first character that will be visualized) and stop character (the final character that will be visualized).
We can then build trees from each subset:
output_vector <- generate_tree_vis(sample_df = sample_df, alignment = aln_path, tree = tree, phy_mat = bears)
## Final p-score 2 after 0 nni operations
## Final p-score 2 after 0 nni operations
## Final p-score 2 after 0 nni operations
## Final p-score 2 after 1 nni operations
## Final p-score 2 after 1 nni operations
The above code saves the trees to a vector, but does not visualize them. They can be viauslized by naming the vector, like so:
## [[1]]
## NULL
##
## [[2]]
## NULL
##
## [[3]]
## NULL
##
## [[4]]
## NULL
##
## [[5]]
Phangorn
(Schliep 2011, Schliep2017) requires a starting tree to estimate a parsimony tree. We specify the tree we read in earlier for this purpose. The trees, which were generated with ggtree
(Yu et al. 2017) have been saved to a vector, which can be displayed in its entirety, or subsetted to look at specific trees.
pscore
argument:output_vector <- generate_tree_vis(sample_df = sample_df, alignment = aln_path, tree = tree, phy_mat = bears,
pscore = TRUE)
## Final p-score 2 after 0 nni operations
## Final p-score 2 after 0 nni operations
## Final p-score 2 after 0 nni operations
## Final p-score 2 after 1 nni operations
## Final p-score 2 after 1 nni operations
## [[1]]
## NULL
##
## [[2]]
## NULL
##
## [[3]]
## NULL
##
## [[4]]
## NULL
##
## [[5]]
Instructor’s Note: The answer is two, as viewed in the first visualization in the vector.
Instructor’s Note: They should be larger. Parsimony estimation will favor the shortest tree while no such criteria is applied to a random tree. This can be checked like so:
sample_df <- generate_sliding(bears, start_char = 31, stop_char = 32, steps = 3)
output_vector <- generate_tree_vis(sample_df = sample_df, alignment = aln_path, tree = tree, phy_mat = bears,
pscore = TRUE, random_tree = TRUE)
## [1] 20
## [1] 18
## [[1]]
## NULL
Instructor’s Note: The answer is Ursavus brevirhinus and Ballusia elmensis. This clade is broken up on the tree for characters 3-4, in which Ballusia elmensis is sister to Zaragocyon_daamsi.
Instructor’s Note: The root of the tree likely blue, with a change to brown on the branch leading to Tremarctos, Melursus, Helarctos, and Ursus. Ursus thibetanus posesses the ancestral blue state, and is the best candidate for reversal. This can be verified like so:
sample_df <- generate_sliding(bears, start_char = 8, stop_char = 9, steps = 2)
output_vector <- generate_tree_vis(sample_df = sample_df, alignment = aln_path, tree = tree, phy_mat = bears,
pscore = TRUE)
## Final p-score 4 after 0 nni operations
## Final p-score 4 after 1 nni operations
## [[1]]
## NULL
Instructor’s Note: To make this decision, we would want to know if there are other bears that are not sampled on this tree that posess this character. We can view the tree like so:
sample_df <- generate_sliding(bears, start_char = 52, stop_char = 53, steps = 1)
output_vector <- generate_tree_vis(sample_df = sample_df, alignment = aln_path, tree = tree, phy_mat = bears,
pscore = TRUE)
## Final p-score 2 after 0 nni operations
## Final p-score 2 after 0 nni operations
## [[1]]
## NULL
output_vector <- generate_tree_vis(sample_df = sample_df, alignment = aln_path, tree = tree, phy_mat = bears,
pscore = TRUE, lscore = TRUE)
## Final p-score 2 after 0 nni operations
## Final p-score 2 after 0 nni operations
## [[1]]
## NULL
##
## [[2]]
Instructor’s Notes: It is possible for characters with the same number of parsimony steps to have different likelihood scores.
Instructor’s Notes: Answers may vary. But generally, a character (character 46) that implies changes on a fairly short branch is replaced with one implying changes on longer branches.
sample_df <- generate_sliding(bears, start_char = 46, stop_char = 49, steps = 3)
output_vector <- generate_tree_vis(sample_df = sample_df, alignment = aln_path, tree = tree, phy_mat = bears,
pscore = TRUE, lscore = TRUE)
## Final p-score 7 after 0 nni operations
## Final p-score 7 after 0 nni operations
## Final p-score 5 after 1 nni operations
## Final p-score 5 after 1 nni operations
## [[1]]
## NULL
##
## [[2]]
## NULL
##
## [[3]]
## NULL
##
## [[4]]
What is the relationship between the likelihood score and increasing the number of characters visualized? > Instructor’s Notes: The likelihood should become more negative (less likely)
What is the minimum number adding a parsimony-informative character can add to the parsimony score?
Instructor’s Notes: 1
Instructor’s Notes: For some trees, we can see that there is no character data supporting the grouping. Parsimony software generally displays fully-resolved trees - this may be misleading if little to no data support the grouping! In our case, we’re seeing strong influence of the starting tree.
Ancestral State: A character state possessed by the ancestor of a group
Autapomorphy: A character state that is unique to a specific taxon.
Derived State: A character state that is different from the ancestral state.
Likelihood Score: The likelihood of the observed data under a specific model.
Maximum likelihood: A phylogenetic optimatlity criterion under which phylogenetic data are modeled according to sets of assumptions. Under this criterion, the tree that has the best (“maximum”) likelihood score under the assumed model is to be preferred.
Maximum parsimony: A phylogenetic optimality criterion. This criterion holds that the tree implying the fewest changes in the characters used to generate it should be preferred.
Monophyletic: A group on a phylogeny of an ancestor and all of its descendents.
Parsimony Score: The number of changes implied by a character on a tree.
Resolved: A node that is bifucating, leaving two descendent lineages.
Reversal: A change from the derived state back to the ancestral state.
Baum, David A., Stacey DeWitt Smith, and Samuel S. S. Donovan. 2005. “The Tree-Thinking Challenge.” Science 310 (5750). American Association for the Advancement of Science: 979–80. https://doi.org/10.1126/science.1117727.
Lewis, Paul O. 2001. “A Likelihood Approach to Estimating Phylogeny from Discrete Morphological Character Data.” Systematic Biology 50 (6). Oxford University Press: 913–25. https://doi.org/10.1080/106351501753462876.
R Core Team. 2018. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Schliep, K.P. 2011. “Phangorn: Phylogenetic Analysis in R.” Bioinformatics 27 (4): 592–93. https://doi.org/10.1093/bioinformatics/btq706.
Wickham, Hadley. 2016. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. http://ggplot2.org.
Yu, Guangchuang, David Smith, Huachen Zhu, Yi Guan, and Tommy Tsan-Yuk Lam. 2017. “Ggtree: An R Package for Visualization and Annotation of Phylogenetic Trees with Their Covariates and Other Associated Data.” Methods in Ecology and Evolution 8 (1): 28–36. https://doi.org/10.1111/2041-210X.12628.


{% extends "header.html" %} {% block body %}
If you happen to be plotting data that has a lot of data points, you will probably find that loading time is fairly extensive. A sample set of data can be found here, though you can also feel free to use your own data.
''' Sample data is located: http://sentdex.com/GBPUSD.zip ''' import matplotlib import matplotlib.pyplot as plt import matplotlib.ticker as mticker import matplotlib.dates as mdates import numpy as np def graphRawFX(): date,bid,ask = np.loadtxt('GBPUSD1d.txt', unpack=True, delimiter=',', converters={0:mdates.strpdate2num('%Y%m%d%H%M%S')}) fig=plt.figure(figsize=(10,7)) ax1 = plt.subplot2grid((40,40), (0,0), rowspan=40, colspan=40) ax1.plot(date,bid) ax1.plot(date,ask) ax1.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m-%d %H:%M:%S')) for label in ax1.xaxis.get_ticklabels(): label.set_rotation(45) plt.subplots_adjust(bottom=.23) plt.gca().get_yaxis().get_major_formatter().set_useOffset(False) plt.grid(True) plt.show()
Here are some example photos of the visible changes by changing granularity:
No change in data granularity before rendering, showing 62,000 plots:
This chart shows the full 62,000 plots, with no change made to the granularity.
1/5th granularity, cutting the data points from 62,000 plots to 12,400 plots:
Aside from the thickness of the lines, the actual trajectory is left unaffected.
This image shows one fifth data granularity, from 62,000 plots to 12,400 plots. No noticeable change for the eyes.
1/15 granularity, from 62,000 plots to just ~4,100 plots. Some noticeable change, but trajectory still very much in tact
Now we can begin to notice the difference, but the meaning of the data is certainly not lost, yet the data size is MUCH smaller


Event
Add
About
Event Add
Add
Event
Save
Event View
Add
Edit
Delete
Event Edit
Add
Event
Save
About
This is a RESTful Event Application using jQuery Mobile, Backbone.js, Node.js, MongoDB, mongoose and express.


--- layout: default footerlogos: yes title: "Speaker" lang: en ---
Milena has over eight years experience working at the intersection of technology, data and social good. She is currently working with Amnesty International, leading Amnesty Decoders - an innovative project engaging digital volunteers in documenting human right violations using new technologies. Previously she worked as programme manager of School of Data where she trained and mentored numerous NGOs and journalists around the world to make the most of their data and reach new audiences. She also worked for over 4 years with Transparency International where she supported TI’s global network to use technology in the fight against corruption.

Program for the first day


One keyword per line. Posts containing at least one keyword will be filtered out.
Case sensitive
Hide Top Growing Communities
Save


Management Team
Founder & Managing Partner
- Over 30 years of health care experience & 10 years of VC experience
- Extensive experience working with top thought leaders
- Successfully launched and managed pharmaceutical products at Novartis & Shering-Plough
CFO
- Over 30 years of health care experience with pre and public companies
- Former CFO Restech and Resmed
Chief Design Officer
- CEO & Founder Wright Designs, Eclipse
- Pioneering innovations in the health industry


Awo’s vision celebrated at the 60th anniversary of the first TV station in Africa on the 31st of October, 2019’
Awo’s vision celebrated at the 60th anniversary of the first TV station in Africa on the 31st of October, 2019’
Awo’s vision celebrated at the 60th anniversary of the first TV station in Africa on the 31st of October, 2019’


---
layout: post
title: 7/16 progress
date: '2014-07-16T21:55:00.000-06:00'
author: Jacob .
tags:
- progress-reports
modified_time: '2014-07-16T21:55:25.453-06:00'
blogger_id: tag:blogger.com,1999:blog-6400085873105072120.post-6803268952645762990
blogger_orig_url: http://www.svlark.com/2014/07/716-progress.html
---
Another night, another room. We are spending tonight in the 12th place since leaving the apartment in Superior. We're having fun now!
Louisville Best Western
Salina Super 8
Bolivar Country Inn
Horn Lake Sleep Inn
New Orleans Sonesta
Gulf Breeze Comfort Inn
Apalachicola Appalachicola River Inn
Haines City Days Inn
Melbourne Harbor Condominiums
Seahorse Beach Bungalows
Ocean Village 411
Ocean Village 3311
We heard that there's a catamaran at Harbortown with 5 kids.


Definitions
What is Accessible Name?
Accessible Name is the name of an HTML element which will be spoken by screen readers and used by Assistive Technologies. The accessible name is the result of the accessible name computation based on the element's contents, HTML attributes, ARIA attributes, or programmatically associated elements. ANDI, the accessibility testing tool, calculates the accessible name and includes it in the ANDI output.
What is Accessible Description?
Accessible Description is additional text related to an HTML element which will be spoken by screen readers and used by Assistive Technologies. The accessible description is the result of the accessible name computation based on the element's HTML attributes and ARIA attributes. ANDI, the accessibility testing tool, calculates the accessible description and includes it in the ANDI output.
Troubleshooting
Why is my favorites/bookmarks bar hidden?
If the browser toolbar or favorites/bookmarks menu is hidden, try this: Press F10, then select the bookmarks/favorites option to show bookmarks/favorites bar.
If the favorites bar is disabled by the web application...
- If using IE, press CTRL + N to open a new window at the current (hidden) url.
- If that doesn't work, if using IE, get the current URL and paste into a new window:
- Right click on the page or press Shift + F10
- Select "Properties"
- Click next to Address (URL), press CTRL + A to select the full url
- CTRL + C to copy the url
- Open a new browser window paste the url into the address field, press enter
- Otherwise, ask the web application's development team to enable the toolbar.
Why won't ANDI work on this page?
If after pressing the ANDI favelet button, ANDI does not launch after a few seconds, it could be due to a few things:
-
It could be due to the page telling the browser to enforce a Content Security Policy directive.
To determine if this is the issue, open the browser's Developer Tools (F12)
and attempt to launch ANDI.
If the Dev Tools console shows an error message that says
"Refused to load the script...because it violates the following Content Security Policy directive"
then this is the issue.
To use ANDI on this test page, make a request to the page owner to add the ANDI script to the whitelist of approved scripts.
If the user desires to use ANDI immediately, Content Security Policy (CSP) can be disabled. Note: It is the user's decision to disable CSP and the user's responsibility to re-enable CSP when testing with ANDI has concluded. If users decide not to disable CSP, and ANDI cannot be launched, it is recommended to use other accessibility testing procedures. - There could be a JavaScript error occurring. Open the browser's Developer Tools (F12) and attempt to launch ANDI. If the Dev Tools console shows a JavaScript error that relates to ANDI, then send a link to the test page to ANDI's development team by creating an issue on the GitHub page.
-
Your browser or organization could be preventing scripts (such as ANDI) from launching from a favorite/bookmark.
To determine if this is the case, try this:
- Drag and Drop this link: Test Favelet into your browser's favorites/bookmark bar.
- Activate the favorite/bookmark you just created.
- If you don't see an alert pop up, then your browser is blocking JavaScript in favorites/bookmarks, in which case you will not be able to use ANDI in this browser.
- The test page may be very large and have many elements which takes ANDI a longer than normal time to analyze. Try waiting a little longer.
Why won't ANDI find an element?
If ANDI has been launched and something is not "inspectable," it could be due to one of these reasons:
- The page content may have changed after ANDI was launched, therefore ANDI is not aware of the changed content. Try refreshing ANDI.
- A different module may need to be selected to detect the element.
-
The test page's CSS may be disguising the element.
For example, an element may look like a button,
but it's not actually a focusable, keyboard accessible button.
ANDI (focusable elements) would not recognize this element
if it wasn't built using a semantic button tag, or doesn't have
tabindex
. - If mouse hover is not able to inspect an element, try the Next or Previous Element buttons.
Can I use ANDI for mobile accessibility testing?
ANDI does not run on mobile browsers. However, ANDI can be used test accessibility on small screen sizes.
Mobile testing can be simulated in a desktop browser using developer tools (F12) with the Toggle Device Toolbar feature (Ctrl + Shift + M). This feature will adjust the screen resolution to that of a mobile or tablet device. Once this mode is engaged, ANDI can be used to test accessibility on smaller screen sizes.
Why is ANDI's Output different from the screen reader's?
ANDI's Output may occasionally be different from a screen reader's output. Here are some reasons why the Output may differ:
- ANDI, the accessibility testing tool, uses the DOM (Document Object Model) to calculate the Output according to the Accesible Name Computation. Whereas, screen readers rely on more layers than just the DOM. Layers such as the browser's Accessibility API, the accessibility tree, the DOM, their own "tutor text" phrasings.
- Some screen reader software has guessing algorithms to assist its users. Guessing makes up for missing or improper accessibility coding. However, screen readers should not be guessing when explicit accessibility coding is present. Since ANDI is an accessibility testing tool, it doesn't guess. Instead, its Output is a direct result of programmatic associations.
- When humans write complex software (such as screen readers and accessibility testing tools), bugs happen. Let's squash them! If you are think there is a bug the ANDI Output, please reach out to our development team by creating a github issue.
Why is ANDI covering up the test page?
Since ANDI dynamically adjusts to the data being displayed for a particular element, mouse hovering can sometimes cause ANDI to overlap with the Test Page.
If ANDI is continually blocking the view of the Test Page, use the next previous element buttons which should always keep ANDI out of the way. You can also try Mini Mode.
If ANDI is still blocking the test page, it is usually due to the responsive design of the test page reacting to the space that ANDI is filling. Try maximizing the browser window, or switching to a larger resolution.
Why is ANDI making the test page look different?
If after launching ANDI, the page looks odd or different, you may have to bear with it for your testing. ANDI has to manipulate some CSS in order allow itself to appear on the page - This can sometimes cause display interference.
If elements become obscured to the point that you can't test the content behind "floating" elements, you may have to engage ANDI's advanced setting, linearize page.
Where is cANDI (ANDI color contrast)?
The c
ANDI module relies upon functionality
that is not available in versions of Internet Explorer prior to 9
(including compatibility modes).
Testers will need to perform a manual contrast test when
c
ANDI is not available.
Where are ANDI's lasers?
ANDI's "lasers" will not work in versions of Internet Explorer prior to 9 (including compatibility modes) because ANDI's "lasers" are built using SVG (scalable vector graphics) which are not supported by older IE versions.
Other Questions
How do I host my own ANDI?
To host your own copy of ANDI, make a few simple changes in the andi.js file:
- Download ANDI's source code from GitHub.
- Determine if you are going to host ANDI on a server (
https://[ALTERNATE_HOST]/andi/
) or locally (file://C:/WS/repos/[ALTERNATE_HOST]/andi/
). - In the andi.js file:
- modify the
host_url
to point to the location that you decided in step 2. - (optional) modify the
jqueryDownloadSource
location to a custom jquery location.
- modify the
- If you are hosting on a server, upload the modified ANDI source code to the server that you decided in step 2. If you are running locally, skip this step.
- Create a new bookmark called "my ANDI" or whatever you choose to name it.
- Modify the bookmark's launch script to point to your instance of ANDI:
Contact ANDI's Development Team
Where do I file an ANDI bug?
Uh oh! You found a bug in ANDI? Please create an issue on the GitHub page. Be sure to provide a link to a page with the issue, sample code, or embedded screenshots.
I have an ANDI feature request!
That's wonderful! If you have an idea for how to make ANDI better, please create an issue on the GitHub page and explain away.


JavaScript Variable Scope and Hoisting
September 06, 2015Variable Scope
Definition: A variable’s scope is the context in which the variable exists. The scope specifies from where you can access a variable and whether you have access to the variable in that context. Variables have either a local scope or a global scope.
Local Variables (Function-level scope) Variables declared within a function are local variables and are only accessible within that function or by functions inside that function.
-
No Block-Level Scope(variables scoped to surrounding curly brackets) example: variable in function without “var” is to be a global variable.
-
Local Variables Have Priority Over Global Variables in Functions If you declare a global variable and a local variable with the same name, the local variable will have priority when you attempt to use the variable inside a function (local scope)
var name = "Paul"; function users () { // Here, the name variable is local and it takes precedence over the same name variable in the global scope var name = "Jack"; // The search for name starts right here inside the function before it attempts to look outside the function in the global scope console.log (name); } users (); // Jack
Global Variables All variables declared outside a function are in the global scope. In the browser, which is what we are concerned with as front-end developers, the global context or scope is the window object (or the entire HTML document).
- If a variable is initialized (assigned a value) without first being declared with the var keyword, it is automatically added to the global context and it is thus a global variable
Pay attention to i:
for (var i = 1; i <= 10; i++) {
console.log (i); // outputs 1, 2, 3, 4, 5, 6, 7, 8, 9, 10;
};
// The variable i is a global variable and it is accessible in the following function with the last value it was assigned above
function aNumber () {
console.log(i);
}
// The variable i in the aNumber function below is the global variable i that was changed in the for loop above. Its last value was 11, set just before the for loop exited:
aNumber (); // 11
- setTimeout Variables are Executed in the Global Scope Note that all functions in setTimeout are executed in the global scope. This is a tricky bit; consider this:
// The use of the “this” object inside the setTimeout function refers to the Window object, not to myObj
var highValue = 200;
var constantVal = 2;
var myObj = {
highValue: 20,
constantVal: 5,
calculateIt: function () {
setTimeout (function () {
console.log(this.constantVal * this.highValue);
}, 2000);
}
}
// The "this" object in the setTimeout function used the global highValue and constantVal variables, because the reference to "this" in the setTimeout function refers to the global window object, not to the myObj object as we might expect.
myObj.calculateIt(); // 400
// This is an important point to remember.
Do not Pollute the Global Scope you have to know that it is important to avoid creating many variables in the global scope, such as this:
// These two variables are in the global scope and they shouldn't be here
var firstName, lastName;
function fullName () {
console.log ("Full Name: " + firstName + " " + lastName );
}
This is the improved code and the proper way to avoid polluting the global scope
// Declare the variables inside the function where they are local variables
function fullName () {
var firstName = "Michael", lastName = "Jackson";
console.log ("Full Name: " + firstName + " " + lastName );
}
Variable Hoisting All variable declarations are hoisted (lifted and declared) to the top of the function, if defined in a function, or the top of the global context, if outside a function.
It is important to know that only variable declarations are hoisted to the top, not variable initialization or assignments (when the variable is assigned a value).
Variable Hoisting Example:
function showName () {
console.log ("First Name: " + name);
var name = "Ford";
console.log ("Last Name: " + name);
}
showName ();
// First Name: undefined
// Last Name: Ford
// The reason undefined prints first is because the local variable name was hoisted to the top of the function
// Which means it is this local variable that get calls the first time.
// This is how the code is actually processed by the JavaScript engine:
function showName () {
var name; // name is hoisted (note that is undefined at this point, since the assignment happens below)
console.log ("First Name: " + name); // First Name: undefined
name = "Ford"; // name is assigned a value
// now name is Ford
console.log ("Last Name: " + name); // Last Name: Ford
}
Function Declaration Overrides Variable Declaration When Hoisted
Both function declaration and variable declarations are hoisted to the top of the containing scope.
And function declaration takes precedence over variable declarations (but not over variable assignment).
As is noted above, variable assignment is not hoisted, and neither is function assignment. As a reminder, this is a function assignment: var myFunction = function () {}.
Here is a basic example to demonstrate:
// Both the variable and the function are named myName
var myName;
function myName () {
console.log ("Rich");
}
// The function declaration overrides the variable name
console.log(typeof myName); // function ------
// But in this example, the variable assignment overrides the function declaration
var myName = "Richard"; // This is the variable assignment (initialization) that overrides the function declaration.
function myName () {
console.log ("Rich");
}
console.log(typeof myName); // string
It is important to note that function expressions
, such as the example below, are not hoisted
.
var myName = function () {
console.log ("Rich");
}
In strict mode
, an error will occur if you assign a variable a value without first declaring the variable. Always declare your variables.


ZikaV HIV virus Zika â€" HIV/AIDS []. health risk {border-style global spread px;} viral #ffffff;} epidemic infection transmission AIDS HIV- Global Medicine, United disease diseases Aedes Fig. Figure blood cases Disease Virus mutation pandemic virology Control Health Prevention States University [], drug including infected investigating public px; related sexual {font-family children font-size human identifying increased medical people use Publications YorkSpringer clinical diseasesNew double; emergence modes pathway potential reported symptoms viruses years .Centers Available Florida, HIV, In: Moreover, New Nov Organization School South ZikaV. [Cited [Internet] approach border-color border-width brain from: genome major microcephaly, new rapid rate solid; vaccines virulent {font-style {font-weight Diseases Health, However, Infectious Internal International Publications, Shapshak Somboonwit Springer Transmission World ZikaV, additional adults albopictus and/or arbovirus associated bold care case cause cells collaborative countries courier; current detected diagnosed different difficult efforts em; evolved factors fear fever, globally hemorrhagic highly immune impact indicated infection. infections injection living man molecular mosquitoes mothers occurred proliferation pt;} report research sex transmitted tree virus,


I design things, I make them, I reinvent them and teach them. #DIY, #research, #IoT, #embedded, #RaspberryPi, #Arduino, #EEE. I have been working with electronics for over two decades and have helped establish research labs and products for industrial automation in India. I have a passion to create something new and improve the existing world and technology is my tool. I offer the following commercial services. Product Design: From Idea to concept to production, I can help you realize your dream project. Growth Hacking: Use the in ternet to your advantage. I can help you get to your target audience. Training: Add skills to your organization. I can help introduce new technology and workflows to your existing team. My blog is available at The Embedded Code
In case you want me to review your product or have a project in mind that you would like my help or you need a demo project on your product, I request you to contact me on twitter since it is the quickest way for me to respond. Alternatively, there is a contact segment on my blog which allows you to send me an email and I will get back to you when I can.
Here is a list of Skills I have accumulated over time.


Contributing to Lookyloo Documentation
You can contribute by:
-
Raising an issue you find in the documentation
-
Fixing issues by opening a pull request
-
Improving documentation
Contributor Behavior
Examples of behavior that contributes to creating a positive environment include:
-
Using welcoming and inclusive language
-
Focusing on what is best for the community
-
Being respectful of differing viewpoints and experiences
-
Gracefully accepting constructive criticism
Before You Begin
-
Get started by installing and getting familiar with Git.
-
Use AsciiDoc to contribute to Lookyloo documentation.
-
Refer to our style guide for guidance on formatting your documentation contributions.
General Information
Thank you for your contributions. To start contributing to Lookyloo:
-
Create an issue describing your proposed change to the repository.
-
The repository owners will triage and respond to your issue promptly.
-
Fork the repository.
-
Submit a pull request with the proposed changes. The required format for the Git commit message is "Issue # <Issue ID> - <issue_message>".
Raising an Issue
GitHub issues can be used to report bugs or submit feature requests.
When reporting an issue, you can use the issues template.
Updating Documentation
The documentation is written in the Asciidoc markup language and uses Antora.
The source documentation files are located in the modules/ROOT/pages/
directory in the Lookyloo documentation repository.
Procedure
To update existing documentation:
-
Checkout the branch you want to update, for example:
git checkout main
-
Ensure that your forked repository is in sync with the selected upstream repository:
git fetch <lookyloo/docs>
git reset --hard <lookyloo/docs> main
-
Checkout a new feature branch with the <Issue #> as the name, for example:
git checkout -b Issue#123
-
Make changes to the documentation.
To add a new module to the repository, add the module to the
/modules/ROOT/nav.adoc
file in the following format:* xref:<module_filename.adoc>[<module title>]
.The number of asterisks (*) denotes the heading level.
-
Preview the documentation on the branch.
-
Push the changes to your fork’s feature branch.
-
Submit a pull request against to the dsignated repository.


Reports any uses of java.util.Date, java.util.Calendar, java.util.GregorianCalendar, java.util.TimeZone,
and java.util.SimpleTimeZone.
While still supported, these classes were made obsolete by the JDK8 Date-Time API, and should probably
not be used in new development.


Fill the radius in the text field:
(or with
+1
A disk with radius {{ radius }}
has an area of {{ area }}.
A ball with the same radius would have a volume of {{ area * radius * 4 / 3 }}


JsUnit Asynchronous Load Tests
This page tests loading data documents asynchronously. To see them, take a look at the source.


Otter Grader Test Generator
View on GitHub
Scored
Visible
Cases
Hidden
Locked
Delete Test
Add a Test
Generate


You can contact me on Discord, Kongregate, Reddit (i'm Fariz) or Github. For bug reporting opening an issue on Github is optimal. If you want to make a donation to support the development you can use PayPal.


Major Features |
Phase One |
Phase Two |
Phase Three |
|---|---|---|---|
Hardware |
|||
Number of servers |
100 | 100 | 300 |
Networking |
3x10 Gbps (OpenFlow enabled, 3 SDN switches) | 3x10 Gbps (OpenFlow enabled, 3 SDN switches) | 3x10 Gbps (OpenFlow enabled, 9 SDN switches) |
|
Core Software
|
|||
VM & Bare Metal Machines Provisioning |
|||
| * | * | * | |
Support of Configuration Management System |
- | Ansible | Ansible, Puppet, etc. |
Support of Container Technology |
- | Docker | Docker |
| - | * | ||
Web built-in Console
|
# | # | |
Data & Data Repository |
|||
Database |
Up to 3 (e.g. Rainbow table for GSM A5/1, Click-flaw data, mobile anti-virus apps) | Up to 5 (additional: Alexa top 1,000,000 websites, Internet traffic) | Up to 10 (including data for network, malware, web, mobile and human behavior etc) |
Access to External Databases |
US IMPACT | US IMPACT | US IMPACT |
Seamless Integration with NCL Testbed |
(uploading and downloading only | (uploading and downloading only | (direct access within experiment nodes) |
| - | - | - | |
Services and Tools |
|||
GUI to Visualize Network Topology |
|||
GUI to Configure Network Topology |
- | - | |
Billing & Cost Management Tool |
|||
Monitoring & Logging Tool |
|||
Usage & Resource Auditing Tool |
|||
Network Traffic Generator
|
Open-source software | Software and high-end dedicated hardware | Software and high-end dedicated hardware |
| - | |||
VM-based Router, Firewall |
- | ||
Support of hot plugging of external devices |
- | - | |
1-Click Ready-to-use experiment setups |
|||
System vulnerability environments |
Up to 20 (well-known ones like Heartbleed, shellshock etc) | Up to 20 (include main vulnerability types like memory corruption, data corruption, crypto implementation flaws) | Up to 100 (include most of the vulnerability types) |
Big-Data Platforms |
Hadoop, Spark | Additional: Storm, HBase, HIVE | Additional: H2O, Flink, etc |
| Bitcoin | Bitcoin and possible others | Bitcoin and possible others | |
| - | 1 simple example (e.g. a test lab) | More complex example (e.g. an organization) | |
Cyber-physical system simulators |
- | - | Smart grid simulator |
Emulated android environment |
- | ||
Emulated ARM environment |
- | - | |
Teaching-ready environments |
- | Up to 5 | Up to 10 |
Security |
|||
| Isolated (no outgiong Internet access ) | Controlled | Controlled | |
Federation |
|||
NUS Sensor Network Testbed (Indriya) |
|||
SUTD Water Testbed |
- | - |
*Feature(s) only available in bare metal provisioning
#Feature(s) only available in VM provisioning


Kids play with smart toys and learn how to code
The toy can understand your kid, can find and fetch things, and play tricks
When the toy is changed the code is updated.
Please us where to send info about toy availability
Help us improve your product with by participating in a quick survey.


Couplette
Profile Info
Partner 1 Info
Partner 2 Info
Location
Couple Gender(s)
Male/Female
Male/Male
Female/Female
Tell Us About Yourselves!
e.g. how did you two meet?...
What Activities Do You Enjoy? (check all that apply)
Arts
Dining
Films
Music
Gaming
Outdoor
Travel
Sports
Sign Up


Q Sentral, KL
RM5,000-RM7,000
-
Responsibilities:
-
Work closely with the rest of the software engineers in the team to design, build, test, deploy, maintain and continuously improve the company's software solutions, including the core mobile point-of-sales (MPOS) system. Additionally, contribute to new product research projects, which may include research and the building of proof-of-concepts.
-
Work with the company's clients in various countries to understand their requirements and help build customizations to the company's software solutions that will meet their needs.
-
Mentor junior software engineers and provide guidance to them when needed.
-
Required:
- Experienced in developing iOS or Android applications.
-
Ideally at least 4 years of prior iOS or Android development experience
-
Design / develop reusable frameworks, components, and services.
-
Knowledge and passion on Mobile Technology and Architecture
-
Great to have:
-
Extensive knowledge in SWIFT / C++ / RESTful API / MVC
-
Knowledge and understanding of RFID / NFC / Bluetooth
-
Knowledge on hardware programming
-
Knowledge of payment domain requirements and standards (EMV, PCI for example)
-
Added Advantage:
-
Great understanding of software design (patterns) and debugging skills (using Xcode, Eclipse, Android Studio, etc).
-
Good communication skills, that is you can probably get along great with our clients, and us.
-
Above all, great attitude and a deep desire to constantly learn and improve.


--- title: "What is the expected value on a scale-location plot?" author: "Drew Tyre" date: "2020-09-03" output: html_document: default og_image: /post/scale-location_files/figure-html/featured_image-1.png draft: no slug: scale-location tags_include: - R - Statistics categories: Teaching ---
I teach my students to check the assumptions of their models by making various diagnostic plots of residuals. One of the niftiest is the scale-location plot, which is useful for diagnosing changes in variance across the range of the model. If all’s well, a smooth line on that plot is flat. But how flat is flat?
The problem is that real data is never “flat” even if all the assumptions of a model are met. Any smooth line fitted to the residuals will bump up and down a bit. Take a look at the classic choice of example dataset for demonstrating this plot, the cars data.
library("tidyverse")
library("broom")
data("cars")
cars.lm <- lm(dist ~ speed, data = cars)
dist_speed <- ggplot(data = cars,
mapping = aes(x = speed, y = dist)) +
geom_point()
dist_speed + geom_smooth(method = "lm")
A scale-location plot puts the fitted values of the model (blue line) on the x axis, and the square root of the absolute value of the standardized residuals on the y axis. Say that last three times fast, I dare you.
diag_data <- augment(cars.lm)
diag_data <- mutate(diag_data,
sqrabs_resid = sqrt(abs(.std.resid)))
ggplot(data = diag_data,
mapping = aes(x = .fitted, y = sqrabs_resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 1) +
labs(x = "Distance [feet]",
y = expression(sqrt(abs(epsilon))))
I like using ggplot to make these plots just for the ability to put confidence limits on that smooth line. This helps distinguish what is flat, and what is not really flat. To aid that interpretation, I’ve always told students that the expected value of the square root of the absolute value of the standardized residuals is 1. By that measure, the model of stopping distance vs. speed is showing a bit less variance than expected at low speed, and about right at higher speeds.
All was right with the world until earlier this week a student asked why the expected value was 1. Um. My brain coughed up an answer having to do with the variance of the standardized residuals being 1, but even as I said it I realized I hadn’t a clue. Searching around online I could find lots of people saying the plot should be flat, but no indication of what the value ought to be.
So what should it be? A couple quick simulations showed that it certainly isn’t 1!
# standardized residuals should be N(0,1), get 100
mean(sqrt(abs(rnorm(100)))) # run this as often as you like, it's never 1
## [1] 0.855145
Stumped. Then I thought maybe the residuals are t distributed!
mean(sqrt(abs(rt(100, df = 2)))) # try different values of df.
## [1] 1.092028
So if the degrees of freedom are 2, a t distribution works. But as the degrees of freedom increase the expected value drops fast. This begs the question, why 2 degrees of freedom as the magic number?
This morning I realized I should be able to calculate the expected value using numeric integration. I’m sure someone could do it with pencil and paper but I’m not that smart. The expected value of a random variable is the integral of the value times the probability of getting that value.
\[ \begin{align} E\left(\sqrt{\left|\epsilon\right|}\right) = & \int_{-\infty}^\infty p(\epsilon) \sqrt{\left|\epsilon\right|} \, d\epsilon \\ = & \int_0^\infty 2 \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\epsilon}\, \sqrt{\epsilon} \, d\epsilon \end{align} \]
Now that I write out the equation it doesn’t look as bad because we’re using the standard normal with mean 0 and variance 1. But still.
integrand <- function(x){
2 * dnorm(x) * sqrt(x)
}
integrate(integrand, 0, Inf)
## 0.822179 with absolute error < 9.5e-05
What if we use that value in our scale-location plot?
ggplot(data = diag_data,
mapping = aes(x = .fitted, y = sqrabs_resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 0.822179) +
labs(x = "Speed [mph]",
y = expression(sqrt(abs(epsilon))))
OK! That’s actually really nice. Although the expected value is everywhere inside the confidence envelope on the smooth curve, now that the line is in the middle what we’re seeing looks more like an increase in variance everywhere, instead of an abrupt bump up at ~ 30 mph. Perception is very sensitive to these guides. What does it look like if we simulate some data where the assumptions are perfectly met?
N = nrow(cars)
fakedata1 <- tibble(speed = runif(N, 0, 25),
dist = coef(cars.lm)[1] + coef(cars.lm)[2]*speed +
rnorm(N, sd = summary(cars.lm)$sigma))
fakemod1 <- lm(dist ~ speed, data = fakedata1)
dist_speed %+% fakedata1 + geom_smooth(method = "lm")
Alright … well, ignoring the obvious problem that cars stopping at low speeds don’t go backwards, how do these residuals look?
diag_fake <- augment(fakemod1)
diag_fake <- mutate(diag_fake,
sqrabs_resid = sqrt(abs(.std.resid)))
ggplot(data = diag_fake,
mapping = aes(x = .fitted, y = sqrabs_resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 0.822179) +
labs(x = "Distance [feet]",
y = expression(sqrt(abs(epsilon))))
And that is very nice.
Cars don’t go backwards. Looking at the intercept of the first fitted model -17.5790949, a car that is stopped should go backwards nearly 18 feet. Clearly the intercept should be zero. But is the relationship between stopping distance and speed linear? To answer that we need to look at a plot of the residuals vs. speed.
ggplot(data = diag_data,
mapping = aes(x = speed, y = .resid)) +
geom_point() +
geom_smooth()
The increase in variance is a little bit apparent in this plot too. The zero line is everywhere inside the confidence envelope, although there is a hint that the residuals are higher at the ends, which would indicate a non-linear relationship. What if we fix the intercept to be zero, as we know it should be based on simple physics!
cars.lm2 <- lm(dist ~ speed + 0, data = cars)
summary(cars.lm2)
##
## Call:
## lm(formula = dist ~ speed + 0, data = cars)
##
## Residuals:
## Min 1Q Median 3Q Max
## -26.183 -12.637 -5.455 4.590 50.181
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## speed 2.9091 0.1414 20.58 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 16.26 on 49 degrees of freedom
## Multiple R-squared: 0.8963, Adjusted R-squared: 0.8942
## F-statistic: 423.5 on 1 and 49 DF, p-value: < 2.2e-16
Wow, that’s interesting, this is one of those rare occasions when reducing the number of parameters increases the fit of the model! What about the diagnostic plots?
diag_data2 <- augment(cars.lm2) %>%
mutate(sqrabs_resid = sqrt(abs(.std.resid)))
sl_plot <- ggplot(data = diag_data2,
mapping = aes(x = .fitted, y = sqrabs_resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 0.822179) +
labs(x = "distance [feet]",
y = expression(sqrt(abs(epsilon))))
nl_plot <- ggplot(data = diag_data2,
mapping = aes(x = speed, y = .resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 0) +
labs(x = "Speed [mph]",
y = expression(epsilon))
egg::ggarrange(sl_plot, nl_plot, nrow=1)
So now the failure of the linearity assumption is clear in the right hand plot. There is a hint of an increase in the variance, but still not significant. What about fitting a quadratic model?
cars.lm3 <- lm(dist ~ I(speed^2) + speed + 0, data = cars)
summary(cars.lm3)
##
## Call:
## lm(formula = dist ~ I(speed^2) + speed + 0, data = cars)
##
## Residuals:
## Min 1Q Median 3Q Max
## -28.836 -9.071 -3.152 4.570 44.986
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## I(speed^2) 0.09014 0.02939 3.067 0.00355 **
## speed 1.23903 0.55997 2.213 0.03171 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 15.02 on 48 degrees of freedom
## Multiple R-squared: 0.9133, Adjusted R-squared: 0.9097
## F-statistic: 252.8 on 2 and 48 DF, p-value: < 2.2e-16
That model is slightly better, what do the residuals look like?
diag_data3 <- augment(cars.lm3) %>%
mutate(sqrabs_resid = sqrt(abs(.std.resid)))
sl_plot <- ggplot(data = diag_data3,
mapping = aes(x = .fitted, y = sqrabs_resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 0.822179) +
labs(x = "distance [feet]",
y = expression(sqrt(abs(epsilon))))
nl_plot <- ggplot(data = diag_data3,
mapping = aes(x = speed, y = .resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 0) +
labs(x = "Speed [mph]",
y = expression(epsilon))
egg::ggarrange(sl_plot, nl_plot, nrow=1)
Nice! That’s got the non-linearity solved, but we’re still seeing that
increase in variance. I’m getting a bit carried away now, but I wonder if
that can be solved too. A little internet searching lead me to
package gamlss
. This allows me to specify a model for the variance of the model as well as the mean.
library("gamlss")
cars.lss <- gamlss(dist ~ I(speed^2) + speed + 0, ~speed, data = cars)
## GAMLSS-RS iteration 1: Global Deviance = 404.4768
## GAMLSS-RS iteration 2: Global Deviance = 404.4393
## GAMLSS-RS iteration 3: Global Deviance = 404.4393
summary(cars.lss)
## ******************************************************************
## Family: c("NO", "Normal")
##
## Call:
## gamlss(formula = dist ~ I(speed^2) + speed + 0, sigma.formula = ~speed,
## data = cars)
##
## Fitting method: RS()
##
## ------------------------------------------------------------------
## Mu link function: identity
## Mu Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## I(speed^2) 0.08494 0.02868 2.962 0.00483 **
## speed 1.33137 0.47851 2.782 0.00780 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## ------------------------------------------------------------------
## Sigma link function: log
## Sigma Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.71050 0.37738 4.533 4.14e-05 ***
## speed 0.05941 0.02362 2.515 0.0155 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## ------------------------------------------------------------------
## No. of observations in the fit: 50
## Degrees of Freedom for the fit: 4
## Residual Deg. of Freedom: 46
## at cycle: 3
##
## Global Deviance: 404.4393
## AIC: 412.4393
## SBC: 420.0874
## ******************************************************************
And now for the residual plots. broom
doesn’t appear to work with these
models.
diag_data4 <- cars %>%
mutate(.fitted = fitted(cars.lss),
.resid = resid(cars.lss, what = "mu"),
.std.resid = resid(cars.lss, what = "z-scores"),
sqrabs_resid = sqrt(abs(.std.resid)))
sl_plot <- ggplot(data = diag_data4,
mapping = aes(x = .fitted, y = sqrabs_resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 0.822179) +
labs(x = "distance [feet]",
y = expression(sqrt(abs(epsilon))))
nl_plot <- ggplot(data = diag_data4,
mapping = aes(x = speed, y = .resid)) +
geom_point() +
geom_smooth() +
geom_hline(yintercept = 0) +
labs(x = "Speed [mph]",
y = expression(epsilon))
egg::ggarrange(sl_plot, nl_plot, nrow=1)
And that’s done it. For comparison, the AIC of the quadratic model with a fixed variance is<PHONE_NUMBER>, so modeling the variance has given us a better model by quite a bit.
I learned quite a bit from this little exercise, more than I expected! A really nice example of how thinking about the underlying process represented by a set of data can dramatically improve a statistical model.


--- layout: post title: A Video Metaphor for What I Do At Work - Plate Spinner categories: [] tags: - funny - geek - work status: publish type: post published: true meta: _original_post_id: '' _original_import_author: John Duprey author: ---
[youtube http://www.youtube.com/watch?v=Zhoos1oY404?wmode=transparent]
.. and I get the feeling they'll keep giving me more plates to spin until I start dropping and breaking them.. .. and sometimes it feels like this: [youtube http://www.youtube.com/watch?v=4wp3m1vg06Q?wmode=transparent]


Struct SimulatedHandControllerPose
Simulated pose of an hand controller defined by recorded joint poses.
Assembly: XRTK.dll
Syntax
public struct SimulatedHandControllerPose : IEqualityComparer
Constructors
|
Improve this Doc
View Source
SimulatedHandControllerPose(String)
Creates a new hand pose with all joints in their
local space origin.
Declaration
public SimulatedHandControllerPose(string id)
Parameters
| Type |
Name |
Description |
| String |
id |
Unique pose identifier.
|
|
Improve this Doc
View Source
SimulatedHandControllerPose(SimulatedHandControllerPose)
Declaration
public SimulatedHandControllerPose(SimulatedHandControllerPose pose)
Parameters
Properties
|
Improve this Doc
View Source
DefaultHandPose
Gets the configured default pose for simulation hands.
Declaration
public static HandControllerPoseProfile DefaultHandPose { get; }
Property Value
|
Improve this Doc
View Source
Id
Gets the unique identifier for the simulated pose.
Declaration
public string Id { get; }
Property Value
|
Improve this Doc
View Source
LocalJointPoses
Joint poses are stored as right-hand poses in camera space.
Output poses are computed in world space, and mirroring on the x axis for the left hand.
Declaration
public MixedRealityPose[] LocalJointPoses { get; }
Property Value
Methods
|
Improve this Doc
View Source
Equals(Object)
Declaration
public override bool Equals(object obj)
Parameters
| Type |
Name |
Description |
| Object |
obj |
|
Returns
Overrides
|
Improve this Doc
View Source
Equals(SimulatedHandControllerPose)
Determines whether the specified object is equal to this instance.
Declaration
public bool Equals(SimulatedHandControllerPose other)
Parameters
Returns
| Type |
Description |
| Boolean |
True, if the specified object is equal to this instance, otherwise false.
|
|
Improve this Doc
View Source
FromJson(String)
Deserialize pose data from JSON format.
Declaration
public void FromJson(string json)
Parameters
| Type |
Name |
Description |
| String |
json |
|
|
Improve this Doc
View Source
GetHashCode()
Declaration
public override int GetHashCode()
Returns
Overrides
|
Improve this Doc
View Source
GetPoseByName(String)
Gets the pose for a given pose name. Will throw it pose not registered.
Declaration
public static SimulatedHandControllerPose GetPoseByName(string name)
Parameters
| Type |
Name |
Description |
| String |
name |
The name of the pose to lookup.
|
Returns
|
Improve this Doc
View Source
Initialize(IReadOnlyList<HandControllerPoseProfile>)
Initialize pose data for use in editor from files.
Declaration
public static void Initialize(IReadOnlyList<HandControllerPoseProfile> poses)
Parameters
|
Improve this Doc
View Source
Lerp(ref SimulatedHandControllerPose, SimulatedHandControllerPose, SimulatedHandControllerPose, Single)
Interpolates between the poses a and b by the interpolant t.
Declaration
public static void Lerp(ref SimulatedHandControllerPose interpolatedPose, SimulatedHandControllerPose a, SimulatedHandControllerPose b, float t)
Parameters
|
Improve this Doc
View Source
SetZero()
Declaration
|
Improve this Doc
View Source
ToJson()
Serialize pose data to JSON format.
Declaration
Returns
|
Improve this Doc
View Source
ToString()
Declaration
public override string ToString()
Returns
Overrides
|
Improve this Doc
View Source
TryGetPoseByName(String, out SimulatedHandControllerPose)
Gets the pose for a given pose name, if it's registered.
Declaration
public static bool TryGetPoseByName(string name, out SimulatedHandControllerPose result)
Parameters
Returns
| Type |
Description |
| Boolean |
True, if found.
|
Operators
|
Improve this Doc
View Source
Equality(SimulatedHandControllerPose, SimulatedHandControllerPose)
Declaration
public static bool operator ==(SimulatedHandControllerPose left, SimulatedHandControllerPose right)
Parameters
Returns
|
Improve this Doc
View Source
Inequality(SimulatedHandControllerPose, SimulatedHandControllerPose)
Declaration
public static bool operator !=(SimulatedHandControllerPose left, SimulatedHandControllerPose right)
Parameters
Returns
Explicit Interface Implementations
|
Improve this Doc
View Source
IEqualityComparer.Equals(Object, Object)
Declaration
bool IEqualityComparer.Equals(object left, object right)
Parameters
Returns
|
Improve this Doc
View Source
IEqualityComparer.GetHashCode(Object)
Declaration
int IEqualityComparer.GetHashCode(object obj)
Parameters
| Type |
Name |
Description |
| Object |
obj |
|
Returns
Implements
Extension Methods


“Cirarattaṁ vatātāpī,
dhammaṁ anuvicintayaṁ;
Samaṁ cittassa nālatthaṁ,
pucchaṁ samaṇabrāhmaṇe.
For a long time, sadly,
though I keenly contemplated the teaching,
I gained no peace of mind.
So I asked this of ascetics and brahmins:
‘Ko so pāraṅgato loke,
ko patto amatogadhaṁ;
Kassa dhammaṁ paṭicchāmi,
paramatthavijānanaṁ’.
“Who has crossed over the world?
Whose attainment culminates in the deathless?
Whose teaching do I accept
to understand the highest goal?
Antovaṅkagato āsi,
macchova ghasamāmisaṁ;
Baddho mahindapāsena,
vepacityasuro yathā.
I was hooked inside,
like a fish gulping bait;
bound like the demon Vepaciti
in Mahinda’s trap.
Añchāmi naṁ na muñcāmi,
asmā sokapariddavā;
Ko me bandhaṁ muñcaṁ loke,
sambodhiṁ vedayissati.
Dragging it along, I’m not free
from grief and lamentation.
Who will free me from bonds in the world,
so that I may know awakening?
Samaṇaṁ brāhmaṇaṁ vā kaṁ,
ādisantaṁ pabhaṅgunaṁ;
Kassa dhammaṁ paṭicchāmi,
jarāmaccupavāhanaṁ.
What ascetic or brahmin
points out what is frail?
Whose teaching do I accept
to sweep away old age and death?
Vicikicchākaṅkhāganthitaṁ,
sārambhabalasaññutaṁ;
Kodhappattamanatthaddhaṁ,
abhijappappadāraṇaṁ.
Tied up with uncertainty and doubt,
secured by the power of pride,
rigid as a mind beset by anger;
the arrow of covetousness,
Taṇhādhanusamuṭṭhānaṁ,
dve ca pannarasāyutaṁ;
Passa orasikaṁ bāḷhaṁ,
bhetvāna yadi tiṭṭhati.
propelled by the bow of craving,
is stuck in my twice-fifteen ribs—
see how it stands in my breast,
breaking my strong heart.
Anudiṭṭhīnaṁ appahānaṁ,
saṅkappaparatejitaṁ;
Tena viddho pavedhāmi,
pattaṁva māluteritaṁ.
Speculative views are not abandoned,
they are sharpened by memories and intentions;
and pierced by this I tremble,
like a leaf blowing in the wind.
Ajjhattaṁ me samuṭṭhāya,
khippaṁ paccati māmakaṁ;
Chaphassāyatanī kāyo,
yattha sarati sabbadā.
Having arisen within,
what belongs to me burns quickly,
in that place where the body always heads
with its six sense-fields of contact.
Taṁ na passāmi tekicchaṁ,
yo metaṁ sallamuddhare;
Nānārajjena satthena,
nāññena vicikicchitaṁ.
I don’t see a healer
who can pull out my dart of doubt
without a lance
or some other blade.
Ko me asattho avaṇo,
Sallamabbhantarapassayaṁ;
Ahiṁsaṁ sabbagattāni,
Sallaṁ me uddharissati.
Without knife or wound,
who will pull out this dart
that’s stuck inside me,
without harming any part of my body?
Dhammappati hi so seṭṭho,
visadosappavāhako;
Gambhīre patitassa me,
thalaṁ pāṇiñca dassaye.
He really would be the Lord of the Dhamma,
the best one to cure the damage of poison;
when I have fallen into deep waters,
he would give me his hand and bring me to the shore.
Rahadehamasmi ogāḷho,
ahāriyarajamattike;
Māyāusūyasārambha,
thinamiddhamapatthaṭe.
I’ve plunged into a lake,
and I can’t wash off the mud and dirt.
It’s full of fraud, jealousy, pride,
and dullness and drowsiness.
Uddhaccameghathanitaṁ,
saṁyojanavalāhakaṁ;
Vāhā vahanti kuddiṭṭhiṁ,
saṅkappā rāganissitā.
Like a thunder-cloud of restlessness,
like a rain-cloud of fetters;
lustful thoughts are winds
that sweep off a person with bad views.
Savanti sabbadhi sotā,
latā ubbhijja tiṭṭhati;
Te sote ko nivāreyya,
taṁ lataṁ ko hi checchati.
The streams flow everywhere;
a weed springs up and remains.
Who will block the streams?
Who will cut the weed?”
Velaṁ karotha bhaddante,
sotānaṁ sannivāraṇaṁ;
Mā te manomayo soto,
rukkhaṁva sahasā luve.
“Venerable sir, build a dam
to block the streams.

Evaṁ me bhayajātassa,
apārā pāramesato;
Tāṇo paññāvudho satthā,
isisaṅghanisevito.
That is how the teacher whose weapon is wisdom,
surrounded by the Saṅgha of hermits,
was my shelter when I was full of fear,
seeking the far shore from the near.
Sopāṇaṁ sugataṁ suddhaṁ,
dhammasāramayaṁ daḷhaṁ;
Pādāsi vuyhamānassa,
‘mā bhāyī’ti ca mabravi.
As I was being swept away,
he gave me a strong, simple ladder,
made of the heartwood of Dhamma,
and he said to me: “Do not fear.”
Satipaṭṭhānapāsādaṁ,
āruyha paccavekkhisaṁ;
Yaṁ taṁ pubbe amaññissaṁ,
sakkāyābhirataṁ pajaṁ.
I climbed the tower of mindfulness meditation,
and checked back down
at people delighting in identity,
as I’d obsessed over in the past.
Yadā ca maggamaddakkhiṁ,
nāvāya abhirūhanaṁ;
Anadhiṭṭhāya attānaṁ,
titthamaddakkhimuttamaṁ.
When I saw the path,
as I was embarking on the ship,
without fixating on the self,
I saw the supreme landing-place.
Sallaṁ attasamuṭṭhānaṁ,
bhavanettippabhāvitaṁ;
Etesaṁ appavattāya,
desesi maggamuttamaṁ.
The dart that arises in oneself,
and that which stems from attachment to rebirth:
he taught the supreme path
for the canceling of these.
Dīgharattānusayitaṁ,
cirarattamadhiṭṭhitaṁ;
Buddho mepānudī ganthaṁ,
visadosappavāhano”ti.
… Telakāni thero ….
For a long time it had lain within me;
for a long time it was fixed in me:
the Buddha cast off the knot,
curing the damage of poison.


Canvas Life
Home
About
Guide
Your browser does not support the Canvas element.
Run
Step
Clear
Grid Off
FPS
Save RLE
Load RLE
?
bo$2bo$3o!
Examples:
None
Title
Glider
Canada Goose
R-Pentomino
Line
Generation:
0
Position:
0, 0
Living:
0
Start:
0
Max:
0
(
0
)


Hey All,
This was a long email but wanted to do a quick followup on the <a h="ref="http://www.onemedical.com/sf/doctors/?gclid=3DCJi_-9bP9sUCFUiGfgodMA=" uaxq"="">One Medical portion. I've only received a few responses so far.&n= bsp;
Signing up would not require you to change your current primary care doc= tor or pay additional insurance, rather it is a primary care center (think = Kaiser) where you can make all of your appointments. Super convenient right= !?
Get back to me if this is something you want to participate in. Nylas wo= uld cover the cost.
Thanks,
Makala


Preventive healthcare technology — the future of medicine?
DUBAI: Dubai is the first Middle Eastern city to get its very own edition of Monopoly. On Monday night, the new version of the famed Hasbro boardgame featuring some of the Emirati city’s most iconic landmarks was unveiled at the Burj Al-Arab.
Some of the prominent landmarks featured on the board are the Dubai Airport, Burj Khalifa, Global Village and Sheikh Zayed Road, while the Burj Al-Arab holds the top spot with a value of 400 Monopoly money.
Also included are Bluewaters, Dubai Marina, City Walk, Mall of the Emirates, Dubai Taxi, Nord Anglia International Dubai School, Dubai Opera, Festival City, Al Seef, Dubai Bus, Dubai Frame, Dubai Miracle Garden, La Mer, Xclusive Yachts, Dubai Metro, Al Fahidi, Atlantis the Palm, Jumeirah Beach Hotel, Atlantis Aquaventure and Jumeirah Emirates Towers.


Success!.
Welcome to Global Platfoms!
Please do not close this window while you are connected to the Internet!


Development
Space-Time was made in Python 2.7, with the help of NLTK, BeautifulSoup, and Python Magic. The document data was stored in Sqlite3 databases, built with SQLiteStudio.There are four primary parts to this search engine: indexing, preloading (optional), ranking, and searching.
The indexing is performed once, before the program is used. This consists of opening and reading every single document, parsing the html via BeautifulSoup, getting unique words and their frequencies, and finally storing those counts along with any extra data into a database. The indexer uses inverted indexing, and ignores stop words, and other unwanted strings (hex, numbers, or different languages) with the help of regular expressions and NLTK. NLTK is also used to stem words, so that words like "computer" and "computers" are considered the same. The indexer also ignores certain file types like audio, video, and images with the help of Python Magic. Finally, the indexer gives higher weight to terms in the strong tag, or in headers and titles.
Once the indexing is complete, you can run the search engine and perform searches, OR you can choose to preload some data. Preloading data will calculate some necessary data before running searches, to improve performance at the cost of extra space usage. This includes calculating document frequencies, and the tf-idf weights.
The next two parts go hand-in-hand. To search for a term, the user simply types in whatever search terms into the command line. The search engine parses the search query, and again ignores stop words via NLTK. Then, the engine passes the parsed query to the ranker, which will rank the query and its terms against every document in the database. It calculates a normalized tf-idf weight for query, and compares it to every document via Cosine similarity. It finally returns the top K documents.
The search engine returns a list of url links, along with their scores. It also shows how many relevant documents there were, and how many strongly-relevant (i.e. if the score was higher than some threshold). Finally, it also shows how long it took to get the documents. The final part is especially important, as the time elapsed and relevancy of the top K results were used to determine how effective the search engine was.
Closing Thoughts
As you can see from the images, this search engine isn't perfect. There is still much room to grow and improve on. There are numerous, more complex algorithms and theories that can be utilized to improve the relevancy of the top K results. However, for the purposes and scope of the course, this search engine worked perfectly well. It was able to (most of the time) get very relevant results for queries, and in a reasonable time. All in all, I'm quite happy with how it turned out.


The other day I had just started working on puzzles in Advent of Code after seeing it on HackerNews (The puzzles are fun. You should definitely try them out). The first half of the first problem was very easy and so I finished it in less than 5 minutes. Only to find out that my program was generating the wrong solution for the given input. I was confounded. There was nothing logically wrong with it. This was the problem and the solution looked like this:
class day_1:
str = ""
floor = 0
def __init__(self, str):
self.str = str
def calculate(self):
for c in str:
if c == '(':
self.floor += 1
else:
self.floor -= 1
if __name__ == "__main__":
str = input("Enter input string: ")
print(len(str))
x = day_1(str)
x.calculate()
print(x.floor)
As you can see the program is pretty simple; it reads an input string from the commandline and processes the string. On checking the size of the input string in the program, I found that its size was 4095 characters. Comparing that to the actual input string from the website I found that its size was some 7000 odd characters. The reason for this discrepancy as I later found out is that the Linux commandline (and I believe all POSIX compliant commandlines) can take upto 4096 characters of input (4kB) only.
If you just want to know what the solution to this problem is then here it is. There are two options:
-
Read from a file instead of taking input from the command line.
-
Or enter the command
stty -icanon
before starting your program. This will change the input mode tononcanonical
mode which allows your inputs to be as big as you want. Be aware that this will remove all text processing capabilities from commandline input. I.e. you won’t be able to use your arrow keys, backspace, delete, ability to end input using ctrl-D etc.stty icannon
will revert the input tocanonical
mode.
If you’re now wondering what canonical and non-canonical mean then read on.
What is canonical mode? Why does it limit the input size?
How the data input in the terminal by the user is provided to a process depends on whether the terminal is in the canonical or non-canonical mode.
-
In Canonical mode the terminal provides basic text processing abilities on input: You can navigate inside the input buffer using arrow keys; erase, delete, EOF signals are captured and applicable text processing is performed. Text is passed to the process only when a new line is encountered or when the buffer is full. The size of the buffer is usually 4096 and this is why the input size is limited. As mentioned above you can switch from canonical to non-canonical mode using
stty -icanon
. -
Non-Canonical mode has none of these text-processing capabilities. I.e if you wanted to implement a commandline tool like cat and you wanted it to operate in a non-canonical environment you’d have to implement some of these capabilities yourself within your program. You can switch back to canonical mode using
stty icanon
.
More info is available here:
-
The Open Group Base Specifications Issue 7 - General Terminal Interface
-
Also, you can see the actual buffer size defined here (on line #308)


| lifecycle {rlang} | R Documentation |
The rlang package is currently maturing. Unless otherwise stated, this applies to all its exported functions. Maturing functions are susceptible to API changes. Only use these in packages if you're prepared to make changes as the package evolves. See sections below for a list of functions marked as stable.
The documentation pages of retired functions contain life cycle sections that explain the reasons for their retirements.
These functions are not yet part of the rlang API. Expect breaking changes.
pkg_env()
, pkg_env_name()
, ns_env()
, ns_imports_env()
, ns_env_name()
is_definition()
, new_definition()
, is_formulaish()
,
dots_definitions()
local_options()
, with_options()
, push_options()
,
peek_options()
, peek_option()
In the questioning stage as of rlang 0.4.0
These functions are likely to be moved to the vctrs package:
new_logical()
, new_integer()
, etc.
na_lgl, na_int, is_lgl_na()
, is_int_na()
, etc.
In the questioning stage as of rlang 0.3.0
with_restarts()
, rst_list()
, rst_exists()
, rst_jump()
,
rst_maybe_jump()
, rst_abort()
. It is not clear yet whether we
want to recommend restarts as a style of programming in R.
return_from()
and return_to()
.
expr_label()
, expr_name()
, and expr_text()
.
In the questioning stage as of rlang 0.2.0
Soft-deprecated in rlang 0.4.0
exiting()
: Handlers are now treated as exiting by default.
as_logical()
, as_integer()
, etc. => vctrs::vec_cast()
.
type_of()
, switch_type()
, coerce_type()
, switch_class()
,
coerce_class()
Bumped to deprecated in rlang 0.4.0
new_logical_along()
, new_integer_along()
,
new_double_along()
, new_complex_along()
,
new_character_along()
, new_raw_along()
, new_list_along()
.
is_unary_lang()
=> Use the n
argument of is_call()
is_binary_lang()
=> Use the n
argument of is_call()
call_modify()
: .standardise
and .env
arguments.
quo_expr()
=> quo_squash()
Assigning non-quosure objects to quosure lists.
as.character()
on quosures.
cnd_signal()
: .cnd
=> cnd
cnd_signal()
: The .mufflable
argument no longer has any effect
scoped_names()
=> base::search()
is_scoped()
=> is_attached()
scoped_env()
=> search_env()
scoped_envs()
=> search_envs()
env_bind_exprs()
=> env_bind_lazy()
env_bind_fns()
=> env_bind_active()
Passing a function or formula to env_depth()
,
env_poke_parent()
, env_parent<-
, env_tail()
, set_env()
,
env_clone()
, env_inherits()
, env_bind()
,
local_bindings()
, with_bindings()
, env_poke()
,
env_has()
, env_get()
, env_names()
, env_bind_exprs()
and
env_bind_fns()
. This internal genericity was causing confusion
(see issue #427). You should now extract the environment
separately before calling these functions.
get_env()
: The env
argument no longer has a default and must be supplied
is_frame()
, global_frame()
, current_frame()
,
ctxt_frame()
, call_frame()
, frame_position()
,
caller_frame()
ctxt_depth()
, call_depth()
, ctxt_stack()
, call_stack()
,
stack_trim()
The width
and printer
arguments of exprs_auto_name()
and
quos_auto_name()
no longer have any effect. For the same
reason, passing a width as .named
argument of dots collectors
like quos()
is deprecated.
as_overscope()
=> as_data_mask()
new_overscope()
=> new_data_mask()
overscope_eval_next()
=> eval_tidy()
overscope_clean()
Defunct as of rlang 0.4.0
length()
and names()
on tidy eval .data
pronouns.
Supplying a named !!!
call.


vignettes/animating_covid19_mobility.Rmd
animating_covid19_mobility.Rmd
Hello everyone! This vignette shows how to plot social distancing data on a map using the covid19mobility package.
The packages used to perform the analyses are below, in the approximate order they were used, and what they were used for.
library(covid19mobility) #to get data on mobility from Apple/Google library(dplyr) # for joining, piping data frames library(lubridate) # for dealing with our date data library(sf) # shapefile manipulation library(urbnmapr) #to get county and state data in a shapefile library(gganimate) # to animate plots
First, we call the data from Apple using the refresh_covid19mobility_apple_subregion()
function. There are multiple data sets in the covid19mobility package, so you could theoretically put in any of the other functions to look at different regions and breakdowns of data. I recommend running the refresh
functions by one so you can see what the data structure looks like before you start tidying. Here is what the raw data looks like below. We can see that this data is global and breaks down the type of movement (driving, transit, walking) and also includes location codes and types.
head(refresh_covid19mobility_apple_subregion()) #> # A tibble: 6 x 9 #> date location location_type location_code location_code_t… data_type #> <date> <chr> <chr> <chr> <chr> <chr> #> 1 2020-01-13 Abruzzo… state IT-65 iso_3166_2 driving_… #> 2 2020-01-14 Abruzzo… state IT-65 iso_3166_2 driving_… #> 3 2020-01-15 Abruzzo… state IT-65 iso_3166_2 driving_… #> 4 2020-01-16 Abruzzo… state IT-65 iso_3166_2 driving_… #> 5 2020-01-17 Abruzzo… state IT-65 iso_3166_2 driving_… #> 6 2020-01-18 Abruzzo… state IT-65 iso_3166_2 driving_… #> # … with 3 more variables: value <dbl>, alternative_name <chr>, country <chr>
What we are then going to do is tidy up that above data using dplyr
verbs (my favorite!) First, we want to select all of the datapoints from the US. We’re going to summarize and group each state by mean by date. We also want to only graph weekly intervals to limit the amount of frames in the gif: If you try to put them all in the same gif at once it looks like a nightmare.
apple_us <- refresh_covid19mobility_apple_subregion() %>% filter(country=="United States") %>% # we are only going to map the US group_by(location,date) %>% #we are going to group by location (which is the state) and date summarize_if(is.numeric, mean) %>% #we are going to take the mean mobility score across counties by date and state mutate(weeks=round_date(date, "weeks")) %>% #we are going to take only the first date of each week group_by(weeks) %>% # we are going to group by each week filter(date==min(weeks)) #again we want to take only the first date of each week
The next thing we are going to do is we’re going to call shape files for the US states and counties. We get this from the urbnmapr package. We need to also call the sf package because we are working with shapefiles.
states_sf <- urbnmapr::get_urbn_map("states", sf=TRUE) %>% st_as_sf() # coerce this to a shapefile.
Here, we want to left_join()
our shapefile tibble and our mobility tibble together so that we can graph them all together. Sometimes ggplot will throw an error because when you join the tibbles, you lose the identification that we do have shapefiles in our data set (see this github post. So you always want to use st_as_sf()
to save the day!)
total_apple_us <- left_join(apple_us,states_sf, by=c("location"="state_name")) %>% st_as_sf() # this function makes sure the shape files are readable. total_apple_us <- as.data.frame(total_apple_us) %>% st_as_sf()
Now, we are going to set up our ggplot graph. First, we need to check and make sure that the state shapefile data is good, so we graph it below.
ggplot() + geom_sf(data=states_sf)
Finally, we can construct the graph. There are a lot of moving parts here, so I’ve commented each line to indicate what it does.
ggplot() + #call ggplot geom_sf(data=states_sf) + # map the shape files to the map of the USA geom_sf(data= total_apple_us, aes(fill=value)) + # fill with our data frame and value, which is mobility score labs(fill="% change in mobility", title="Date: {current_frame}", subtitle="Percent Change in Movement, Source: Apple Mobility Data", fill = "% change in mobility") + #add titles scale_fill_viridis_c() + transition_manual(date) #animate by day - this is a gganimate function!
In the graph, darker colors indicate less movement than usual (compared to the baseline) and lighter colors indicate more movement. When you look at this graph, it’s pretty cool to notice how the percent change in mobility decreases as the pandemic progresses and social distancing continues on. However, you can also see that towards the end of the gif, some of the states start moving around more as reopenings begin.
And that’s pretty much it! You can apply this method of graphing to multiple segments of the data and functions in the package, including the Apple data, the Google data, and you can also use ggmap to make worldwide maps. Happy mapping, and stay safe out there!
Cleo Falvey
UMass Boston Biology Major, Math Minor


DINOSAUR!!!:
I made this badass Lyrasauraus a couple years ago. I planed some lyra down to about 1/8th thick and then cut all the bones out on the waterjet. Then I gave it to my indolent brother for Christmas so of course now it's destroyed.  So now all that remains is this photo. What do you think? Email me<EMAIL_ADDRESS>Lyrasaurus:
© Copyright 2018 by laconiclizard


Original design Card hours from BootstrapTema
You can use all basic classes of colors, change background color, change text color and border color hours
: :


Asking the right questions
Why asking the right questions is important?
- Frustration will come along with learning something new and challenging, such as coding. Inevitably, you will need help from other people, either other students, mentors, or teachers. The ability to convey your frustration and explain the problems/issues that you are facing will be critical for your success in both Dev Bootcamp, personal life, and professional career.
- It is important to ask the right questions to get the answers that you want. Otherwise, you will get responses that won't help to solve your problem.
- Ask the wrong questions and you will get the wrong answers!
Tips on asking the right questions
- Provide context on why you are asking the question. What is the problem that you are trying to solve and why?
- Be specific and provide details to ensure that there is no ambiguity in your question. A good question, if asked the right way, should not confuse other people. It doesn't mean that other people will know the answer, but they will know exactly what you are asking and trying to solve.
- Ask your questions in a way that anyone can understand. In other words, try to simplify the question and use words and language that convey what you are asking but also easy for others to understand.
- Be ready to ask follow up questions for clarification. A single question and answer may not do the trick. It's important to probe and ask follow up questions to get the answer that you are looking for if the initial responses do not address your questions.