class: center, middle, inverse, title-slide # Introduction to Data Science ## Session 5: Web data and technologies ### Simon Munzert ### Hertie School |
GRAD-C11/E1339
---  # Table of contents
 1. [Web data for data science](#webdata) 2. [HTML basics](#html) 3. [XPath basics](#xpath) 4. [CSS basics](#css) 5. [Scraping static webpages with R](#scrapingstatic) 6. [Web scraping: good practice](#goodpractice) 7. [Summary](#summary) --- class: inverse, center, middle name: webdata # Web data for data science

 --- # What is web data?

 --- # What is web data? (cont.)

 --- # What is web data? (cont.) .pull-left[ ### So what is web data, really? - Not all data you get from the web is "web data". - Web data is **data that is created on, for, or via the web**. By that definition, a survey dataset that you download from a data repository is not web data. - On the other hand, survey data collected online (i.e., web/mobile questionnaires) is web data but we don't consider it in today's session. - Examples of web data: - Online news articles - Social media network structures - Crowdsourced databases (e.g., Wikidata) - Server logs (e.g., viewership statistics) - Data from surveys, experiments, clickworkers - Just any website ] -- .pull-right[ ### And why is web data attractive? - Data is abundant online. - Human behavior increasingly takes place online. - Countless services track human behavior. - Getting data from the web is cheap and often quick. - An analysis workflow that involves web data can often be easily updated. - The vast majority of web data was not created with a data analysis purpose in mind. This fact is often a feature, not a bug.
 Today, we focus on one particular way of collecting data from the web: web scraping. This also limits the type of web data we'll be talking about (basically: data from static webpages). But it'll be fun nevertheless.
 ] --- # Web scraping .pull-left[ ### What is web scraping? 1. Pulling (unstructured) data from websites (HTMLs) 2. Bringing it into shape (into an analysis-ready format) ### The philosophy of scraping with R - No point-and-click procedure - Script the entire process from start to finish - **Automate** - The downloading of files - The scraping of information from web sites - Tapping APIs - Parsing of web content - Data tidying, text data processing - Easily scale up scraping procedures - Scheduling of scraping tasks ] .pull-right-center[

 `Credit` [prowebscraping.com](http://prowebscraping.com/web-scraping-vs-web-crawling/) ] --- # Technologies of the world wide web .pull-left[ - To fully unlock the potential of web data for data science, we draw on certain web technologies. - Importantly, often a basic understanding of these technologies is sufficient as the focus is on web data collection, not [web development](https://en.wikipedia.org/wiki/Web_development). - Specifically, we have to understand - How our machine/browser/R communicates with web servers (→ **HTTP/S**) - How websites are built (→ **HTML**, **CSS**, basics of **JavaScript**) - How content in webpages can be effectively located (→ **XPath**, **CSS selectors**) - How dynamic web applications are executed and tapped (→ **AJAX**, **Selenium**) - How data by web services is distributed and processed (→ **APIs**, **JSON**, **XML**) ] .pull-right-center[

 `Credit` [ADCR](http://r-datacollection.com/) ] --- class: inverse, center, middle name: html # HTML basics

 --- # HTML background .pull-left-wide[ ### What is HTML? - **H**yper**T**ext **M**arkup **L**anguage - Markup language = plain text + markups - Originally specified by [Tim Berners-Lee](https://en.wikipedia.org/wiki/Tim_Berners-Lee) at [CERN](https://en.wikipedia.org/wiki/CERN) in 1989/90 - [W3C](https://en.wikipedia.org/wiki/World_Wide_Web_Consortium) standard for the construction of websites. - The fundamentals of HTML haven't changed much recently. Current version is HTML 5.2 (published in 2017). ### What is it good for? - In the early days, the internet was mainly good for sharing texts. But plain text is boring. Markup is *fun*! - HTML lies underneath of what you see in your browser. You don't see it because your browser interprets and renders it for you. - A basic understanding of HTML helps us locate the information we want to retrieve. ] .pull-right-small-center[

 ] --- # HTML tree structure .pull-left[ ### The DOM tree - HTML documents are hierarchically structured. Think of them as a tree with multiple nodes and branches. - When a webpage (HTML resource) is loaded, the browser creates a [Document Object Model](https://en.wikipedia.org/wiki/Document_Object_Model) of that page - the **DOM Tree**. - Think of it as a representation that considers all HTML elements as objects than can be accessed. ### Parts of the tree - The DOM is constituted of **nodes**, which are just data types that can be referred to - such as "attribute node", "text node", or "element node". - A **node set** is a set of nodes. This will become relevant when you learn about XPath, which you can use to access multiple nodes (e.g., all `title` nodes). ] .pull-right[ ```html <!DOCTYPE html> <html> <head> <title id=1>First HTML</title> </head> <body> I am your first HTML file! </body> </html> ```

 ] --- # HTML: elements and attributes .pull-left[ ### Elements - Elements are a combination of start tags, content, and end tags. - Example: `<title>First HTML</title>` - An element is everything from (including) the element's start tag to (including) the element's end tag, but also other elements that are nested within that element. - Syntax: | Component | Representation | |---|---| | Element title | `title` | | Start tag | `<title>` | | End tag | `</title>` | | Value | `First HTML` | ] .pull-right[ ### Attributes - Describe elements and are stored in the start tag. - There are specific attributes for specific elements. - Example: `<a href="http://www.r-datacollection.com/">Link to Homepage</a>` - Syntax: - Name-value pairs: `name="value"` - Simple and double quotation marks possible - Several attributes per element possible ### Why tags and attributes are important - Tags structure HTML documents. - In the context of web scraping, the structure can be exploited to locate and extract data from websites. ] --- # Important tags and attributes ### Anchor tag `<a>` - Links to other pages or resources. - Classical links are always formatted with an anchor tag. - The `href` attribute determines the target location. - The value is the name of the link. Link to another resource: ```html <a href="en.wikipedia.org/wiki/List_of_lists_of_lists">Link with absolute path</a> ``` Reference within a document: ```html <a id="top">Reference point</a> ``` Link to a reference within a document: ```html <a href="#top">Link to reference point</a> ``` --- # Important tags and attributes ### Heading tags `<h1>`, `<h2>`, ..., and paragraph tag `<p>` - Structure text and paragraphs. - Heading tags range from level 1 to 6. - Paragraph tag induces a line break. Examples: ```html <p>This text is going to be a paragraph one day and separated from other text by line breaks.</p> ``` ```html <h1>heading of level 1 - this will be BIG</h1> ... <h6>heading of level 6 - the smallest heading</h6> ``` --- # Important tags and attributes ### Listing tags `<ul>`, `<ol>`, and `<dl>` - The `<ol>` tag creates a numeric list. - The `<ul>` tag creates an unnumbered list. - The `<dl>` tag creates a description list. - List elements within `<ol>` and `<ul>` are indicated with the `<li>` tag. Example: ```html <ul> <li>Dogs</li> <li>Cats</li> <li>Fish</li> </ul> ``` --- # Important tags and attributes ### Organizational and styling tags `<div>` and `<span>` - They are used to group content over lines (`<div>`, creating a block-level element) or within lines (`<span>`, creating an inline-element). - By grouping or dividing content into blocks, it's easier to identify or apply different styling to them. - They do not change the layout themselves but work together with CSS (see later!). .pull-left[ Example of CSS definition: ```css div.happy { color:pink; font-family:"Comic Sans MS"; font-size:120% } span.happy { color:pink; font-family:"Comic Sans MS"; font-size:120% } ``` ] .pull-right[ In the HTML document: ```html <div class="happy"> <p>I am a happy-styled paragraph</p> </div> unhappy text with <span class="happy">some happiness</span> ``` ] --- # Important tags and attributes ### Form tag `<form>` - Allows to incorporate HTML forms. - Client can send information to the server via forms. - Whenever you type something into a field or click on radio buttons in your browser, you are interacting with forms. Example: ```html <form name="submitPW" action="Passed.html" method="get"> password: <input name="pw" type="text" value=""> <input type="submit" value="SubmitButtonText" </form> ``` --- # Important tags and attributes ### Table tags `<table>`, `<tr>`, `<td>`, and `<th>` - Standard HTML tables always follow a standard architecture. - The different tags allow defining the table as a whole, individual rows (including the heading), and cells. - If the data is hidden in tables, scraping will be straightforward. Example: ```html <table> <tr> <th>Rank</th> <th>Nominal GDP</th> <th>Name</th> </tr> <tr> <th></th> <th>(per capita, USD)</th> <th></th> </tr> <tr> <td>1</td> <td>170,373</td> <td>Lichtenstein</td> </tr> <tr> <td>2</td> <td>167,021</td> <td>Monaco</td> </tr> <tr> <td>3</td> <td>115,377</td> <td>Luxembourg</td> </tr> <tr> <td>4</td> <td>98,565</td> <td>Norway</td> </tr> <tr> <td>5</td> <td>92,682</td> <td>Qatar</td> </tr> </table> ``` --- # More resources on HTML .pull-left-wide[ ### More HTML - All in all there are over 100 HTML elements. - But overall, it's still a fairly tight and easy-to-understand markup language. - Knowing more about the rest is probably not necessary to become a good web scraper, but it helps parsing (in your brain) HTML documents quicker. ### More resources - Check out the excellent [MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web/HTML) for an overview, which also point to additional tutorials and references. - The [W3Schools tutorials](https://www.w3schools.com/) are also a classic. - While you're at it, you might also want to learn about related technologies such as CSS (used to specify a webpage's appearance/layout) and JavaScript (used to enrich HTMLs with additional functionality and options to interact). ] .pull-right-small[

 ] --- # Accessing the web using your browser vs. R .pull-left-wide[ ### Using your browser to access webpages 1. You click on a link, enter a URL, run a Google query, etc. 2. Browser/your machine sends request to server that hosts website. 3. Server returns resource (often an HTML document). 4. Browser interprets HTML and renders it in a nice fashion. ### Using R to access webpages 1. You manually specify a resource. 2. R/your machine sends a request to the server that hosts the website. 3. The server returns a resource (e.g., an HTML file). 4. R parses the HTML, but does not render it in a nice fashion. 5. It's up to you to tell R what content to extract. ] .pull-right-small[

 ] --- # Interacting with your browser ### On web browsers - Modern browsers are complex pieces of software that take care of multiple operations while you browse the web. And they're basically all doing a good job.1 Common operations are to retrieve resources, render and display information, and provide interface for user-webpage interaction. - Although our goal is to automate web data retrieval, the browser is an important tool in web scraping workflow. ### The use of browsers for web scraping - Give you an intuitive impression of the architecture of a webpage - Allow you to inspect the source code - Let you construct XPath/CSS selector expressions with plugins - Render dynamic web content (JavaScript interpreter) .footnote[1 Check out this Wikipedia article on the [Browser Wars](https://en.wikipedia.org/wiki/Browser_wars) that happened in the 1990s and 2000s (yes, there was Browser War I and Browser War II - and for once Germany was not to blame) to relive some of your instructor's pains when he started to look into this "internet".] --- # Inspecting HTML source code .pull-left-small[

 - Goal: retrieving data from a Wikipedia page on [List of tallest buildings](https://en.wikipedia.org/wiki/List_of_tallest_buildings) - Right-click on page (anywhere) - Select `View Page Source` - HTML (CSS, JavaScript) code can be ugly - But looking more closely, we find the displayed information ] .pull-right-wide[
 <video width="768" height="432" controls> <source src="pics/inspect-pagesource.mp4" type="video/mp4"> Your browser does not support the video tag. </video> ] --- # Inspecting the live HTML source code with the DOM explorer .pull-left-small[ - Goal: retrieving data from a Wikipedia page on [List of tallest buildings](https://en.wikipedia.org/wiki/List_of_tallest_buildings) - Right-click on the element of interest - Select `Inspect` - The Web Developer Tools window pops up - Corresponding part in the HTML tree is highlighted - Interaction with the tree possible! ] .pull-right-wide[ <video width="768" height="432" controls> <source src="pics/inspect-pagedom.mp4" type="video/mp4"> Your browser does not support the video tag. </video> ] --- # When to do what with your browser .pull-left-wide[ ### When to inspect the complete page source - Check whether data is in static source code (the search function helps!) - For small HTML files: understand structure ### When to use the DOM explorer - Almost always - Particularly useful to construct XPath/CSS selector expressions - To monitor dynamic changes in the DOM tree ### A note on browser differences - Inspecting the source code (as shown on the following slides) works more or less identically in Chrome and Firefox. - In Safari, go to → `Preferences`, then → `Advanced` and select `Show Develop menu in menu bar`. This unlocks the `Show Page Source` and `Inspect` options and the Web Developer Tools. ] .pull-right-small-center[

 `Credit` [watershedcreative.com](http://watershedcreative.com/naked/html-tree.html) ] --- class: inverse, center, middle name: xpath # XPath basics

 --- # Accessing the DOM tree with R ### Different perspectives on HTML - HTML documents are human-readable. - HTML tags structure the document, comprising the DOM. - **Web user perspective**: The browser interprets the code and renders the page. - **Web scraper perspective**: Parse the document retaining the structure, use the tree/tags to locate information. -- ### HTML parsing - Our goal is to get HTML into R while retaining the tree structure. That's similar to getting a spreadsheet into R and retaining the rectangular structure. - HTML is human-readable, so we could also import HTML files as plain text via `readLines()`. That's a bad option though - the document's structure would not be retained. - The `xml2` package allows us to parse XML-style documents. HTML is a "flavor" of XML, so it works for us. - The `rvest` package, which we will mainly use for scraping, wraps the `xml2` package, so we rarely have to load it manually. - There is one high-level function to remember: `read_html()`. It represents the HTML in a list-style fashion. --- # Accessing the DOM tree with R (cont.) ### Getting HTML into R Parsing a website is straightforward: ```r R> library(rvest) R> parsed_doc <- read_html("https://google.com") R> parsed_doc ``` ``` ## {html_document} ## <html lang="de" dir="ltr"> ## [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ... ## [2] <body>\n<div class="signin"><a href="https://accounts.google.com/ServiceL ... ``` There are various functions to inspect the parsed document. They aren't really helpful - better use the browser instead if you want to dive into the HTML. ```r R> xml2::html_structure(parsed_doc) R> xml2::as_list(parsed_doc) ``` --- # What's XPath? ### Definition - Short for **XML Path Language**, another W3C standard. - A query language for XML-based documents (including HTML). - With XPath we can access node sets (e.g., elements, attributes) and extract content. ### Why XPath for web scraping? - Source code of webpages (HTML) structures both layout and content. - Not only content, but context matters! - XPath enables us to extract content based on its location in the document (and potentially other features). - With XPath, we can tell R to do things like: 1. Give me all `<li>` elements in the document! 2. Look for all `<table>` elements in the document and give me the third one! 3. Extract all content in `<p>` elements that is labelled with `class=newscontent`! --- # Example: source code ```html <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> <html> <head> <title>Collected R wisdoms</title> </head> <body> <div id="R Inventor" lang="english" date="June/2003"> <h1>Robert Gentleman</h1> <p><i>'What we have is nice, but we need something very different'</i></p> <p><b>Source: </b>Statistical Computing 2003, Reisensburg</p> </div> <div lang="english" date="October/2011"> <h1>Rolf Turner</h1> <p><i>'R is wonderful, but it cannot work magic'</i> <br><emph>answering a request for automatic generation of 'data from a known mean and 95% CI'</emph></p> <p><b>Source: </b><a href="https://stat.ethz.ch/mailman/listinfo/r-help">R-help</a></p> </div> <address> <a href="http://www.rdatacollectionbook.com"><i>The book homepage</i></a> </address> </body> </html> ``` --- # Example: DOM tree

 --- # Applying XPath on HTML in R - Load package `rvest` - Parse HTML document with `read_html()` ```r R> library(rvest) R> parsed_doc <- read_html("materials/fortunes.html") R> parsed_doc ``` ``` ## {html_document} ## <html> ## [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ... ## [2] <body>\n<div id="R Inventor" lang="english" date="June/2003">\n <h1>Robe ... ``` - Query document using `html_elements()` - `rvest` can process XPath queries as well as CSS selectors. - Today, we'll focus on XPath: ```r R> html_elements(parsed_doc, xpath = "//div[last()]/p/i") ``` ``` ## {xml_nodeset (1)} ## [1] <i>'R is wonderful, but it cannot work magic'</i> ``` --- # Grammar of XPath ### Basic rules 1. We access nodes/elements by writing down the hierarchical structure in the DOM that locates the element set of interest. 2. A sequence of nodes is separated by `/`. 3. The easiest localization of a element is given by the absolute path (but often not the most efficient one!). 4. Apply XPath on DOM in R using `html_elements()`. ```r R> html_elements(parsed_doc, xpath = "//div[last()]/p/i") ``` ``` ## {xml_nodeset (1)} ## [1] <i>'R is wonderful, but it cannot work magic'</i> ``` --- # Grammar of XPath ### Absolute vs. relative paths **Absolute paths** start at the root element and follow the whole way down to the target element (with simple slashes, `/`). ```r R> html_elements(parsed_doc, xpath = "/html/body/div/p/i") ``` ``` ## {xml_nodeset (2)} ## [1] <i>'What we have is nice, but we need something very different'</i> ## [2] <i>'R is wonderful, but it cannot work magic'</i> ``` **Relative paths** skip nodes (with double slashes, `//`). ```r R> html_elements(parsed_doc, xpath = "//body//p/i") ``` ``` ## {xml_nodeset (2)} ## [1] <i>'What we have is nice, but we need something very different'</i> ## [2] <i>'R is wonderful, but it cannot work magic'</i> ``` Relative paths are often preferrable. They are faster to write and more comprehensive. On the other hand, they are less targeted and therefore potentially less robust, and running them takes more computing time, as the entire tree has to be evaluated. But that's usually not relevant for reasonably small documents. --- # Grammar of XPath ### The wildcard operator - Meta symbol `*` - Matches any element - Works only for one arbitrary element - Far less important than, e.g., wildcards in content-based queries (regex!) ```r R> html_elements(parsed_doc, xpath = "/html/body/div/*/i") ``` ``` ## {xml_nodeset (2)} ## [1] <i>'What we have is nice, but we need something very different'</i> ## [2] <i>'R is wonderful, but it cannot work magic'</i> ``` ```r R> # the following does not work: R> html_elements(parsed_doc, xpath = "/html/body/div/*/i") ``` ``` ## {xml_nodeset (2)} ## [1] <i>'What we have is nice, but we need something very different'</i> ## [2] <i>'R is wonderful, but it cannot work magic'</i> ``` --- # Grammar of XPath ### Navigational operators `"."`and `".."` - `"."` accesses elements on the same level ("self axis"), which is useful when working with predicates (see later!). - `".."` accesses elements at a higher hierarchical level. ```r R> html_elements(parsed_doc, xpath = "//title/..") ``` ``` ## {xml_nodeset (1)} ## [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ... ``` ```r R> html_elements(parsed_doc, xpath = "//div[starts-with(./@id, 'R')]") ``` ``` ## {xml_nodeset (1)} ## [1] <div id="R Inventor" lang="english" date="June/2003">\n <h1>Robert Gentl ... ``` --- # Element (node) relations ("axes") in XPath .pull-left[ ### Family relations between elements - The tools learned so far are sometimes not sufficient to access specific elements without accessing other, undesired elements as well. - Relationship statuses are useful to establish unambiguity. - Can be combined with other elements of the grammar - Basic syntax: `element1/relation::element2` - We describe relation of `element2` to `element1` - `element2` is to be extracted - we always extract the element at the end! ] .pull-right[

 ] --- # Element (node) relations in XPath |
Axis name
 | Description | |---|---| | `ancestor` | All ancestors (parent, grandparent etc.) of the current element | | `ancestor-or-self` | All ancestors of the current element and the current element itself | | `attribute` | All attributes of the current element | | `child` | All children of the current element | | `descendant` | All descendants (children, grandchildren etc.) of the current element | | `descendant-or-self` | All descendants of the current element and the current element itself | | `following` | Everything in the document after the closing tag of the current element | | `following-sibling` | All siblings after the current element | | `parent` | The parent of the current element | | `preceding` | All elements that appear before the current element, except ancestors/attribute elements | | `preceding-sibling` | All siblings before the current element | | `self` | The current element | --- # Element (node) relations in XPath Example: access the `<div>` elements that are ancestors to an `<a>` element: ```r R> html_elements(parsed_doc, xpath = "//a/ancestor::div") ``` ``` ## {xml_nodeset (1)} ## [1] <div lang="english" date="October/2011">\n <h1>Rolf Turner</h1>\n <p><i ... ``` Another example: Select all `<h1>` nodes that precede a `<p>` node: ```r R> html_elements(parsed_doc, xpath = "//p/preceding-sibling::h1") ``` ``` ## {xml_nodeset (2)} ## [1] <h1>Robert Gentleman</h1> ## [2] <h1>Rolf Turner</h1> ``` --- # Predicates ### What are predicates? - Predicates are conditions based on an element's features (`true/false`). - Think of them as ways to filter nodesets. - They are applicable to a variety of features: name, value attribute. - Basic syntax: `element[predicate]` Select all first `<p>` elements that are children of a `<div>` element, using a **numeric predicate**: ```r R> html_elements(parsed_doc, xpath = "//div/p[1]") ``` ``` ## {xml_nodeset (2)} ## [1] <p><i>'What we have is nice, but we need something very different'</i></p> ## [2] <p><i>'R is wonderful, but it cannot work magic'</i> <br><emph>answering ... ``` -- Can you find out what the following expressions do? ```r R> html_elements(parsed_doc, xpath = "//div/p[last()-1]") R> html_elements(parsed_doc, xpath = "//div[count(./@*)>2]") R> html_elements(parsed_doc, xpath = "//*[string-length(text())>50]") ``` --- # Predicates (cont.) Select all `<div>` nodes that contain an attribute named `’October/2011’`, using a **textual predicate**: ```r R> html_elements(parsed_doc, xpath ="//div[@date='October/2011']") ``` ``` ## {xml_nodeset (1)} ## [1] <div lang="english" date="October/2011">\n <h1>Rolf Turner</h1>\n <p><i ... ``` Rudimentary string matching is also possible using string functions like `contains()`, `starts-with()`, or `ends-with()`. -- Can you tell what the following calls do? ```r R> html_elements(parsed_doc, xpath = "//div[starts-with(./@id, 'R')]") R> html_elements(parsed_doc, xpath = "//div[substring-after(./@date, '/')='2003']//i") ``` --- # Content extraction - Until now, we used XPath expressions to extract complete nodes or nodesets (that is, elements with tags). - However, in most cases we're interested in extracting the content only. - To that end, we can use extractor functions that are applied on the output of XPath query calls. | Function | Argument | Return value | |---|---|---| | `html_text()` | | Element value | | `html_text2()` | | Element value (with a bit more cleanup) | | `html_attr()` | `name` | Element attribute | | `html_attrs()` | | (All) element attributes | | `html_name()` | `trim` | Element name | | `html_children()` | | Element children | --- # Content extraction (cont.) Extracting **element values/content**: ```r R> html_elements(parsed_doc, xpath = "//title") %>% html_text2() ``` ``` ## [1] "Collected R wisdoms" ``` Extracting **attributes**: ```r R> html_elements(parsed_doc, xpath = "//div[1]") %>% html_attrs() ``` ``` ## [[1]] ## id lang date ## "R Inventor" "english" "June/2003" ``` Extracting **attribute values**: ```r R> html_elements(parsed_doc, xpath = "//div") %>% html_attr("lang") ``` ``` ## [1] "english" "english" ``` --- # More XPath? ### Training resources - XPath is a little language of its own. As always with languages, mastery comes with practice. - A good environment for practice is the [XPath expression testbed at whitebeam.org](http://www.whitebeam.org/library/guide/TechNotes/xpathtestbed.rhtm). - Also check out this [cheat sheet](https://devhints.io/xpath). ### XPath creator tools - Now, do you really have to construct XPath expressions by your own? No! At least not always. - **SelectorGadget**: [http://selectorgadget.com](http://selectorgadget.com) is a browser plugin that constructs XPath statements via a point-and-click approach. The generated expressions are not always efficient and effective though (more on this later). - Web developer tools - the internal browser functionality to study the DOM, among other things, also lets you extract XPath statements for selected nodes. These are specific to unique nodes/elements though, and therefore less helpful to extract node sets. (But they come in handy when we want to script live navigation, e.g. for Selenium.) --- class: inverse, center, middle name: css # CSS basics

 --- # What is CSS? .pull-left[ ### Background - **C**ascading **S**tyle **S**heets (CSS) is a style sheet language that allows web developers to adjust the "look and feel" of websites. - By using CSS to adjust style features such as layout, colors, and fonts, it's easier to separate content (HTML) from presentation (CSS). ### Three ways to insert CSS into HTML 1. **External CSS.** Inside `<head>` with a reference to the external file inside the `<link>` element. 2. **Internal CSS.** Inside `<head>` and stored in `<style>` elements. 3. **Inline CSS.** Inside `<body>` using the `style` attribute of elements. ] -- .pull-right[ **External CSS** ```html <head> <link rel="stylesheet" href="mystyle.css"> </head> ``` **Internal CSS** ```html <head> <style> h1 { color: red; margin-left: 20px; } </style> </head> ``` **Inline CSS** ```html <p style="color: blue;">This is a paragraph.</p> ``` ] --- # CSS selectors .pull-left[ ### Selectors - CSS selectors find/select the HTML elements that should be styled. - There are various categories of selectors. In addition to generic element selectors (which selected just based on the element name, such as `<p>`), we often care about: - **CSS id selectors**, which use the `id` attribute of an HTML element. Think of them as "labels", as in `<p id="para1">`. The respective CSS selector would be `#para1`. - **CSS class selectors**, which use the `class` attribute of an HTML element, as in `<p class = "center large">`. Note that these can refer to more than one class (here: `center` and `large`). The respective CSS selector would be `p.center.large`. ] -- .pull-right[ ### Writing CSS selectors - Just as XPath, CSS selectors are a little language of their own. - I won't teach you more about it, but you might nevertheless want to learn it. - Check out the CSS diner tutorial at https://flukeout.github.io/. It's one of the best tutorials of anything out there.

 ] --- class: inverse, center, middle name: scrapingstatic # Scraping static webpages with R

 --- # The scraping workflow .pull-left[ ### Key tools for scraping static webpages 1. You are able to inspect HTML pages in your browser using the web developer tools. 2. You are able to parse HTML into R with `rvest`. 3. You are able to speak XPath (or CSS selectors). 4. You are able to apply XPath expressions with `rvest`. 5. You are able to tidy web data with R/`dplyr`/`regex`. ### The big picture - Every scraping project is different, but the coding pipeline is fundamentally similar. - The (technically) hardest steps are location (XPath, CSS selectors) and extraction (clean-up), sometimes the scaling (from one to multiple sources). ] .pull-right[

 ] --- # Web scraping with rvest .pull-left-wide[ `rvest` is a suite of scraping tools. It is part of the tidyverse and has made scraping with R much more convenient. There are three key `rvest` verbs that you need to learn.1 1. `read_html()`: Read (parsing) an HTML resource. 2. `html_elements()`: Find elements that match a CSS selector or XPath expression. 3. `html_text2()`: Extract the text/value inside the node set. ] .footnote[ 1 There is more in `rvest` than what we can cover today. Have a glimpse at the [overview at tidyverse.org](https://rvest.tidyverse.org/) and at this excellent (unofficial) [cheat sheet](https://github.com/yusuzech/r-web-scraping-cheat-sheet). ] .pull-right-small-center[

 ] --- # Web scraping with rvest: example .pull-left-vsmall[ - We are going to scrape a information from a Wikipedia article on women philosophers available at [https://en.wikipedia.org/wiki/](https://en.wikipedia.org/wiki/List_of_women_philosophers) [List_of_women_philosophers](https://en.wikipedia.org/wiki/List_of_women_philosophers). - The article provides two types of lists - one by period and one sorted alphabetically. We want the alphabetical list. - The information we are actually interested in - names - is stored in unordered list elements. ] .pull-right-vwide[

 ] --- # Scraping HTML tables: example (cont.) **Step 1:** Parse the page ```r R> url_p <- read_html("https://en.wikipedia.org/wiki/List_of_women_philosophers") ``` -- **Step 2:** Develop an XPath expression (or multiple) that select the information of interest and apply it ```r R> elements_set <- html_elements(url_p, xpath = "//h2/span[text()='Alphabetically']//following::li/a[1]") ``` -- The XPath expression reads: - `//h2`: Look for `h2` elements anywhere in the document. - `/span[text()='Alphabetically']`: Within that element look for `span` elements with the content `"Alphabetically"`. - `//following::li`: In the DOM tree following that element (at any level), look for `li` elements. - `/a[1]` within these elements look for the first `a` element you can find. --- # Scraping HTML tables: example (cont.) **Step 3:** Extract information and clean it up ```r R> phil_names <- elements_set %>% html_text2() R> phil_names[c(1:2, 101:102)] ``` ``` ## [1] "A" "B" "Elisabeth of Bohemia" ## [4] "Dorothy Emmet" ``` -- **Step 4:** Clean up (here: select the subset of links we care about) ```r R> names_iffer <- + seq_along(phil_names) >= seq_along(phil_names)[str_detect(phil_names, "Felicia Nimue Ackerman")] & + seq_along(phil_names) <= seq_along(phil_names)[str_detect(phil_names, "Alenka Zupančič")] R> philosopher_names_clean <- phil_names[names_iffer] R> length(philosopher_names_clean) ``` ``` ## [1] 267 ``` ```r R> philosopher_names_clean[1:5] ``` ``` ## [1] "Felicia Nimue Ackerman" "Marilyn McCord Adams" "Aedesia" ## [4] "Alia Al-Saji" "Lilli Alanen" ``` --- # Quick-n-dirty static webscraping with SelectorGadget .pull-left[ ### The hassle with XPath - The most cumbersome part of web scraping (data tidying aside) is the construction of XPath expressions that match the components of a page you want to extract. - It will take a couple of scraping projects until you’ll truly have mastered XPath. ### A much-appreciated helper - **SelectorGadget** is a JavaScript browser plugin that constructs XPath statements (or CSS selectors) via a point-and-click approach. - It is available here: http://selectorgadget.com/ (there is also a Chrome extension). - The tool is magic and you will love it. ] -- .pull-right[ ### What does SelectorGadget do? - You activate the tool on any webpage you want to scrape. - Based on your selection of components, the tool learns about your desired components and generates an XPath expression (or CSS selector) for you. ### Under the hood - Based on your selection(s), the tool looks for similar elements on the page. - The underlying algorithm, which draws on Google’s diff-match-patch libraries, focuses on CSS characteristics, such as tag names and `<div>` and `<span>` attributes. ] --- # SelectorGadget: example <video width="980" height="551" controls> <source src="pics/selectorgadget-nytimes.mp4" type="video/mp4"> Your browser does not support the video tag. </video> --- # SelectorGadget: example (cont.) ```r R> library(rvest) R> url_p <- read_html("https://www.nytimes.com") R> # xpath: paste the expression from Selectorgadget! R> # note: we use single quotation marks here (' instead of ") to wrap around the expression! R> xpath <- '//*[contains(concat( " ", @class, " " ), concat( " ", "erslblw0", " " ))]//*[contains(concat( " ", @class, " " ), concat( " ", "e1lsht870", " " ))]' R> headlines <- html_elements(url_p, xpath = xpath) R> headlines_raw <- html_text(headlines) R> length(headlines_raw) R> head(headlines_raw) ``` ``` ## [1] 29 ``` ``` ## [1] "Retailers’ Latest Headache: Shutdowns at Their Vietnamese SuppliersRetailers’ Latest Headache: Shutdowns at Their Vietnamese Suppliers" ## [2] "With virus restrictions waning, it’s becoming clear: Britain’s gas crisis is a Brexit crisis, too. Here’s why." ## [3] "Business updates: U.S. stock futures signaled a rebound as bond yields fell back." ## [4] "Republicans at Odds Over Infrastructure Bill as Vote ApproachesRepublicans at Odds Over Infrastructure Bill as Vote Approaches" ## [5] "Liberals Dig In Against Infrastructure Bill as Party Divisions Persist" ## [6] "Successful programs from around the world could guide Congress in designing a paid family leave plan." ``` --- # SelectorGadget: when to use and not to use it Having learned about a semi-automated approach to generating XPath expressions, you might ask: **Why bother with learning XPath at all?** Well... - SelectorGadget is not perfect. Sometimes, the algorithm will fail. - Starting from a different element sometimes (but not always!) helps. - Often the generated expressions are unnecessarily complex and therefore difficult to debug. - In my experience, SelectorGadget works 50-60% of the times when scraping from static webpages. - You are also prepared for the remaining 40-50%! --- # Scraping HTML tables

 --- # Scraping HTML tables - HTML tables are everywhere. - They are easy to spot in the wild - just look for `<table>` tags! - Exactly because scraping tables is an easy and repetitive task, there is a dedicated `rvest` function for it: `html_table()`. .pull-left-vsmall[

 **Function definition** ```r R> html_table(x, + header = NA, + trim = TRUE, + dec = ".", + na.strings = "NA", + convert = TRUE + ) ``` ] .pull-right-vwide[
 | Argument | Description | |---|---| | `x` | Document (from `read_html()`) or node set (from `html_elements()`). | | `header` | Use first row as header? If `NA`, will use first row if it consists of `<th>` tags. | | `trim` | Remove leading and trailing whitespace within each cell? | | `dec` | The character used as decimal place marker. | | `na.strings` | Character vector of values that will be converted to `NA` if `convert` is `TRUE`. | | `convert` | If `TRUE`, will run `type.convert()` to interpret texts as int, dbl, or `NA`. | ] --- # Scraping HTML tables: example .pull-left-small[ - We are going to scrape a small table from the Wikipedia page [https://en.wikipedia.org/wiki/](https://en.wikipedia.org/wiki/List_of_human_spaceflights) [List_of_human_spaceflights](https://en.wikipedia.org/wiki/List_of_human_spaceflights). - (Note that we're actually using an old version of the page (dating back to May 1, 2018), which is accessible [here](https://en.wikipedia.org/w/index.php?title=List_of_human_spaceflights&oldid=778165808). Wikipedia pages change, but this old revision and associated link won't.)) - The table is not entirely clean: There are some empty cells, but also images and links. - The HTML code looks straightforward though. ] .pull-right-wide[

 ] --- # Scraping HTML tables: example (cont.) ```r R> library(rvest) R> url <- "https://en.wikipedia.org/wiki/List_of_human_spaceflights" R> url_p <- read_html(url) R> tables <- html_table(url_p, header = TRUE) R> spaceflights <- tables[[1]] R> spaceflights ``` ``` ## # A tibble: 7 × 5 ## `` `Russia Soviet Union` `United States` China Total ## <chr> <chr> <chr> <int> <chr> ## 1 1961–1970 16 25 NA 41 ## 2 1971–1980 30 8 NA 38 ## 3 1981–1990 *25 *38 NA *63 ## 4 1991–2000 20 63 NA 83 ## 5 2001–2010 24 34 3 61 ## 6 2011–2020 24 3 3 30 ## 7 Total *139 *171 6 *316 ``` --- class: inverse, center, middle name: goodpractice # Web scraping: good practice

 --- # Scraping: the rules of the game
 1. You take all the responsibility for your web scraping work. 2. Think about the nature of the data. Does it entail sensitive information? Do not collect personal data without explicit permission. 3. Take all copyrights of a country’s jurisdiction into account. If you publish data, do not commit copyright fraud. 4. If possible, stay identifiable. Stay polite. Stay friendly. Obey the scraping etiquette. 5. If in doubt, ask the author/creator/provider of data for permission—if your interest is entirely scientific, chances aren’t bad that you get data. --- # Consult robots.txt .pull-left[ ### What's robots.txt? - "[Robots exclusion standard](https://en.wikipedia.org/wiki/Robots_exclusion_standard)", informal protocol to prohibit web robots from crawling content - Located in the root directory of a website (e.g., [google.com/robots.txt](https://www.google.com/robots.txt)) - Documents which bot is allowed to crawl which resources (and which not) - Not a technical barrier, but a sign that asks for compliance ### What's robots.txt? - Not an official W3C standard - Rules listed bot by bot - General rule listed under `User-agent: *` (most interesting entry for R-based crawlers) - Directories folders listed separately ] .pull-right[ **Example** ```txt User-agent: Googlebot Disallow: /images/ Disallow: /private/ ``` **Universal ban** ```txt User-agent: * Disallow: / ``` **Allow declaration** ```txt User-agent: * Disallow: /images/ Allow: /images/public/ ``` **Crawl delay (in seconds)** ```txt User-agent: * Crawl-delay: 2 ``` ] --- # Downloading HTML files .pull-left[ ### Stay modest when accessing lots of data - Content on the web is publicly available. - But accessing the data causes server traffic. - Stay polite by querying resources as sparsely as possible. ### Two easy-to-implement practices 1. Do not bombard the server with requests - and if you have to, do so at modest pace. 2. Store web data on your local drive first, then parse. ] .pull-right[ ### Looping over a list of URLs ```r R> for (i in 1:length(list_of_urls)) { + if (!file.exists(paste0(folder, file_names[i]))) { + download.file(list_of_urls[i], + destfile = paste0(folder, file_names[i]) + ) + Sys.sleep(runif(1, 1, 2)) + } + } ``` - `!file.exists()` checks whether a file does not exist in the specified location. - `download.file()` downloads the file to a folder. The destination file (location + name) has to be specified. - `Sys.sleep()` suspends the execution of R code for a given time interval (in seconds). ] --- # Staying identifiable .pull-left[ ### Don't be a phantom - Downloading massive amounts of data may arouse attention from server administrators. - Assuming that you've got nothing to hide, you should stay identifiable beyond your IP address. ### Two easy-to-implement practices 1. Get in touch with website administrators / data owners. 2. Use HTTP header fields `From` and `User-Agent` to provide information about yourself. ] .pull-right[ ### Staying identifiable in practice ```r R> url <- "http://a-totally-random-website.com" R> rvest_session <- session(url, + add_headers(From =<EMAIL_ADDRESS>+ `UserAgent` = + R.Version()$version.string + ) + ) R> headlines <- rvest_session %>% + html_elements(xpath = "p//a") %>% + html_text() ``` - `rvest`'s `session()` creates a session object that responds to HTTP and HTML methods. - Here, we provide our email address and the current R version as `User-Agent` information. - This will pop up in the server logs: The webpage administrator has the chance to easily get in touch with you. ] --- # Scraping etiquette (cont.)

 --- class: inverse, center, middle name: summary # Summary

 --- # Outlook Until now, the toy examples were limited to single HTML pages. However, often we want to **scrape data from multiple pages**. You might think of newspaper articles, Wikipedia pages, shopping items and the like. In such scenarios, automating the scraping process becomes really powerful. Also, principles of polite scraping are more relevant then. In other cases, you might be confronted with - forms, - authentication, - dynamic (JavaScript-enriched) content, or want to - automatically navigate through pages interactively. Moreover, we've ignored a major alternative way to collect data from the web so far which goes beyond scraping: accessing [web APIs](https://en.wikipedia.org/wiki/Web_API). Be sure to check out the respective sessions in the workshop. There's only so much we can cover in one session. Check out more material online [here](https://github.com/hertie-data-science-lab/ds-workshop-webscraping) and [there](https://github.com/yusuzech/r-web-scraping-cheat-sheet) to learn about solutions to some of these problems. --- # Coming up

 ### Assignment Assignment 3 is about to go online on GitHub Classroom. Check it out and start scraping the web (politely). ### Next lecture Model fitting and simulation. Now that we know how to retrieve data, let's learn how to run and learn from them.


Multiprizer uses the most sophisticated Provable Random Number currently known, to generate winner - Ledger based random of Oraclize, with on-chain proof verification. This creates purest 'random' to select winners from the strategic lot. Read More
This DApp is built with game parameters such as DirectPlay and a fixed significant house edge, which make sure that the game cannot be controlled or manipulated by any party – neither malicious players, nor rogue miners, not even the admin or creator of this game. Read More
The game tokens, once purchased, can be undone by the ‘revert’ feature in which all the player's purchased tokens are cancelled, and the token value refunded to the player. This is applicable at any point in time as long as the timer for that round has not expired.
Players don’t even have to visit the DApp site to play. One can send a token value multiple of a specific game, from their Ethereum Wallet to the Contract Address, and the player participation gets auto-confirmed. Read More
Games are designed such that there is a lot of leverage on how you play, rather than dependence on chance, to maximizes your outcome of victory. The per token win probability itself is extremely high, teamed with option to purchase and revert at will, brings strategy as key to victory. Read More
The Smart Contract follows best-in-class secure methodology derived from various top safety standards such as Consensys, OpenZeppelin etc. which prevents attacks such as re-entrancy, overflow/underflow etc. The contract is also publicly verified, providing full transparency of game algorithm. Read More
The MegaPrize uses part of admin’s edge to create a lucky draw for participants who have played any Game Board at least once, without reverting. The winner is again chosen using provable random like game rounds & MegaPrize amount automatically credited. Read More
The games cannot be obstructed in any manner while the round is still on. The game locks can only be implemented by admin on completion of game rounds, so does contract kill. Also, to maintain consistent win probability, new game boards are added instead of increasing total tokens for a specific game.


Reglas de Escaneo pasivo - Beta
General Configuration
Trusted Domains
You can specify a comma separated list of URL regex patterns using the rules.domains.trusted
parameter via the Options 'Rule configuration' panel.
Any link URL that matches one of these patterns will be considered to be in a trusted domain and will therefore not be reported.
Following rules supports Trusted Domains :
- Sub Resource Integrity Attribute Missing
The following beta status passive scan rules are included in this add-on:
Content Cacheability
This scan rule analyzes the cache control and pragma headers in HTTP traffic and reports on the cacheability of the requests from a RFC7234 point of view.
Alerts generated:
- Non-Storable Content
- Storable but Non-Cacheable Content
- Storable and Cacheable Content
Latest code: CacheableScanRule.java
Dangerous JS Functions
This scan rule checks for any dangerous JS functions present in a site response.
Note: If the Custom Payloads addon is installed you can add your own function names (payloads) in the Custom Payloads options panel.
They will also be searched for in responses as they're passively scanned. Keep in mind that the greater the number of payloads the greater the amount of time needed to passively scan.
Note: $ is stripped from the start of the strings/payloads and is optionally included when the patterns are assembled.
Latest code: JsFunctionScanRule.java
In Page Banner Information Leak
Analyzes response body content for the presence of web or application server banners (when the responses have error status codes).
If the Threshold is Low then status 200 - Ok responses are analyzed as well.
The presence of such banners may facilitate more targeted attacks against known vulnerabilities.
Latest code: InPageBannerInfoLeakScanRule.java
Java Serialization Object
Java Serialization Object (JSO) is a way to save and exchange objects between Java applications.
Different problems are associated with JSO. Sensitive data can leak to the stream of bytes.
An attacker can also modify the data and exploit JSO to do a Remote Code Execution on the server.
JSO should not be used by Java programs. Strong controls must be done on serialized data.
JSO are a type of vulnerabilities associated to A8:2017-Insecure Deserialization.
Latest code: JsoScanRule.java
Permissions Policy Header Not Set
This rule checks the HTTP response headers (on HTML and JavaScript responses) for inclusion of a "Permissions-Policy" header,
and alerts if one is not found. It also alerts if the deprecated header "Feature-Policy" is found.
Redirects are ignored except at the Low threshold.
Latest code: PermissionsPolicyScanRule.java
Site Isolation Scan Rule
Spectre is a side-channel attack allowing an attacker to read data
from memory. One of the counter-measures is to prevent sensitive data
from entering the memory and to separate trusted and untrusted documents in
different browsing contexts. Three headers have been defined to enable that:
- Cross-Origin-Resource-Policy
- Cross-Origin-Embedder-Policy
- Cross-Origin-Opener-Policy
The Cross-Origin-Embedder-Policy (COEP) header prevents a document from
loading any non-same-origin resources which don't explicitly grant the document
permission to be loaded.
(from
COOP and COEP explained).
The Cross-Origin-Resource-Policy (CORP) header allows you
to control the set of origins that are empowered to include
a resource. It is a robust defense against attacks like Spectre,
as it allows browsers to block a given response before it enters
an attacker's process.
For example, an attacker site can include an image tag with an attribute
src to an internal content. The browser will load the data. With a
side-channel attack, an attacker will be able to read it.
The Cross-Origin-Opener-Policy (COOP) header forces the browser to create
multiple browsing contexts to separate trusted and untrusted documents.
Site Isolation is complementary with Cross-Origin-Resource-Blocking, a
mechanism managed independently by the browser.
Alerts generated:
- Cross-Origin-Resource-Policy Header Missing or Invalid
- Cross-Origin-Embedder-Policy Header Missing or Invalid
- Cross-Origin-Opener-Policy Header Missing or Invalid
Latest code: SiteIsolationScanRule.java
Servlet Parameter Pollution
Searches response content for HTML forms which fail to specify an action element. Version 3 of the
Java Servlet spec calls for aggregation of query string and post data elements which may result in
unintended handling of user controlled data. This may impact other frameworks and technologies as well.
Note: This scan rule will only analyze responses on LOW Threshold, and in Context URLs for which the Tech JSP/Servlet is applicable.
Latest code: ServletParameterPollutionScanRule.java
Source Code Disclosure
Application Source Code was disclosed by the web server.
NOTE: Ignores CSS, JavaScript, images, and font files.
Latest code: SourceCodeDisclosureScanRule.java
Sub Resource Integrity Attribute Missing
This rule checks whether the integrity attribute in the script or the link element served by an external resource (for example: CDN) is missing.
It helps mitigate an attack where the CDN has been compromised and content has been replaced by malicious content.
Note: A suggested integrity hash value will be present in the relevant Alert's Other Info details if it can be resolved to a script in the Sites Tree.
This rule supports Trusted Domains, check "General Configuration" for more information.
Latest code: SubResourceIntegrityAttributeScanRule.java


| Position | Prizes |
| Sealed leagues |
| 1st | 60 boosters, 4 tengwar, 3 foil rares |
| 2nd | 55 boosters, 3 tengwar, 2 foil rares |
| 3rd | 50 boosters, 2 tengwar, 1 foil rare |
| 4th | 45 boosters, 1 tengwar |
| 5th-8th | 40 boosters |
| 9th-16th | 35 boosters |
| 17th-32nd | 20 boosters |
| 33rd-64th | 10 boosters |
| 65th-128th | 5 boosters |
| Collector's leagues |
| 1st | 30 boosters, 4 tengwar, 3 foil rares |
| 2nd | 25 boosters, 3 tengwar, 2 foil rares |
| 3rd | 20 boosters, 2 tengwar, 1 foil rares |
| 4th | 15 boosters, 1 tengwar |
| 5th-8th | 10 boosters, 2 promos |
| 9th-16th | 5 boosters, 1 promo |
| 17th-32nd | 2 boosters |
| Constructed leagues |
| 1st | 10 boosters, 4 tengwar, 3 foil rares |
| 2nd | 8 boosters, 3 tengwar, 2 foil rares |
| 3rd | 6 boosters, 2 tengwar, 1 foil rares |
| 4th | 4 boosters, 1 tengwar |
| 5th-8th | 3 boosters, 2 promos |
| 9th-16th | 2 boosters, 1 promo |
| 17th-32nd | 1 booster |


Struct HRESULT
The HRESULT is a data type used to represent error and warning conditions.
HRESULTs are numerical error codes. Various bits within an HRESULT encode information about the nature of the error code, and where it came from.
Namespace: THNETII.WinApi
Assembly: THNETII.WinApi.ErrorCodes.dll
Syntax
public struct HRESULT : IEquatable<HRESULT>, IEquatable<int>
Remarks
HRESULTs are 32 bit values layed out as follows:
3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
+-+-+-+-+-+---------------------+-------------------------------+
|S|R|C|N|r| Facility | Code |
+-+-+-+-+-+---------------------+-------------------------------+
where
| Field | Description | ||||||
|---|---|---|---|---|---|---|---|
S |
Severity – indicates success/fail
| ||||||
R | reserved portion of the facility code, corresponds to NT's second severity bit. | ||||||
C | reserved portion of the facility code, corresponds to NT's C field. | ||||||
N | reserved portion of the facility code. Used to indicate a mapped NT status value. | ||||||
r | reserved portion of the facility code. Reserved for internal use. Used to indicate HRESULT values that are not status values, but are instead message ids for display strings. | ||||||
| Facility | is the facility code | ||||||
| Code | is the facility's status code |
The HRESULT numbering space is vendor-extensible. Vendors can supply their own values for this field, as long as the C bit (0x20000000
) is set, indicating it is a customer code.
Constructors
| Improve this Doc View SourceHRESULT(Int32)
Initializes a new HRESULT with the specified integer value.
Declaration
public HRESULT(int code)
Parameters
| Type | Name | Description |
|---|---|---|
| Int32 | code | The HRESULT error code as a 32-bit integer value. |
HRESULT(UInt32)
Initializes a new HRESULT with the specified integer value.
Declaration
public HRESULT(uint code)
Parameters
| Type | Name | Description |
|---|---|---|
| UInt32 | code | The HRESULT error code as an unsigned 32-bit integer value. |
Properties
| Improve this Doc View SourceCode
Gets the the facility's status code.
Declaration
public int Code { get; }
Property Value
| Type | Description |
|---|---|
| Int32 | An integer value containing the status code that is specific to the Facility. |
Facility
Indicates the system service that is responsible for the error.
Declaration
public HResultFacility Facility { get; }
Property Value
| Type | Description |
|---|---|
| HResultFacility | A value from the HResultFacility enumeration. |
IsCustomerCode
Whether the HRESULT value is a customer-defined value.
Declaration
public bool IsCustomerCode { get; }
Property Value
| Type | Description |
|---|---|
| Boolean |
|
IsFailure
Whether the HRESULT value indicates a failed operation.
Declaration
public bool IsFailure { get; }
Property Value
| Type | Description |
|---|---|
| Boolean |
|
IsMessageId
Whether the HRESULT value is not a status value, but a message id for display string.
Declaration
public bool IsMessageId { get; }
Property Value
| Type | Description |
|---|---|
| Boolean |
|
IsNTMapped
Declaration
public bool IsNTMapped { get; }
Property Value
| Type | Description |
|---|---|
| Boolean |
|
IsSuccess
Whether the HRESULT value indicates a successful operation.
Declaration
public bool IsSuccess { get; }
Property Value
| Type | Description |
|---|---|
| Boolean |
|
NTStatusSeverity
Gets the severity for a mapped NTSTATUS value.
Declaration
public NTStatusSeverity NTStatusSeverity { get; }
Property Value
| Type | Description |
|---|---|
| NTStatusSeverity | A NTStatusSeverity value provide one extra bit of information compared to Severity. |
See Also
| Improve this Doc View SourceSeverity
Indicates success or failure
Declaration
public HResultSeverity Severity { get; }
Property Value
| Type | Description |
|---|---|
| HResultSeverity |
Value
The full 32-bit integer value of of the system error code.
Declaration
public int Value { get; }
Property Value
| Type | Description |
|---|---|
| Int32 | The entire value as a signed 32-bit integer. |
Methods
| Improve this Doc View SourceEquals(Int32)
Determines whether the current instance is equal to a 32-bit integer value.
Declaration
public bool Equals(int code)
Parameters
| Type | Name | Description |
|---|---|---|
| Int32 | code | The integer to compare with. |
Returns
| Type | Description |
|---|---|
| Boolean |
|
Equals(Object)
Determines whether the current instance is equal to a specified object.
Declaration
public override bool Equals(object obj)
Parameters
| Type | Name | Description |
|---|---|---|
| Object | obj | The object to compare with. |
Returns
| Type | Description |
|---|---|
| Boolean |
|
Equals(HRESULT)
Determines whether the current value is equal to another HRESULT value.
Declaration
public bool Equals(HRESULT hr)
Parameters
| Type | Name | Description |
|---|---|---|
| HRESULT | hr | The Windows Error code to compare with. |
Returns
| Type | Description |
|---|---|
| Boolean |
|
FromWin32(Int32)
Declaration
public static HRESULT FromWin32(int x)
Parameters
| Type | Name | Description |
|---|---|---|
| Int32 | x | The raw Win32 Error Code to convert. |
Returns
| Type | Description |
|---|---|
| HRESULT |
GetException()
Converts the HRESULT error code to a corresponding managed
Declaration
public Exception GetException()
Returns
| Type | Description |
|---|---|
| Exception | A regular .NET Exception instance if Value can be mapped to a .NET |
GetHashCode()
Returns the hash code for this instance.
Declaration
public override int GetHashCode()
Returns
| Type | Description |
|---|---|
| Int32 | The value obtained from calling |
ToString()
Returns a string representing the HRESULT as a 32-bit numeric value in capitalized prefixed hexadecimal notation.
Declaration
public override string ToString()
Returns
| Type | Description |
|---|---|
| String | Value formatted in capitalized 8 digit hexadecimal notation, prefixed with |
Operators
| Improve this Doc View SourceEquality(HRESULT, HRESULT)
Checks whether two HRESULT value are equal.
Declaration
public static bool operator ==(HRESULT left, HRESULT right)
Parameters
| Type | Name | Description |
|---|---|---|
| HRESULT | left | The left side operand of the comparison. |
| HRESULT | right | The right side operand of the comparison. |
Returns
| Type | Description |
|---|---|
| Boolean |
|
Explicit(HRESULT to Int32)
Casting operator converting the HRESULT to a 32-bit integer value.
Declaration
public static explicit operator int (HRESULT e)
Parameters
| Type | Name | Description |
|---|---|---|
| HRESULT | e | The error code to cast. |
Returns
| Type | Description |
|---|---|
| Int32 |
Explicit(Win32ErrorCode to HRESULT)
Declaration
public static explicit operator HRESULT(Win32ErrorCode x)
Parameters
| Type | Name | Description |
|---|---|---|
| Win32ErrorCode | x | The raw Win32 Error Code to cast. |
Returns
| Type | Description |
|---|---|
| HRESULT |
Implicit(Int32 to HRESULT)
Implicityly casts the 32-bit integer value to a HRESULT value.
Declaration
public static implicit operator HRESULT(int code)
Parameters
| Type | Name | Description |
|---|---|---|
| Int32 | code | The HRESULT error code as a 32-bit integer value. |
Returns
| Type | Description |
|---|---|
| HRESULT |
Implicit(UInt32 to HRESULT)
Implicityly casts the unsigned 32-bit integer value to a HRESULT value.
Declaration
public static implicit operator HRESULT(uint code)
Parameters
| Type | Name | Description |
|---|---|---|
| UInt32 | code | The HRESULT error code as an unsigned 32-bit integer value. |
Returns
| Type | Description |
|---|---|
| HRESULT |
Inequality(HRESULT, HRESULT)
Checks whether two HRESULT value are not equal.
Declaration
public static bool operator !=(HRESULT left, HRESULT right)
Parameters
| Type | Name | Description |
|---|---|---|
| HRESULT | left | The left side operand of the comparison. |
| HRESULT | right | The right side operand of the comparison. |
Returns
| Type | Description |
|---|---|
| Boolean |
|


NE
CHANGE
Tournament Widget Demo
Show tournament information on your website without a hassle.
Today


Preparation time
less than 30 mins
Cooking time
1 to 2 hours
Serves
Serves 4
This juicy roast pork belly recipe has crisp, crunchy crackling infused with rosemary and fennel. A luxurious and economical dinner party classic.
Pat the pork belly dry with kitchen paper and score the skin with a sharp kitchen knife (or DIY knife), if your butcher has not already done so. The scoring should be right across the width of the meat, in parallel lines close together. The scoring should go through the skin, into the fat, but not into the meat below.
Preheat the oven to 220C/425F/Gas 7.
Chop together the rosemary leaves, fennel seeds, pepper, lemon zest and salt. Rub the pork all over with the sunflower oil before rubbing the herb and salt mixture into the skin, pushing it in between the scored lines. Place the pork onto a rack sat in a roasting tin, skin-side up, and cook for 35-40 minutes or until the skin starts to puff and crisp at the edges.
Meanwhile, prepare the fennel. Remove the stalks of the fennel and set aside any feathery fronds. Slice the bulb into thin wedges. Flatten the garlic and remove the peel, leaving the cloves whole.
Remove the pork belly from the oven and turn the oven down to 160C/300F/Gas 2.  Set the rack back on top. Return the pork and fennel to the oven and roast for a further hour at 160C/300F/Gas 2.
Remove the fennel from the oven, when it is tender and golden brown (this will take approximately 30-40 minutes). Squeeze over some lemon juice, then cover and set aside in a warm place.
To finish the pork, turn the oven back up to 200C/400F/Gas 6. Cook the pork for another 20 minutes, or until the skin is very crisp. Slice the pork belly and serve the fennel alongside, sprinkling over the chopped reserved fennel fronds for garnish.


Trouble-shooting and Analysis
Calibration and "First Use" Problems
If
you are just starting to use PHD2 or are connecting to new equipment
for the first time, you may have trouble getting the guider calibration
done. This problem usually takes one of two forms, each requiring
different responses:
- The star moves during calibration but
it moves "too far" or "too little." This is easy to fix - simply
adjust the 'calibration step-size' parameter in the 'Mount' tab of Advanced Settings. The help content there describes how this parameter is used, and you should be able to resolve the problem quickly.
- The
star doesn't move at all or it moves only very small amounts and in
unpredictable ways. Actually, the small movements in this
scenario are often the result of seeing deflections and the mount isn't
really moving at all. Dealing with this sort of problem is
described next.
In
nearly all cases, the "no movement" problem is caused by failures in
the hardware or, even more likely, problems in the cabling and
connections. The best tool for trouble-shooting this is the
'Manual Guide' option under the 'Tools' menu, as described in the Tools
section of this help document. Simply use the directional
controls in the 'Manual Guide' window to send commands directly to the
mount. If the mount does not respond, you know you have either
hardware or connectivity problems to resolve - nothing to do with PHD2.
If you're using a Shoestring device to connect to the mount,
watch its indicator lights to see if the commands are reaching it.
Similarly, your ST-4 compatible guide camera may have indicator
lights to show when guide commands are being received. If you're
using an ASCOM connection to the mount, be sure the COM port
assignments are correct. You can also use some of the
ASCOM-supplied tools like POTH to be sure the ASCOM driver is
communicating correctly with the mount.
Calibration Sanity-Checks
It is also possible that the calibration process will complete but PHD2 will post a "calibration
sanity-check" alert message saying that some of the results are
questionable. This "sanity check" dialog will show an explanation
of the issue and some details of the calibration results:
In the initial release (2.4.0), there are four things checked by PHD2:
- Too
few steps (shown above) - resolving this issue is quite easy.
Just adjust the calibration step-size downward until you get at
least 8 steps in both the west and north calibrations.
- Non-orthogonal
camera axes - the camera axes are normally computed independently even
though they should
be perpendicular. The angle calculations do not require
great precision, but if they are signfiicantly non-orthogonal, you
should repeat the calibration. Alternatively, you can set the
option to'Assume Dec orthogonal to RA' in the 'Mount' tab of the
Advanced Setup dialog. The graph at the left shows the RA and Dec axis
orientation as measured by PHD2..
- Suspicious
RA and Dec rates - the guide rate for right ascension should be related
to the declination guiding rate by approximately a factor
of cosine(Declination). In other words, the RA rate gets
smaller as you move the scope further away from the celestial equator
(Dec=0). In this initial release, PHD2 won't try to identify
which rate is incorrect - it is simply alerting you that something
looks wrong with the rates. You can sanity check these rates
yourself quite simply. If you are guiding at 1X sidereal rate,
your declination guide rate should be approximately 15 arc-sec/sec;
with a guide rate of 0.5X sidereal, the declination rate would be 7.5 arc-sec/sec,
etc.
- Inconsistent results - if the calibration results are
significantly different from your last-used calibration, an alert
message will be generated. This may happen because you've made a
change in your configuration. That doesn't imply a real problem,
but you should probably consider creating a separate profile for the
new configuration. By doing so, PHD2 will remember settings
for each of your profiles, letting you switch between them easily.
If you haven't made a configuration change, you will probably
want to determine why the results are so different.
With any
of these alerts, the relevant data field will be highlighted based on the
type of message. You can choose to ignore the warning ('Accept
calibration'), re-run the calibration ('Discard calibration'), or
restore your last good calibration ('Restore old calibration').
With the third option, you can defer calibration until later and
start guiding with your last good calibration data. If you see
repeated alerts on the same topic and are convinced there really
isn't a problem, you can use the 'don't show' checkbox to block
future alerts of that type. Program "Hangs" and Camera Download Problems
In
some cases, you may experience problems where guider images aren't
downloaded or displayed. In extreme cases, this may even cause
PHD2 or other camera-related applications to be non-responsive (i.e. to
"hang"). Again, this is almost always due to hardware, camera
driver, or connectivity issues, with one of the most common
culprits being a faulty USB cable or device. It is highly
unlikely to be caused by an application like PHD2, so you should begin
your investigation at the lower levels of the system. You can start by
confirming that the guide camera is working - try using a short, direct
cable from the camera to the computer and taking exposures with the
"native" or test application that came with the camera. If the
camera is functional, you can start moving "upstream" by looking at USB
hubs and cables, swapping them one at a time to see if you can isolate
the problem. It's worth remembering that we work in a hostile
environment while doing our imaging, and many of the components we use
are not designed for cold, outdoor conditions. So something that worked
just last week or last month may no longer be reliable.Poor Guiding Performance
Once
you've gotten everything running, you will probably get reasonably good
guiding results almost immediately. You will have to decide what
"good enough" means, and everyone's standard is likely to be different.
But if you find your imaging results are not acceptable
because the stars are streaked or elongated, you'll need to take a
systematic approach to correcting the problems. This can be a
complex task and not something to be covered here. However, you
can get help on the web from a variety of sources, with the document by
Craig Stark being one of the best:
http://www.cloudynights.com/page/articles/cat/fishing-for-photons/what-to-do-when-phd-guiding-isnt-push-here-dummy-r2677.
Log Analysis
Any
sort of problem isolation or tuning will invariably require use of
the PHD2 log files. Both are formatted for straightforward
interpretation by a human reader, and the guide log is constructed to
enable easy import into other applications. Logging of debug and
guiding data is controlled by two menu items under "Tools", and these
should be left checked so that diagnostic information is available if
problems occur. As mentioned in the
'Tools' section, applications such as PHDLogViewer or Excel
can
be used to visualize overall performance, compute performance
statistics, and examine time periods when guiding was problematic.
With Excel or similar applications, simply specify that the guide
log uses a comma as a column separator.
Guiding Log Contents
The
contents of the guiding log will continue to evolve as new capabilities
are added. But the basic content is stable, and considerable care
is taken to not "break" applications that parse it. If you wish
to analyze the log yourself, the following information will be helpful.
The PHD2 guide log will contain zero or more sequences of
calibration and zero or more sequences of guiding. Each of these sections has a
header that provides most of the information about the guiding algorithms being
used and the internal parameters used by PHD2 for guiding. At the start of either a calibration run or
a guiding sequence, the last line of the header information defines a set of
column headings. The meanings of those
columns are shown below:
Calibration columns:
- dx, dy
are offsets from the starting position, in pixels, in the camera coordinate
system
- x, y are the camera x/y coordinates of the guide
star at the end of each calibration step
- Dist is the total distance moved in the camera
coordinate system (dist = sqrt(dx*dx + dy*dy).
This is the value used by PHD2 to compute the calibration parameters
Guiding columns:
- dx, dy are the same as for calibration - offsets
from the "lock position" of the guide star in the camera coordinate
system
- RARawDistance and DECRawDistance - these are the
transforms of dx and dy into the mount coordinates - in other words, they use
the arbitrary angle of the guide camera to map from X/Y on the camera to RA/Dec
on the mount
- RAGuideDistance and DECGuideDistance - these are
the outputs from the various guiding algorithms. The guide algorithms operate on the
"raw" distances and decide how far, if any, the telescope position
should be adjusted in each axis. For
example, with a "minimum move" parameter set, the "guide"
distances can be zero even when the "raw" distances are non-zero.
- RADuration, RADirection, DECDuration, DECDirection
- these are the values determined by the two "guide" distances
above. The "durations" are the
lengths of the guide pulses, in milliseconds, needed to move the mount by the
distances specified by RAGuideDistance and DECGuideDistance
- XStep, YStep - step-adjustment durations for the
adaptive optics device if one is being used
- StarMass - a brightness measure of the guide
star image
- SNR - an internal "signal-to-noise"
ratio used by PHD2
- ErrorCode - an indication that some error
occurred, typically a "guide star faded" error
All distance values are in units of pixels. The header for the guiding section will show
the image scale as it is known by PHD2, and that can be used to scale the pixel
distance values into units of arc-seconds if desired.
Problem Reporting
If
you encounter application problems that are specific to PHD2, you
are encouraged to report them to the open-phd-guiding Google group: https://groups.google.com/forum/?fromgroups=#!forum/open-phd-guiding.
Obviously, the more information you can provide, the more likely
we will be able to resolve the problem. Using the following
guidelines will help in that regard:
- Try to reproduce the
problem - if we have a clear set of steps to follow, we are more likely
to find a solution quickly. If you can reproduce it, try to
reduce things to the minimum number of steps. Remember, we won't
have your hardware or computer environment when we try to reproduce it
ourselves.
- Try to be complete about describing your configuration - operating system, equipment types, PHD2 version, etc.
- Attach
the PHD2
debug log from the session in which you encountered the
problem. You can find the debug log in the PHD2 folder in your
Documents folder. If you can't reproduce the problem, try to
estimate the
time of day when you first saw it - this could help us find evidence in
the debug log without having to sift through hundreds of lines of
output.


--- layout: page title: About header-img: "img/about-bg.png" ---
Love gaming, walks in the park, a good laugh, a good book and just fun!


Toggle navigation
PBL Science - Science Project
Home
Reactions
Reactions
Presentation
If you see this, please enable javascript.


Reporting Status
Overall Reporting Status 244 indicators
198Exploring data sources81%
0Statistics in progress0%
46Reported online19%
Status by goal
Good health and well-being 27 indicators
18Exploring data sources67%
0Statistics in progress0%
9Reported online33%
Quality education 11 indicators
3Exploring data sources27%
0Statistics in progress0%
8Reported online73%
Gender equality 14 indicators
10Exploring data sources71%
0Statistics in progress0%
4Reported online29%
Clean water and sanitation 11 indicators
10Exploring data sources91%
0Statistics in progress0%
1Reported online9%
Affordable and clean energy 6 indicators
4Exploring data sources67%
0Statistics in progress0%
2Reported online33%
Decent jobs and economic growth 17 indicators
13Exploring data sources76%
0Statistics in progress0%
4Reported online24%
Industry, innovation and infrastructure 12 indicators
10Exploring data sources83%
0Statistics in progress0%
2Reported online17%
Reduced inequalities 11 indicators
10Exploring data sources91%
0Statistics in progress0%
1Reported online9%
Sustainable cities & communities 15 indicators
14Exploring data sources93%
0Statistics in progress0%
1Reported online7%
Responsible consumption and production 13 indicators
13Exploring data sources100%
0Statistics in progress0%
0Reported online0%
Climate action 8 indicators
7Exploring data sources88%
0Statistics in progress0%
1Reported online12%
Life below water 10 indicators
10Exploring data sources100%
0Statistics in progress0%
0Reported online0%
Peace and justice - strong institutions 23 indicators
21Exploring data sources91%
0Statistics in progress0%
2Reported online9%
Partnerships for the goals 25 indicators
23Exploring data sources92%
0Statistics in progress0%
2Reported online8%


Entree
Quick
Local Fav.
Ingredients
- 3 to 6 pieces boneless/skinless chicken breasts cut into 1/4 inch cubes
- 1 tablespoon olive oil
- 2 tablespoons favorite BBQ sauce (we use a spicy-sweet sauce)
- 1 basic pizza dough
- etc. etc. etc...
Entree
Quick
Local Fav.
Ingredients
- 3 to 6 pieces boneless/skinless chicken breasts cut into 1/4 inch cubes
- 1 tablespoon olive oil
- 2 tablespoons favorite BBQ sauce (we use a spicy-sweet sauce)
- 1 basic pizza dough
- etc. etc. etc...
Entree
Quick
Local Fav.
Ingredients
- 3 to 6 pieces boneless/skinless chicken breasts cut into 1/4 inch cubes
- 1 tablespoon olive oil
- 2 tablespoons favorite BBQ sauce (we use a spicy-sweet sauce)
- 1 basic pizza dough
- etc. etc. etc...
Entree
Quick
Local Fav.
Ingredients
- 3 to 6 pieces boneless/skinless chicken breasts cut into 1/4 inch cubes
- 1 tablespoon olive oil
- 2 tablespoons favorite BBQ sauce (we use a spicy-sweet sauce)
- 1 basic pizza dough
- etc. etc. etc...
Entree
Quick
Local Fav.
Ingredients
- 3 to 6 pieces boneless/skinless chicken breasts cut into 1/4 inch cubes
- 1 tablespoon olive oil
- 2 tablespoons favorite BBQ sauce (we use a spicy-sweet sauce)
- 1 basic pizza dough
- etc. etc. etc...
Entree
Quick
Local Fav.
Ingredients
- 3 to 6 pieces boneless/skinless chicken breasts cut into 1/4 inch cubes
- 1 tablespoon olive oil
- 2 tablespoons favorite BBQ sauce (we use a spicy-sweet sauce)
- 1 basic pizza dough
- etc. etc. etc...
Entree
Quick
Local Fav.
Ingredients
- 3 to 6 pieces boneless/skinless chicken breasts cut into 1/4 inch cubes
- 1 tablespoon olive oil
- 2 tablespoons favorite BBQ sauce (we use a spicy-sweet sauce)
- 1 basic pizza dough
- etc. etc. etc...


Please keep the following things in mind when working with the samples:
|
Category |
Samples |
|---|---|
| Introduction |
HTTP Responses HTTP Spy |
| Intermediate |
Sample Server SMTP Client |
| Advanced |
FTP Sync Virtual FTP |
| Name | Versions | Category | Description |
|---|---|---|---|
| FtpSync | C#, Oxygene, VB | Advanced | This sample compares files on the local computer against those on an ftp folder and downloads files that are new or have changed. |
| HTTP Responses | C#, Oxygene, VB | Introduction | This sample shows how to create and configure a simple HTTP server. |
| HTTP Spy | C#, Oxygene, VB | Introduction | This example demonstrates how HTTP works. With help of the HttpClient components, we can send a HttpClientRequest and retrieve back a HttpClientResponse. |
| Sample Server | C#, Oxygene, Delphi, VB | Intermediate | This example shows how easy it is to create a simple custom Http server. |
| SMTP Client | C#, Oxygene, VB | Intermediate | The sample shows how we can use the SMTP client component for sending emails. |
| Virtual FTP | C#, Oxygene, VB | Advanced | This sample shows how to create and configure virtual FTP. |
If you encounter any problems or have questions regarding the samples, please feel free to ask on our newsgroup at news://news.remobjects.com/remobjects.public.internetpack.net.
Thank you very much,
Your RemObjects Team
http://www.remobjects.com


val tag: String?
Log tag to identify where does this log call. It is recommended to use auto tag.
tag - Log tag to identify where does this log call. It is recommended to use auto tag.
tag


Dave Gooderham: Nagging doubts... But should we worry? Look at the table!
by Mike BaconIt's been five days of cup football for Town. DAVE GOODERHAM asks what do Town fans make of it all? Should we have any worries?
So what do we make of the last five days of cup football?
An unwanted FA Cup replay, Paul Lambert's controversial team selection (more of that in a minute) and a defeat to neighbours Colchester United including a debate over a goal of the season or a fluke clearance - the latter for me.
Some of these are barely sub plots in a season that remains firmly on course, in the league at least. But while we shouldn't worry when the League 1 table looks so rosy, football fans, by nature, always have nagging doubts. So here goes...
As a fan of a team sitting at the top of the table - I will never tire of saying that - my biggest concern is actually the stop-start nature of this season. I don't blame Paul Lambert for wanting to postpone games at any opportunity, especially when you would be without the likes of Flynn Downes. It also gives those in the treatment table more time to be, well, treated. Kane Vincent-Young is a case in point.
MORE: 9 things we learned from Town's defeat to Col U
But the lack of Saturday afternoon entertainment isn't just a problem for football fans starved of their fix (though I would like to point to the many non-league clubs in our area as a more than adequate alternative). The postponements and cup squads are surely hampering any real momentum within the first team squad.
That might not seem the case when you look at the league table, but it's not exactly like we are playing teams off the park. Does that matter? Probably not, for most, especially when we are winning. But performances would certainly be scrutinised a whole lot harder if those narrow 1-0 wins suddenly become draws or worse.
Then we have the ill-fated 1-1 draw at home to Lincoln City. Among my other EADT columnists, I usually appear last on the weekly list when it comes to publication on the website or in print. This means many of the choice opinions have already been voiced. Last Saturday is one such case.
Did Paul Lambert really have to make so many changes? No, not for me. Could he have blended a better side by introducing new faces alongside a few who have played the majority of games? Absolutely.
Does it matter that Ipswich Town didn't make it on their own into the second round but instead had an 'Or' after their name in the draw? For many, probably not. But in a season that is promising so much, how nice would it be to actually taste a cup victory - and while we are at it, success in a televised game as well.
There are a number of questions, too many for me. Lambert is paid to get Ipswich promoted and if that comes at the expense of the most famous club competition in the world, I get that. Doesn't mean I have to like it though.
Then came the EFL trophy. Now this was a good chance to introduce some youngsters and it is great to see the likes of Brett McGavin and Ben Folami get some deserved praise after decent performances on Tuesday night. And while I'm not going to sit here and say I had heard of 16 year olds Tawanda Chirewa and Liam Gibbs, they will forever remember the match - even though it won't live long in the memory of the 2,500 diehard Town fans who attended.
MORE: You think you're feeling old! MIKE BACON ON THE BLUES
Having just read a piece about how former Town favourite Jimmy Bullard is one of a handful to have earned the unwanted tag of an England squad member but with no full cap to his name, let's give a lot of credit to Lambert for throwing the teenagers in for the first time. It was the perfect game to do so.
Where does the cup events of the last few days leave Ipswich? Probably, exactly in the same position as before FA Cup Round One. They are still the league team to beat, the scalp to be had, still flying high on confidence and still searching for a bit more in terms of performance from first minute to last.
Only once Blackpool and Wycombe come calling at the end of the month will we begin to know whether a stop-start League 1 campaign will have any lasting problems. Roll on Saturday 23rd November, I say.


{% extends "base.html" %} {% block content %}
Please fill in the form below. When Ownblock officially launches, we'll get in touch with you to find out more about the properties you wish to include in Ownblock so we can get your site set up. Don't worry - we'll help you get started!
{% if form.errors %}


---
title: "Reading List, GEOG 215"
author: "Varun Goel"
date: "Spring Sememster, 2020"
output:
html_document:
toc: true
toc_float: true
---
Updated: December 24 2019
The “Weeks” below correspond to enumerated weeks found in the syllabus schedule. Please email Dr. D immediately if any links are broken!
Week #4
- Relative Paths and the Working Directory in R (Video)
- R for Data Science, Chapter 8. Workflow: projects
- Sarah B. Bouchat, Introduction to Statistical Computing, Data Management in R
- Pebesma, E., Nüst, D. and Bivand, R., (2012), The R software environment in reproducible geoscientific research, Eos Trans. AGU, 93(16), 163. pdf also available on Sakai
Week #11
- Spatial Data Science with R, Spatial data analysis, Chapter 3. Spatial autocorrelation
- Manuel Gimond, Intro to GIS and Spatial Analysis, Chapter 13. Spatial Autocorrelation
- Manuel Gimond, Intro to GIS and Spatial Analysis, Chapter 11. Point Pattern Analysis (minus these sections: 11.1, <IP_ADDRESS>, 11.2.3, 11.3.2, 11.3.3) )


title
=$tclive_times;?>
series_title; ?>
series_subtitle; ?>
Part series_part; ?> · event_date; ?> · live_speaker; ?>Everywhere we look, we find false narratives that people believe and build their lives around. This even happens with the Gospel, and the enemy of our souls attempts to twist the Good News into something that it’s not. We want to detect and identify what the Fake News is, so we can truly embrace the Gospel.
Review Questions
- Why is the Scripture reliable? If you believe that the Scripture is reliable, how will that show up in your everyday life?
- What were some of the examples of false teachers given in this message? Why should we be aware of such things and guard our hearts from them?
- What is one action step you will take this week based on what you heard in this message?
Daily Readings
Memory Verse
All Scripture is God-breathed and is useful for teaching, rebuking, correcting and training in righteousness, so that the servant of God may be thoroughly equipped for every good work. (2 Timothy 3:16-17)
Give to Thursday Church and take part in helping us reach every
man, woman and child with the Gospel of Jesus Christ.
Text any amount to
716.333.5511
Come join other people like you who want to grow in faith and learn more about what it means to follow Jesus.
Join A Small GroupReady to join us in person? The Chapel is one church with multiple locations. We'd love to have you join us for a Sunday service!
Find A LocationPrayer Request
Please know that your requests and praises are emailed out daily to a group of diligent Chapel prayer partners.
If you would like to be contacted personally, please include your phone number or email address in the space provided. To assure confidentiality, your contact information will only be made available to Thursday Church ministry team and will not be included in the outgoing email as part of your prayer request.
Technical Support Request
If you're having trouble viewing the live stream, visit our Livestream page. For additional support, contact us below.


If you follow these rules you will be on the path to success and the path to being a good leader.
Find Out MoreIntroduction to self improvement is necessary for people to get out of a life of despair and changelessness his or her lifestyle so that a person is happy and content with himself and his abilities. It tries to help someone develop his own personality and his mind in a manner which is fruitful to his health and abilities. It does not seek to end pain or fear in our lives. All these factors are necessary elements in shaping our lives. Rather this helps us use these as stepping stones to developing our life.
 One of these reason is because it is morally right. Another reason is because if you look out for your subordinates they will help and look out for you.
773-796-0508


Class: Rails::Generators::Actions::CreateMigration
| Relationships & Source Files | |
| Super Chains via Extension / Inclusion / Inheritance | |
|
Class Chain:
self,
Thor::Actions::CreateFile
|
|
|
Instance Chain:
self,
Thor::Actions::CreateFile
|
|
| Inherits: |
Thor::Actions::CreateFile
|
| Defined in: | railties/lib/rails/generators/actions/create_migration.rb |
Instance Attribute Summary
-
#exists?
readonly
Alias for #existing_migration.
- #identical? ⇒ Boolean readonly
Instance Method Summary
Instance Attribute Details
#exists? (readonly)
Alias for #existing_migration.
# File 'railties/lib/rails/generators/actions/create_migration.rb', line 38
alias :exists? :existing_migration
#identical? ⇒ Boolean
(readonly)
[ GitHub ]
# File 'railties/lib/rails/generators/actions/create_migration.rb', line 16
def identical? exists? && File.binread(existing_migration) == render end
Instance Method Details
#existing_migration Also known as: #exists?
[ GitHub ]# File 'railties/lib/rails/generators/actions/create_migration.rb', line 32
def existing_migration @existing_migration ||= begin @base.class.migration_exists?(migration_dir, migration_file_name) || File.exist?(@destination) && @destination end end
#migration_dir
[ GitHub ]# File 'railties/lib/rails/generators/actions/create_migration.rb', line 8
def migration_dir File.dirname(@destination) end
#migration_file_name
[ GitHub ]# File 'railties/lib/rails/generators/actions/create_migration.rb', line 12
def migration_file_name @base.migration_file_name end
#relative_existing_migration
[ GitHub ]# File 'railties/lib/rails/generators/actions/create_migration.rb', line 28
def relative_existing_migration base.relative_to_original_destination_root(existing_migration) end
#revoke!
[ GitHub ]# File 'railties/lib/rails/generators/actions/create_migration.rb', line 20
def revoke! say_destination = exists? ? relative_existing_migration : relative_destination say_status :remove, :red, say_destination return unless exists? ::FileUtils.rm_rf(existing_migration) unless pretend? existing_migration end


StringLessOrEqualsTest.testComparableEqualsFailNoMessage: String 'referenceA'(a.abd) is less than or equal to String 'referenceB'(a.abd). StringLessOrEqualsTest.testComparableLessFailMessage: String 'referenceA'(a.abc) is less than or equal to String 'referenceB'(a.abd). Extra info goes here. StringLessOrEqualsTest.testComparableEqualsFailMessage: String 'referenceA'(_a.abd) is less than or equal to String 'referenceB'(_a.abd). Extra info goes here. StringLessOrEqualsTest.testComparableLessFailNoMessage: String 'referenceA'(_a.abc) is less than or equal to String 'referenceB'(a.abc).


kml.poly3
Create a 3D polygon with vertices given by longitude, latitude and altitude.
Contents
Syntax
kml.poly3(long, lat, alt) kml.poly3(...,'PropertyName',PropertyValue,...)
Description
Creates a 3D polygonal form, with vertices given by longitude, latitude and altitude.
The units for latitude and longitude are normally given in degrees, but this can be changed by calling: kml.useDegrees; or kml.useRadians; before plotting. Altitude is always given in meters.
It is possible to fine tune the polygon properties using name-value pairs:
kml.poly3(...,'PropertyName',PropertyValue,...)
Properties
| Property Name | Type | Description |
|---|---|---|
| 'name' | string | Name of the polygon polygon the kml file |
| 'id' | string | Internal id of this plot inside the kml |
| 'description' | string | A short description of what the polygon represents |
| 'visibility' | true/false | Control the initial visibility of the polygon |
| 'lineColor' | kml color string | Defines the color of the polygon lines. Must be a valid hex color string input, in the style AABBGGRR |
| 'polyColor' | kml color string | Defines the color of the polygon faces. Must be a valid hex color string input, in the style AABBGGRR |
| 'lineWidth' | double | Defines the width of the polygon lines. Set to zero to show no lines. |
| 'altitudeMode' | string | Choose if the altitude value is absolute to the earth model, relative to the ground level, or should be clamped to the ground. Valid inputs: 'absolute', 'relativeToGround', 'clampToGround' |
| 'extrude' | true/false | Enables or disables extruding the polygon to the ground. |
| 'tessellate' | true/false | Tesselates the polygon into the ground, so that it will be visible within the terrain contour |
| 'timeStamp' | kml date string | Associates the polygon to a moment in time. Should not be used together with timeSpan. Should be a string in the XML time format (more information available here) |
| 'timeSpanBegin' | kml date string | Defines the moment in time where the polygon starts to exist. Should not be used together with timeStamp. Should be a string in the XML time format (more information available here) |
| 'timeSpanEnd' | kml date string | Defines the moment in time where the polygon finishes to exist. Should not be used together with timeStamp. Should be a string in the XML time format (more information available here) |
Example
% Create a new kml object k = kml('my kml file'); % Creates two rings around the earth t = linspace(0,360,50); for i = 1:numel(t)-1; k.poly3([t(i) t(i+1) t(i+1) t(i) t(i)],[0 0 1 1 0]+5, [1 1 3 3 1]*1e6,'polyColor','BB00FF00','extrude',false,'lineWidth',0) k.poly3([t(i) t(i+1) t(i+1) t(i) t(i)],[0 0 1 1 0]-5, [1 1 3 3 1]*1e6,'polyColor','BB00FF00','extrude',false,'lineWidth',0) end % Save the kml and open it in Google Earth k.run;
This is the result of running this example:
This file is part of the kml toolbox. Copyright 2012 Rafael Fernandes de Oliveira (rafael@rafael.aero)


In my Design Methods class, my teammates and I worked together to redesign the Washington State Unemployment Eligibility Form into a more user-friendly and interactive survey. This new design enhances usability feedback and includes multiple language options as well as a live chat feature. As a Lead User Researcher on this project, I oversaw the research process. I developed the Interview Protocol, as well as conducted User Interviews and Usability Testings. I also designed low and high-fidelity wireframes using Figma, Whimsical, and Miro.
The Washington State Unemployment Eligibility Form is not user-friendly for the following reasons: it provides no feedback to users, the language is vague and unclear, and lastly the form is only available in English and Spanish. The unemployment eligibility form is currently available in PDF format and can be downloaded from their website. Since it is a PDF document, it is not interactive and lacks feedback, which means that users may not realize that they had made a mistake when filling out the form. Some people, particularly those who do not know the technology and whose English is not their first language, may also find the questionnaire confusing. For example, Question 5 asks if the individual has been laid-off, but does not fully clarify what is considered to be a layoff. Instead, it offers a link that guides users to a different page. Not only can this be frustrating for applicants, but it can also make the process longer.
Our approach was to make this PDF format more interactive and user-friendly by converting it into a survey. Surveys are simple, succinct, and straightforward. In addition, the survey will automatically inform them whether or not they are eligible, providing instant feedback and guiding them in the appropriate next steps. We also wanted to include increased definitions and clarify the language of the survey when possible. These procedures would also reduce the amount of time that applicants spend on filling out the form. We also wanted to ensure that the survey keeps accessibility in mind, and is available in a wide variety of languages. Our approach enabled users to have a clearer understanding of the application process and make the process simple and fast.
We completed a literature review consisting of two parts. First, we examined the literature on the experiences of receiving unemployment benefits in Washington. This particularly focused on the current experience of applicants during the COVID-19 pandemic. The second half examined form design, focusing on creating forms in a government context. We also completed competitive analysis, examining the forms used by other states to determine eligibility for unemployment benefits.
Five people participated in a user interview. The following is a rundown of the interview.
To better categorize and organize our data, my team and I developed an affinity diagram.
Our team worked together to produce a rough draft of our storyboard, and our Project Manager completed the polished storyboard below.
As a blueprint for our wireframes, we developed the following Information Architecture.
For both mobile and desktop, we first produced a low-fidelity wireframe. I worked on the mobile application in particular, as seen below. For a better user experience, we agreed as a team to use a survey rather than a PDF format.
We created High-fidelity wireframes after our low-fidelity wireframes. I worked on both the desktop and mobile versions, but I concentrated on the mobile version. The high-fidelity version is more interactive and user-friendly. New features included a progress bar, live support, and multiple language options.
After designing our high-fidelity prototype, I performed half of our usability tests. In our usability tests, we discovered some key findings. Our participants preferred the survey format to the PDF format 100% of the time. The survey took less time to complete and automatically assessed their eligibility status, which they considered more time-efficient and easier to use. Although our survey received positive user reviews, a few usability issues posed, prompting my team and myself to make final iterations.
Based on our findings we made our final iterations


The Gang Teaches Typing
| File |
Lessons |
Keyboard |
Progress |
Start
The Gang
Teaches Typing
Click to Begin