DotNetVault Quick Start Guide – Visual Studio 2019
1- Open visual studio and Create a new .NET Core 3.0+ or .NET Framework 4.8 Console Application.
2- If you chose .NET Framework 4.8, open the .csproj file and add the following Line <LangVersion>8.0</LangVersion>
under each “Platform/Config” PropertyGroup as shown in highlight below. This is unnecessary if you chose a .NET Core 3.0+ based Console Application.
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugSymbols>true</DebugSymbols>
<DebugType>full</DebugType>
<Optimize>false</Optimize>
<OutputPath>bin\Debug\</OutputPath>
<DefineConstants>DEBUG;TRACE</DefineConstants>
<ErrorReport>prompt</ErrorReport>
<LangVersion>8.0</LangVersion>
<WarningLevel>4</WarningLevel>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugType>pdbonly</DebugType>
<Optimize>true</Optimize>
<OutputPath>bin\Release\</OutputPath>
<DefineConstants>TRACE</DefineConstants>
<ErrorReport>prompt</ErrorReport>
<LangVersion>8.0</LangVersion>
<WarningLevel>4</WarningLevel>
</PropertyGroup>
3- Right click on your project and chose “Manage NuGet packages”. Click on the Browse option, select “Include Prerelease”, enter “cpsusie dotnetvault” into
the search block, select DotNetVault, then click “Install”
4- You are now set up to use DotNetVault in your project.
5- To see what can be done:
a. download the source code for DotNetVault from GitHub.
b. Extract the zip file contents to a folder in a convenient place
c. Go into the folder structure and open the file “DotNetVaultQuickStart.sln” using Visual Studio 2019
d. Build the solution.
e. Run the solution and check output, which should be something like this:
f. As you can see, from Main(), the demo demonstrates the use of the BasicVault, which stores VaultSafe objects and the MutableResourceVault which stores objects that are not VaultSafe.
i. VaultSafe objects are ones that do not require effort to keep isolated. They include
1. Unmanaged value types (e.g. long, enums, DateTime, TimeSpan, etc.)
2. Sealed immutable reference types that are annotated with the VaultSafe attribute (string is automatically considered vault-safe; Immutable collections from System.Collections.Immutable that have only vault-safe type arguments are also considered vault-safe automatically).
3. Value types that are annotated with the VaultSafe attribute and contain only other types that comply with #1, 2 and 3
4. Unmanaged value types, strings and qualifying immutable collections are considered VaultSafe without need for annotation with the VaultSafe attribute.
These objects are easy to isolate because copies of the are either true deep copies (unmanaged types), totally immutable reference types (no danger of a stored reference changing the protected value), or value types that contain only types that are unmanaged value types and immutable reference types. The resource protected in the DemonstrateBasicVault() in Program.cs is a DogActionRecord, found in DogActionRecord.cs.
DogActionRecord is VaultSafe because:
1. It is annotated with the VaultSafe attribute
2. It is a value type with field members that include
a. An unmanaged value type (DateTime) and
b. A sealed immutable reference type (string)
g. The DemonstrateBasicVault() in Program.cs creates a BasicVault<DogActionRecord>, a vault that protects a DogActionRecord and can be used to obtain locks for synchronized access to the value. It then creates a List of Dogs (Dog.cs) then calls their DoDogActions() methods, which causes each to spawn a thread that obtains a lock on the vault and overwrites the value stored therein with an action.
The syntax is straight forward:
Figure 1
At the end of the demonstration, the main thread obtains the lock and prints out the DogActionRecord that happens to be there (because the order is non-deterministic, there will be different results):
Figure 2
h. The DemonstrateMutableResourceVault() method in Program.cs shows protection of a resource that IS NOT vault-safe. It is a SortedSet of DogActionRecords. The dogs in this demo, instead of overwriting a single DogActionRecord, will add their DogActionRecords to the SortedSet (which maintains them ordered by timestamp, convenient for printing in order at end).
Resources that are not VaultSafe are protected by a MutableResourceVault, the lock objects of which are more restrictive because they need to make sure that only VaultSafe types are passed into the protected resource or received out from the protected resource; otherwise, mutable state (to which a reference may exist outside) could mingle with the protected resource or mutable state inside the protected resource could leak to the outside and cause unsynchronized access to the protected resource.
The syntax for creating a MutableResourceVault is shown:
Figure 3
Accessing the mutable resource through the lock is mediated by delegates:
Figure 4
At the end of the simulation, a lock is acquired that returns the SortedSet to the main thread as an ImmutableSortedSet<DogActionRecord> as shown:
Figure 5
i. Further resources.
i. The DotNetVault itself comes with an example of a vault customized to protected StringBuilder resources without resort to the inconvenient delegate syntax. It contains directions for making your own customized vaults and locked resource objects.
ii. Included in the source repository is the ConsoleStressTest using .NET Core 3.1. It can be run in a Linux or Windows environment and demonstrates the effectiveness of DotNetVault at protecting a mutable resource with a high degree of thread contention.
iii. The LaundryMachine.sln and the LaundryStressTest project therein requires Windows because it uses WPF. It demonstrates the usage of many vaults and many threads in contention in a highly (unnecessarily so, but good for demonstration purposes) complex multithreaded state machine scenario where LaundryMachines have their own threads and loader and unloader robots (also with their own threads) contend for access to laundry machine. The simulation runs until all soiled articles are cleaned. The loader robots put dirty laundry into one of the machines then start the cycle, the unloader robots take dirty laundry from the machine and put them in the clean bin. The robots constantly contend with each other for access to the machine and since each Laundry Machines state machine thread is independent, the robots also contend with the state machine threads as well as each other to access the machines.
iv. The ExampleCodePlayground project is used to get a feel for the static analyzer’s rules and how they work.
v. There is a unit test project primarily oriented around ensuring that the static analyzer rules work properly.
j. The DotNetVault Description.pdf provides detailed information on the DotNetVault, the vaults and LockedResourceObjects, its reasoning and static analysis rules as well as examples.


R/compare_df_cols.R
Generate a comparison of data.frames (or similar objects) that indicates if they will successfully bind together by rows.
compare_df_cols(..., return = c("all", "match", "mismatch"), bind_method = c("bind_rows", "rbind"), strict_description = FALSE)
| ... | A combination of data.frames, tibbles, and lists of data.frames/tibbles. The values may optionally be named arguments; if named, the output column will be the name; if not named, the output column will be the data.frame name (see examples section). |
|---|---|
| return | Should a summary of "all" columns be returned, only return "match"ing columns, or only "mismatch"ing columns? |
| bind_method | What method of binding should be used to determine
matches? With "bind_rows", columns missing from a data.frame would be
considered a match (as in |
| strict_description | Passed to |
A data.frame with a column named "column_name" with a value named
after the input data.frames' column names, and then one column per
data.frame (named after the input data.frame). If more than one input has
the same column name, the column naming will have suffixes defined by
sequential use of base::merge()
and may differ from expected naming.
The rows within the data.frame-named columns are descriptions of the
classes of the data within the columns (generated by
describe_class
).
Due to the returned "column_name" column, no input data.frame may be named "column_name".
The strict_description
argument is most typically used to understand
if factor levels match or are bindable. Factors are typically bindable,
but the behavior of what happens when they bind differs based on the
binding method ("bind_rows" or "rbind"). Even when
strict_description
is FALSE
, data.frames may still bind
because some classes (like factors and characters) can bind even if they
appear to differ.
Other Data frame type comparison: compare_df_cols_same
,
describe_class
compare_df_cols(data.frame(A=1), data.frame(B=2))#> column_name data.frame(A = 1) data.frame(B = 2) #> 1 A numeric <NA> #> 2 B <NA> numeric# user-defined names compare_df_cols(dfA=data.frame(A=1), dfB=data.frame(B=2))#> column_name dfA dfB #> 1 A numeric <NA> #> 2 B <NA> numeric# a combination of list and data.frame input compare_df_cols(listA=list(dfA=data.frame(A=1), dfB=data.frame(B=2)), data.frame(A=3))#> column_name dfA dfB data.frame(A = 3) #> 1 A numeric <NA> numeric #> 2 B <NA> numeric <NA>


Hamilton: Money doesn't drive me – I just love racing
by OmnisportLewis Hamilton insists money is not and never has been the motivating factor for him as he continues to build a remarkable legacy in Formula One.
The six-time world champion has all the trappings associated with F1 stardom but says his love of racing is what drives him.
Hamilton is a figure who can divide opinion but he sees himself as a fierce competitor first and foremost, with the financial gain his success brings simply "a bonus".
"The thing is I never got into it for money," he told BBC Sport.
"Of course it is great that that piles up – no problem. That is a bonus. As long as those things don't become the lead factor of what I do.
"The core of what I do is that I love racing. I love the challenge. I love arriving knowing I have got these incredibly talented youngsters who are trying to beat me and outperform me, outsmart me, and I love that battle that I get into every single year.
"And I am working with these guys [his Mercedes engineers] who are so much smarter than me and they make me feel smarter. When I am challenging them and proving them wrong so many times, it is unreal."
The Briton gave an insight into the dynamic that exists in the Mercedes garage, where he and chief engineer Andrew Shovlin often engage in debates about technical issues relating to the car.
"It happens a lot," said Hamilton. "I say something to Shov, and he will say: 'No, the numbers say this', and I will say: 'It's this and this and this.' And he will say: 'Oh you are right.' It feels so good. There are a lot of things like that."
Hamilton is one title win away from equalling Michael Schumacher's record tally of seven.


{{ number }} is not divisible by 3!
{{ "Some Letters" }}
{{ letter | uppercase}} {{ letter }}
{{ "2016's Most Fashionable Snakes" }}
{{ "Très chic!" }}
{{ "Bravo!" }}
Your Favorite Snake
My reasons is because:
Submit!
My favorite snake is: {{ favesnake }} because {{ reason }}


#include <stMNode.h>
Classes | |
| struct | stMNodeHeader |
Public Types | |
| enum | stMNodeType { INDEX = 0x4449, LEAF = 0x464C } |
Public Member Functions | |
| virtual | ~stMNode () |
| stNodeType | GetNodeType () |
| stPage * | GetPage () |
| stPageID | GetPageID () |
| int | GetNumberOfEntries () |
| virtual int | AddEntry (stSize size, const stByte *object)=0 |
| virtual const stByte * | GetObject (int id)=0 |
| virtual stSize | GetObjectSize (int id)=0 |
| virtual stDistance | GetMinimumRadius ()=0 |
| void | RemoveAll () |
Static Public Member Functions | |
| static stMNode * | CreateNode (stPage *page) |
| static stSize | GetGlobalOverhead () |
Protected Types | |
| typedef struct stMNode::stMNodeHeader | stMNodeHeader |
Protected Member Functions | |
| stMNode (stPage *page) | |
Protected Attributes | |
| stMNodeHeader * | Header |
| stPage * | Page |
This abstract class is the basic MTree node. All classes that implement MTree nodes must extend this class.
The main function of this class is to provide a way to identify a disk node and create the required node instance to manipulate the node.
The structure of Index Node follows:
The Header holds the information about the node itself.
The Node Data is the segment of the node which holds the particular information of each type of the node. This class does not know how this information is organized.
|
protected |
This is the structure of the Header of a MTree node.
| enum stMNode::stMNodeType |
|
inlinevirtual |
|
inlineprotected |
Creates a new instance of this class.
| page | An instance of stPage. |
Definition at line 276 of file stMNode.h.
References stPage::GetData().
This is a virtual method that defines a interface in the insertion of a new Object in a Node.
| size | The size of the object in bytes. |
| object | The object data. |
Implemented in stMLeafNode, and stMIndexNode.
This is a virtual method that defines a interface for the instantiate the correct specialization of this class.
| page | The instance of stPage. |
|
inlinestatic |
|
pure virtual |
Returns the minimum radius of this node.
Implemented in stMLeafNode, and stMIndexNode.
|
inline |
Returns the type of this MTree node (Leaf or Index).
Definition at line 137 of file stMNode.h.
References Header, and stMNode::stMNodeHeader::Type.
|
inline |
Returns the number of entries in this node.
Definition at line 175 of file stMNode.h.
References Header, and stMNode::stMNodeHeader::Occupation.
Referenced by stMIndexNode::GetIndexEntry(), and stMLeafNode::GetLeafEntry().
|
pure virtual |
Gets the serialized object. Use GetObjectSize to determine the size of the object.
| id | The id of the entry. |
Implemented in stMLeafNode, and stMIndexNode.
|
pure virtual |
Returns the size of the object in bytes. Use GetObject() to get the object data.
| id | The id of the entry. |
Implemented in stMLeafNode, and stMIndexNode.
|
inline |
|
inline |
Returns the ID of the associated page.
Definition at line 155 of file stMNode.h.
References stPage::GetPageID(), and Page.
|
inline |
Remove All entries.
Definition at line 224 of file stMNode.h.
References stPage::Clear(), Header, stMNode::stMNodeHeader::Occupation, Page, and stMNode::stMNodeHeader::Type.
|
protected |
Header of this page.
Definition at line 264 of file stMNode.h.
Referenced by GetNodeType(), GetNumberOfEntries(), and RemoveAll().
|
protected |
The page related with this class.
Definition at line 269 of file stMNode.h.
Referenced by GetPage(), GetPageID(), and RemoveAll().


DistCp Version 2 Guide
Overview
DistCp Version 2 (distributed copy) is a tool used for large inter/intra-cluster copying. It uses MapReduce to effect its distribution, error handling and recovery, and reporting. It expands a list of files and directories into input to map tasks, each of which will copy a partition of the files specified in the source list.
The erstwhile implementation of DistCp has its share of quirks and drawbacks, both in its usage, as well as its extensibility and performance. The purpose of the DistCp refactor was to fix these shortcomings, enabling it to be used and extended programmatically. New paradigms have been introduced to improve runtime and setup performance, while simultaneously retaining the legacy behaviour as default.
This document aims to describe the design of the new DistCp, its spanking new features, their optimal use, and any deviance from the legacy implementation.
Usage
Basic Usage
The most common invocation of DistCp is an inter-cluster copy:
bash$ hadoop distcp2 hdfs://nn1:8020/foo/bar \
hdfs://nn2:8020/bar/foo
This will expand the namespace under /foo/bar on nn1 into a temporary file, partition its contents among a set of map tasks, and start a copy on each TaskTracker from nn1 to nn2.
One can also specify multiple source directories on the command line:
bash$ hadoop distcp2 hdfs://nn1:8020/foo/a \
hdfs://nn1:8020/foo/b \
hdfs://nn2:8020/bar/foo
Or, equivalently, from a file using the -f option:
bash$ hadoop distcp2 -f hdfs://nn1:8020/srclist \
hdfs://nn2:8020/bar/foo
Where srclist contains
hdfs://nn1:8020/foo/a
hdfs://nn1:8020/foo/b
When copying from multiple sources, DistCp will abort the copy with an error message if two sources collide, but collisions at the destination are resolved per the options specified. By default, files already existing at the destination are skipped (i.e. not replaced by the source file). A count of skipped files is reported at the end of each job, but it may be inaccurate if a copier failed for some subset of its files, but succeeded on a later attempt.
It is important that each TaskTracker can reach and communicate with both the source and destination file systems. For HDFS, both the source and destination must be running the same version of the protocol or use a backwards-compatible protocol; see Copying Between Versions.
After a copy, it is recommended that one generates and cross-checks a listing of the source and destination to verify that the copy was truly successful. Since DistCp employs both Map/Reduce and the FileSystem API, issues in or between any of the three could adversely and silently affect the copy. Some have had success running with -update enabled to perform a second pass, but users should be acquainted with its semantics before attempting this.
It's also worth noting that if another client is still writing to a source file, the copy will likely fail. Attempting to overwrite a file being written at the destination should also fail on HDFS. If a source file is (re)moved before it is copied, the copy will fail with a FileNotFoundException.
Please refer to the detailed Command Line Reference for information on all the options available in DistCp.
Update and Overwrite
-update is used to copy files from source that don't exist at the target, or have different contents. -overwrite overwrites target-files even if they exist at the source, or have the same contents.
Update and Overwrite options warrant special attention, since their
handling of source-paths varies from the defaults in a very subtle manner.
Consider a copy from /source/first/ and
/source/second/ to /target/, where the source
paths have the following contents:
hdfs://nn1:8020/source/first/1
hdfs://nn1:8020/source/first/2
hdfs://nn1:8020/source/second/10
hdfs://nn1:8020/source/second/20
When DistCp is invoked without -update or
-overwrite, the DistCp defaults would create directories
first/ and second/, under /target.
Thus:
distcp2 hdfs://nn1:8020/source/first hdfs://nn1:8020/source/second hdfs://nn2:8020/target
would yield the following contents in /target:
hdfs://nn2:8020/target/first/1
hdfs://nn2:8020/target/first/2
hdfs://nn2:8020/target/second/10
hdfs://nn2:8020/target/second/20
When either -update or -overwrite is
specified, the contents of the source-directories
are copied to target, and not the source directories themselves. Thus:
distcp2 -update hdfs://nn1:8020/source/first hdfs://nn1:8020/source/second hdfs://nn2:8020/target
would yield the following contents in /target:
hdfs://nn2:8020/target/1
hdfs://nn2:8020/target/2
hdfs://nn2:8020/target/10
hdfs://nn2:8020/target/20
By extension, if both source folders contained a file with the same
name (say, 0), then both sources would map an entry to
/target/0 at the destination. Rather than to permit this
conflict, DistCp will abort.
Now, consider the following copy operation:
distcp2 hdfs://nn1:8020/source/first hdfs://nn1:8020/source/second hdfs://nn2:8020/target
With sources/sizes:
hdfs://nn1:8020/source/first/1 32
hdfs://nn1:8020/source/first/2 32
hdfs://nn1:8020/source/second/10 64
hdfs://nn1:8020/source/second/20 32
And destination/sizes:
hdfs://nn2:8020/target/1 32
hdfs://nn2:8020/target/10 32
hdfs://nn2:8020/target/20 64
Will effect:
hdfs://nn2:8020/target/1 32
hdfs://nn2:8020/target/2 32
hdfs://nn2:8020/target/10 64
hdfs://nn2:8020/target/20 32
1 is skipped because the file-length and contents match.
2 is copied because it doesn't exist at the target.
10 and 20 are overwritten since the contents
don't match the source.
If -update is used, 1 is overwritten as well.
Command Line Options
| Flag | Description | Notes |
|---|---|---|
| -p[rbugp] | Preserve r: replication number b: block size u: user g: group p: permission |
Modification times are not preserved. Also, when -update is specified, status updates will not be synchronized unless the file sizes also differ (i.e. unless the file is re-created). |
| -i | Ignore failures | As explained in the Appendix, this option will keep more accurate statistics about the copy than the default case. It also preserves logs from failed copies, which can be valuable for debugging. Finally, a failing map will not cause the job to fail before all splits are attempted. |
| -log <logdir> | Write logs to <logdir> | DistCp keeps logs of each file it attempts to copy as map output. If a map fails, the log output will not be retained if it is re-executed. |
| -m <num_maps> | Maximum number of simultaneous copies | Specify the number of maps to copy data. Note that more maps may not necessarily improve throughput. |
| -overwrite | Overwrite destination | If a map fails and -i is not specified, all the files in the split, not only those that failed, will be recopied. As discussed in the Usage documentation, it also changes the semantics for generating destination paths, so users should use this carefully. |
| -update | Overwrite if src size different from dst size | As noted in the preceding, this is not a "sync" operation. The only criterion examined is the source and destination file sizes; if they differ, the source file replaces the destination file. As discussed in the Usage documentation, it also changes the semantics for generating destination paths, so users should use this carefully. |
| -f <urilist_uri> | Use list at <urilist_uri> as src list | This is equivalent to listing each source on the command line. The urilist_uri list should be a fully qualified URI. |
| -filelimit <n> | Limit the total number of files to be <= n | Deprecated! Ignored in the new DistCp. |
| -sizelimit <n> | Limit the total size to be <= n bytes | Deprecated! Ignored in the new DistCp. |
| -delete | Delete the files existing in the dst but not in src | The deletion is done by FS Shell. So the trash will be used, if it is enable. |
| -strategy {dynamic|uniformsize} | Choose the copy-strategy to be used in DistCp. | By default, uniformsize is used. (i.e. Maps are balanced on the total size of files copied by each map. Similar to legacy.) If "dynamic" is specified, DynamicInputFormat is used instead. (This is described in the Architecture section, under InputFormats.) |
| -bandwidth | Specify bandwidth per map, in MB/second. | Each map will be restricted to consume only the specified bandwidth. This is not always exact. The map throttles back its bandwidth consumption during a copy, such that the net bandwidth used tends towards the specified value. |
| -atomic {-tmp <tmp_dir>} | Specify atomic commit, with optional tmp directory. | -atomic instructs DistCp to copy the source data to a temporary target location, and then move the temporary target to the final-location atomically. Data will either be available at final target in a complete and consistent form, or not at all. Optionally, -tmp may be used to specify the location of the tmp-target. If not specified, a default is chosen. Note: tmp_dir must be on the final target cluster. |
| -mapredSslConf <ssl_conf_file> | Specify SSL Config file, to be used with HSFTP source | When using the hsftp protocol with a source, the security- related properties may be specified in a config-file and passed to DistCp. <ssl_conf_file> needs to be in the classpath. |
| -async | Run DistCp asynchronously. Quits as soon as the Hadoop Job is launched. | The Hadoop Job-id is logged, for tracking. |
Architecture of DistCp
The components of the new DistCp may be classified into the following categories:
- DistCp Driver
- Copy-listing generator
- Input-formats and Map-Reduce components
DistCp Driver
The DistCp Driver components are responsible for:
- Parsing the arguments passed to the DistCp command on the
command-line, via:
- OptionsParser, and
- DistCpOptionsSwitch
- Assembling the command arguments into an appropriate
DistCpOptions object, and initializing DistCp. These arguments
include:
- Source-paths
- Target location
- Copy options (e.g. whether to update-copy, overwrite, which file-attributes to preserve, etc.)
- Orchestrating the copy operation by:
- Invoking the copy-listing-generator to create the list of files to be copied.
- Setting up and launching the Hadoop Map-Reduce Job to carry out the copy.
- Based on the options, either returning a handle to the Hadoop MR Job immediately, or waiting till completion.
The parser-elements are exercised only from the command-line (or if DistCp::run() is invoked). The DistCp class may also be used programmatically, by constructing the DistCpOptions object, and initializing a DistCp object appropriately.
Copy-listing Generator
The copy-listing-generator classes are responsible for creating the list of files/directories to be copied from source. They examine the contents of the source-paths (files/directories, including wild-cards), and record all paths that need copy into a sequence- file, for consumption by the DistCp Hadoop Job. The main classes in this module include:
- CopyListing: The interface that should be implemented by any copy-listing-generator implementation. Also provides the factory method by which the concrete CopyListing implementation is chosen.
- SimpleCopyListing: An implementation of CopyListing that accepts multiple source paths (files/directories), and recursively lists all the individual files and directories under each, for copy.
- GlobbedCopyListing: Another implementation of CopyListing that expands wild-cards in the source paths.
- FileBasedCopyListing: An implementation of CopyListing that reads the source-path list from a specified file.
Based on whether a source-file-list is specified in the DistCpOptions, the source-listing is generated in one of the following ways:
- If there's no source-file-list, the GlobbedCopyListing is used. All wild-cards are expanded, and all the expansions are forwarded to the SimpleCopyListing, which in turn constructs the listing (via recursive descent of each path).
- If a source-file-list is specified, the FileBasedCopyListing is used. Source-paths are read from the specified file, and then forwarded to the GlobbedCopyListing. The listing is then constructed as described above.
One may customize the method by which the copy-listing is constructed by providing a custom implementation of the CopyListing interface. The behaviour of DistCp differs here from the legacy DistCp, in how paths are considered for copy.
The legacy implementation only lists those paths that must definitely be copied on to target. E.g. if a file already exists at the target (and -overwrite isn't specified), the file isn't even considered in the Map-Reduce Copy Job. Determining this during setup (i.e. before the Map-Reduce Job) involves file-size and checksum-comparisons that are potentially time-consuming.
The new DistCp postpones such checks until the Map-Reduce Job, thus reducing setup time. Performance is enhanced further since these checks are parallelized across multiple maps.
Input-formats and Map-Reduce Components
The Input-formats and Map-Reduce components are responsible for the actual copy of files and directories from the source to the destination path. The listing-file created during copy-listing generation is consumed at this point, when the copy is carried out. The classes of interest here include:
- UniformSizeInputFormat: This implementation of org.apache.hadoop.mapreduce.InputFormat provides equivalence with Legacy DistCp in balancing load across maps. The aim of the UniformSizeInputFormat is to make each map copy roughly the same number of bytes. Apropos, the listing file is split into groups of paths, such that the sum of file-sizes in each InputSplit is nearly equal to every other map. The splitting isn't always perfect, but its trivial implementation keeps the setup-time low.
-
DynamicInputFormat and DynamicRecordReader:
The DynamicInputFormat implements org.apache.hadoop.mapreduce.InputFormat, and is new to DistCp. The listing-file is split into several "chunk-files", the exact number of chunk-files being a multiple of the number of maps requested for in the Hadoop Job. Each map task is "assigned" one of the chunk-files (by renaming the chunk to the task's id), before the Job is launched.
Paths are read from each chunk using the DynamicRecordReader, and processed in the CopyMapper. After all the paths in a chunk are processed, the current chunk is deleted and a new chunk is acquired. The process continues until no more chunks are available.
This "dynamic" approach allows faster map-tasks to consume more paths than slower ones, thus speeding up the DistCp job overall.
-
CopyMapper: This class implements the physical
file-copy. The input-paths are checked against the input-options
(specified in the Job's Configuration), to determine whether a
file needs copy. A file will be copied only if at least one of
the following is true:
- A file with the same name doesn't exist at target.
- A file with the same name exists at target, but has a different file size.
- A file with the same name exists at target, but has a different checksum, and -skipcrccheck isn't mentioned.
- A file with the same name exists at target, but -overwrite is specified.
- A file with the same name exists at target, but differs in block-size (and block-size needs to be preserved.
-
CopyCommitter:
This class is responsible for the commit-phase of the DistCp
job, including:
- Preservation of directory-permissions (if specified in the options)
- Clean-up of temporary-files, work-directories, etc.
Appendix
Map sizing
By default, DistCp makes an attempt to size each map comparably so that each copies roughly the same number of bytes. Note that files are the finest level of granularity, so increasing the number of simultaneous copiers (i.e. maps) may not always increase the number of simultaneous copies nor the overall throughput.
The new DistCp also provides a strategy to "dynamically" size maps, allowing faster data-nodes to copy more bytes than slower nodes. Using -strategy dynamic (explained in the Architecture), rather than to assign a fixed set of source-files to each map-task, files are instead split into several sets. The number of sets exceeds the number of maps, usually by a factor of 2-3. Each map picks up and copies all files listed in a chunk. When a chunk is exhausted, a new chunk is acquired and processed, until no more chunks remain.
By not assigning a source-path to a fixed map, faster map-tasks (i.e. data-nodes) are able to consume more chunks, and thus copy more data, than slower nodes. While this distribution isn't uniform, it is fair with regard to each mapper's capacity.
The dynamic-strategy is implemented by the DynamicInputFormat. It provides superior performance under most conditions.
Tuning the number of maps to the size of the source and destination clusters, the size of the copy, and the available bandwidth is recommended for long-running and regularly run jobs.
Copying Between Versions of HDFS
For copying between two different versions of Hadoop, one will usually use HftpFileSystem. This is a read-only FileSystem, so DistCp must be run on the destination cluster (more specifically, on TaskTrackers that can write to the destination cluster). Each source is specified as hftp://<dfs.http.address>/<path> (the default dfs.http.address is <namenode>:50070).
Map/Reduce and other side-effects
As has been mentioned in the preceding, should a map fail to copy one of its inputs, there will be several side-effects.
- Unless -overwrite is specified, files successfully copied by a previous map on a re-execution will be marked as "skipped".
- If a map fails mapred.map.max.attempts times, the remaining map tasks will be killed (unless -i is set).
- If mapred.speculative.execution is set set final and true, the result of the copy is undefined.
SSL Configurations for HSFTP sources
To use an HSFTP source (i.e. using the hsftp protocol), a Map-Red SSL configuration file needs to be specified (via the -mapredSslConf option). This must specify 3 parameters:
- ssl.client.truststore.location: The local-filesystem location of the trust-store file, containing the certificate for the namenode.
- ssl.client.truststore.type: (Optional) The format of the trust-store file.
- ssl.client.truststore.password: (Optional) Password for the trust-store file.
The following is an example of the contents of the contents of a Map-Red SSL Configuration file:
<configuration>
<property>
<name>ssl.client.truststore.location</name>
<value>/work/keystore.jks</value>
<description>Truststore to be used by clients like distcp. Must be specified. </description>
</property>
<property>
<name>ssl.client.truststore.password</name>
<value>changeme</value>
<description>Optional. Default value is "". </description>
</property>
<property>
<name>ssl.client.truststore.type</name>
<value>jks</value>
<description>Optional. Default value is "jks". </description>
</property>
</configuration>
The SSL configuration file must be in the class-path of the
DistCp program.
Frequently Asked Questions
-
Why does -update not create the parent source-directory under
a pre-existing target directory?
The behaviour of -update and -overwrite is described in detail in the Usage section of this document. In short, if either option is used with a pre-existing destination directory, the contents of each source directory is copied over, rather than the source-directory itself. This behaviour is consistent with the legacy DistCp implementation as well.
-
How does the new DistCp differ in semantics from the Legacy
DistCp?
- Files that are skipped during copy used to also have their file-attributes (permissions, owner/group info, etc.) unchanged, when copied with Legacy DistCp. These are now updated, even if the file-copy is skipped.
- Empty root directories among the source-path inputs were not created at the target, in Legacy DistCp. These are now created.
-
Why does the new DistCp use more maps than legacy DistCp?
Legacy DistCp works by figuring out what files need to be actually copied to target before the copy-job is launched, and then launching as many maps as required for copy. So if a majority of the files need to be skipped (because they already exist, for example), fewer maps will be needed. As a consequence, the time spent in setup (i.e. before the M/R job) is higher.
The new DistCp calculates only the contents of the source-paths. It doesn't try to filter out what files can be skipped. That decision is put- off till the M/R job runs. This is much faster (vis-a-vis execution-time), but the number of maps launched will be as specified in the -m option, or 20 (default) if unspecified.
-
Why does DistCp not run faster when more maps are specified?
At present, the smallest unit of work for DistCp is a file. i.e., a file is processed by only one map. Increasing the number of maps to a value exceeding the number of files would yield no performance benefit. The number of maps lauched would equal the number of files.
-
Why does DistCp run out of memory?
If the number of individual files/directories being copied from the source path(s) is extremely large (e.g. 1,000,000 paths), DistCp might run out of memory while determining the list of paths for copy. This is not unique to the new DistCp implementation.
To get around this, consider changing the -Xmx JVM heap-size parameters, as follows:
bash$ export HADOOP_CLIENT_OPTS="-Xms64m -Xmx1024m"
bash$ hadoop distcp2 /source /target


Our website is launching very soon. Leave us your email below, and we'll notify the minute we open the doors.


--- layout: page title: "Section 2: Dalvik Thread Model" ---
Thread is an important concept in nowadays programming model. Java uses its own thread model to create and synchronized all the behaviors of the threads. The java's official guide is HERE, and a quick tutorial is HERE.
In short, java thread model has the following features:
In this section, implementation of the thread model in different level will be discussed. General speaking, there are four level of abstractions of threads top-down: (1) Language level (2) Virtual Machine level (3) library level (4) kernel level.
The main class used by developers is Thread whose source code is in the file libcore/luni/src/main/java/java/lang/Thread.java. However, if you see deep into the code, this is only a wrapper for the real thread. It abstracts some interfaces for developers. Another important class is called VMThread whose function is obvious from its name. It manages the thread structure in virtual machine through JNI bridge.
As described above, class VMThread is the thread's delegate in Java level. It jump into c++ code by calling the native function in file dalvik/vm/native/java_lang_VMThread.cpp. This file needs your special attention, because it is the door to the world of Dalvik threads.
The data structure is alway the first part of code you should understand. The thread's data structure is defined in head file dalvik/vm/Thread.h. It seems a little complicated. Nevertheless, the comments here give you a lot of help to understand the function of each field. After brief reading of the structure, you should start reading the code at the function dvmCreateInterpThread(...) in file dalvik/vm/Thread.cpp. This is the entry of creating a virtual machine thread. In this function, a new vm thread will be created and linked with a new pthread thread. After the initialization, it will directly call the run() overrided by its child.
When you reading the code, try to answer following questions in mind:
The pthread library is implemented in the bionic project. Most of the code which may interest you is placed at file bionic/libc/bionic/pthread.c. The data structure of thread in pthread library is called pthread_internal_t defined in file bionic/libc/bionic/pthread_internal.h. Pthread library uses system call clone to fork a thread. Thread is very similar to the conception of process nowadays except they do not have their own address space. All the threads in the same process share the memory addresses with each other. This can be bot good (easy programming) and bad (heavy contention on memory layout data structure).
We do not mention the structure of thread in kernel here, because default source code of Android do not include such files. While many articles talked about the design and implement of thread structure in kernel. It is pretty convenient to find them.
Above all, from this perspective, all the abstractions in language, virtual machine and library are all wrappers of thread entity in kernel. However, this is not always the case. In the preliminary version of JVM, virtual machine maintains its own threads in its own process and does all the scheduling itself. It seems no good other than increasing complexity to do this. However, there are some reasons for this. Maintaining the threads in higher level have some performance benefit in a situation with huge amount of threads. This is because creating threads and scheduling them in kernel is not free. Trapping into kernel is often an unwelcoming behavior. And the memory contention is another reason to keep thread more lightweight. For other language, we can also see the feature similar to thread but more simple, such as coroutine. And this is maintained inside virtual machine most of the time, such as Lua.


Next: boundp, Previous: get, Up: Symbols Dictionary
remprop
symbol indicator ⇒ generalized-boolean
symbol—a symbol.
indicator—an object.
generalized-boolean—a generalized boolean.
remprop removes from the property list_2 of symbol a property_1 with a property indicator identical to indicator.
If there are multiple properties_1 with the identical key, remprop only removes the first such property.
remprop returns false if no such property was found, or true if a property was found.
The property indicator and the corresponding property value are removed in an undefined order by destructively splicing the property list.
The permissible side-effects correspond to those permitted for remf, such that:
(remprop x y) ≡ (remf (symbol-plist x) y)
(setq test (make-symbol "PSEUDO-PI")) ⇒ #:PSEUDO-PI (symbol-plist test) ⇒ () (setf (get test 'constant) t) ⇒ T (setf (get test 'approximation) 3.14) ⇒ 3.14 (setf (get test 'error-range) 'noticeable) ⇒ NOTICEABLE (symbol-plist test) ⇒ (ERROR-RANGE NOTICEABLE APPROXIMATION 3.14 CONSTANT T) (setf (get test 'approximation) nil) ⇒ NIL (symbol-plist test) ⇒ (ERROR-RANGE NOTICEABLE APPROXIMATION NIL CONSTANT T) (get test 'approximation) ⇒ NIL (remprop test 'approximation) ⇒ true (get test 'approximation) ⇒ NIL (symbol-plist test) ⇒ (ERROR-RANGE NOTICEABLE CONSTANT T) (remprop test 'approximation) ⇒ NIL (symbol-plist test) ⇒ (ERROR-RANGE NOTICEABLE CONSTANT T) (remprop test 'error-range) ⇒ true (setf (get test 'approximation) 3) ⇒ 3 (symbol-plist test) ⇒ (APPROXIMATION 3 CONSTANT T)
The property list of symbol is modified.
Should signal an error of type type-error if symbol is not a symbol.
Numbers and characters are not recommended for use as indicators in portable code since remprop tests with eq rather than eql, and consequently the effect of using such indicators is implementation-dependent. Of course, if you’ve gotten as far as needing to remove such a property, you don’t have much choice—the time to have been thinking about this was when you used setf of get to establish the property.
Next: boundp, Previous: get, Up: Symbols Dictionary


"""
pyhull is a Python wrapper to Qhull (http://www.qhull.org/) for the
computation of the convex hull, Delaunay triangulation and Voronoi diagram.
Useful low-level functions are implemented for direct import in the base
package and can be called as pyhull.qconvex, pyhull.qdelauany, etc.
"""
__author__ = "Shyue Ping Ong"
__version__ = "1.5.4"
__date__ = "Sep 22 2014"
__maintainer__ = "Shyue Ping Ong"
__email__ =<EMAIL_ADDRESS>import pyhull._pyhull as hull
[docs]def qhull_cmd(cmd, options, points):
"""
Generalized helper method to perform a qhull based command.
Args:
cmd:
Command to perform. Supported commands are qconvex,
qdelaunay and qvoronoi.
options:
Options to be provided for qhull command. See specific methods for
info on supported options. Up to two options separated by spaces
are supported.
points:
Sequence of points as input to qhull command.
Returns:
Output as a list of strings. E.g., ['4', '0 2', '1 0', '2 3 ', '3 1']
"""
prep_str = [str(len(points[0])), str(len(points))]
prep_str.extend([' '.join(map(repr, row)) for row in points])
output = getattr(hull, cmd)(options, "\n".join(prep_str))
return list(map(str.strip, output.strip().split("\n")))
[docs]def qconvex(options, points):
"""
Similar to qconvex command in command-line qhull.
Args:
options:
An option string. Up to two options separated by spaces
are supported. See Qhull's qconvex help for info. Typically
used options are:
s - summary of results (default)
i - vertices incident to each facet
n - normals with offsets
p - vertex coordinates (includes coplanar points if 'Qc')
points:
Sequence of points as input.
Returns:
Output as a list of strings. E.g., ['4', '0 2', '1 0', '2 3', '3 1']
"""
return qhull_cmd("qconvex", options, points)
[docs]def qdelaunay(options, points):
"""
Similar to qdelaunay command in command-line qhull.
Args:
options:
An options string. Up to two options separated by spaces
are supported. See Qhull's qdelaunay help for info. Typically
used options are:
s - summary of results (default)
i - vertices incident to each Delaunay region
o - OFF format (shows the points lifted to a paraboloid)
points:
Sequence of points as input.
Returns:
Output as a list of strings.
E.g., ['4', '2 4 0', '4 1 0', '3 4 2', '4 3 1']
"""
return qhull_cmd("qdelaunay", options, points)
[docs]def qvoronoi(options, points):
"""
Similar to qvoronoi command in command-line qhull.
Args:
option:
An options string. Up to two options separated by spaces
are supported. See Qhull's qvoronoi help for info. Typically
used options are:
s - summary of results
p - Voronoi vertices
o - OFF file format (dim, Voronoi vertices, and Voronoi regions)
points:
Sequence of points as input.
Returns:
Output as a list of strings.
E.g., ['2', '5 5 1', '-10.101 -10.101', '0 -0.5', '-0.5 0',
'0.5 0', '0 0.5', '3 2 0 1', '3 4 0 2', '3 3 0 1',
'3 4 0 3', '4 4 2 1 3']
"""
return qhull_cmd("qvoronoi", options, points)
[docs]def qhalf(options, halfspaces, interior_point):
"""
Similar to qvoronoi command in command-line qhull.
Args:
option:
An options string. Up to two options separated by spaces
are supported. See Qhull's qhalf help for info. Typically
used options are:
Fp
halfspaces:
List of Halfspaces as input.
interior_point:
An interior point (see qhalf documentation)
Returns:
Output as a list of strings.
E.g., ['3', '4', ' 1 1 0 ', ' 1 -1 2 ',
' -1 1 2 ', ' 1 1 2 ']
"""
points = [list(h.normal) + [h.offset] for h in halfspaces]
data = [[len(interior_point), 1]]
data.append(map(repr, interior_point))
data.append([len(points[0])])
data.append([len(points)])
data.extend([map(repr, row) for row in points])
prep_str = [" ".join(map(str, line)) for line in data]
output = getattr(hull, "qhalf")(options, "\n".join(prep_str))
return list(map(str.strip, output.strip().split("\n")))


My name is Fajar Norman Harun i'm 23 years old, I Highly motivated front-end developer, i like to explore something new, i started to learn front-end development is on senior high school (SMK PGRI 3) from Since that time I began to explore more and more.


--- layout: default title: Index ---
I am a Marie Skłodowska-Curie Postdoctoral Fellow at the University of Lisbon's Institute of Social Sciences (ICS). More about me and my research.
Contact: guilherme.lambais at ics.ulisboa.pt


Define a media distribution policy in accordance with PCI DSS. Classify sensitive media as "confidential". Log media distribution. Send sensitive media via secured courier or any other delivery method that can be tracked.
A media distribution policy should be defined to facilitate the making of decisions in a timely and secure manner when media has to be distributed. The policy should cover all types of media storage to ensure that the policy provides adequate guidance for all likely situations. Sensitive media should be classified to make it clear to the recipient that it should not be disclosed. Media distribution should be logged to help audit media distribution events if needed. Media should be sent via secured courier or any other delivery method that can be tracked to assure that the media reaches its recipient, to assist with locating it if it is somehow lost, and to have greater control of the delivery process by being able to monitor it and make calls.
Use the following steps to update policy with secure media distribution requirements:
Identify media distribution requirements. Identify the possible scenarios which would require media containing sensitive information to be distributed, for example, an external audit.
Document media distribution procedures. Write down the correct procedures for handling each scenario and add them to the media policy.
Update the policy. Update the information security policy with the media distribution requirements. Add a point to the media policy that requires sensitive media to be classified as "confidential." Add a point to the media policy that requires keeping a log of distributed media. Add a point to the media policy that requires sending sensitive media via secured courier or some other shipping method that can be tracked. Attach the media policy to the information security policy.
Keep a log. Start a log of distributed media that includes the following information: the name of the person authorizing distribution, the name of the intended recipient, the contents of the media, the date, and any other relevant information. Attach the media distribution log to the network documentation.
Comply with the policy. Follow the media distribution policy when distributing sensitive media.


CHANGELOG¶
The following is a summary of changes and improvements to the Finding Aids application. New features in each version should be listed, with the most recent version first. Upgrade or deployment notes should be found in DEPLOYNOTES.
1.11¶
- Change default stylesheet display text for <dao> objects.
1.10.1¶
- Fix whitespace prep and publication check to handle top-level unittitles that include a nested <title> or <emph> tag.
1.10 - Maintenance updates¶
- Update to use migrated taskresult module in eulcommon instead of eullocal
- Install eullocal from private github instead of old CI server
- bugfix: Finding aids containing leading whitespace in the title or main entry of a collection break browse functionality (and the homepage). [Now normalizing whitespace before selecting or filtering on first letter]
- bugfix: Archivists can publish finding aids to the finding aids database that do not pass all the checks from the prep stage. [Now explicitly checking for whitespace in top-level unittitle]
1.9.1¶
- Prep step returns an incorrect message for new finding aids.
1.9.0¶
- As a user, I can see a banner on individual finding aids where materials are stored offsite so that I know retrieval of these materials will take longer.
- As a user, I can see a scope note at the folder level of the finding aid in order to see additional information the archivist has added about that folder’s contents.
- As an administrator, I can change staff contacts listed in the finding aids database website so that I can manage the names and emails displayed on the request materials page.
- As an administrator I can initiate a user account from LDAP without requiring the person to log in so that I can manage user accounts more efficiently.
- bugfix: whitespace inconsistencies in keyword search results
- bugfix: browse by letters label inconsistency in certain rare cases (complete label with partial overlap in next label)
- Upgraded to eulexistdb 0.20 and eXist-db 2.2
- Upgrade to current Django 1.9
- Switched from local ldap implementation to django-auth-ldap
- Now configured to use django-debug-toolbar for development, with a panel for viewing existdb xqueries.
1.8.3¶
- Text content changes for MARBL rename to Rose Library, along with other contact and name updates.
- Updated home page banner image for Pitts.
1.8.2¶
- bugfix: correct RDFa for untitled container-level contents with tagged creator name.
1.8.1¶
- bugfix: repeating biographical note in PDFs for documents with RDFa
- bugfix: Names in correspondence series <scopecontent> are no longer linked to <origination> in the RDF.
1.8 - RDFa enhancements¶
- As a researcher, I want to export title information from a finding aid so that I can analyze and organize the works in a person’s papers according to genre.
- As a researcher, I want to export titles’ ISBN, ISSN, or OCLC numbers from a finding aid so that I can access the data associated with those publications.
- As a researcher, I want to export semantic, relational data from EAD so that I can analyze it in network graph software like Gephi. [See the new script in scripts/rdfa-to-gexf]
- bugfix: PDF generation display issue for long numbers in box/folder
- bugfix: Request materials button not appearing for appropriate series.
- Replaced current finding aids logo with the standard LITS logo.
- Update Database Group initial fixture data to avoid overwriting new production group definitions
1.7.4¶
- made related_materials repeatable
1.7.3 - Aeon integration¶
- As a user viewing individual finding aids in the finding aids database, I can click on a button in a finding aid that takes me into the Aeon system in order to request boxes I want to use.
1.7.2¶
- Minor text corrections on the site home page.
- bugfix: updated the ead_load script so it can be used with the subversion architecture for EAD content that we switched to in 1.5
1.7.1¶
- bugfix: small corrections to itemid_to_dao script for issues encountered in production that were not found in testing
- bugfix: RDFa collection resource should be URI instead of string
1.7 - administrative and internal digital archival object support¶
- When a logged in administrative user is searching the finding aids database, they can optionally restrict results to all collections with digital objects.
- Researchers will be able to see identifiers for digitized content that can only be accessed in the reading room, so that they can search for those items in the kiosk.
- When an administrative user is viewing the web page for a single finding aid, they can search for digital objects within that finding aid either by entering a keyword/keyword phrase in combination with the digital objects only filter or by just searching for all the digital objects in the finding aid.
- When an administrative user is logged into the finding aids database, they will see an indication that they are logged in on each page of the finding aids website in order to remember which view (administrative or public) they are seeing.
- When administrative users view webpages for a finding aid with digital archival object references in the EAD document, they can distinguish when the <dao> display in the administrative view is different from the public view and when <dao> information is hidden from the public view completely.
- A logged in findingaids staff user will be able to see links for internal-only digitized content, including any that are suppressed from display for researchers, so that they can access and manage archival digitized content.
- When administrative users view webpages for a finding aid with digital archival object references in the EAD document, they can see and click on links in order to access all the digital archival objects created or owned by Emory University, both public and for internal use only.
- A developer or system administrator can run a script on all EAD finding aids to add digital archival object references for container-level items that include digital filenames in order to provide administrative access to digitized content and prepare for future access options.
- As a researcher, I want to be able to see archivist supplied notes about processing at the series and subseries level so that I can understand how to locate material within a collection.
- New banners from the home page.
- bugfix: series titles with quotes break RDFa output
- bugfix: As a researcher, I want to be able to see any notes associated with index content, so that I have any additional information provided by the archivists.
- bugfix: As a researcher, I want to be able to see archivist supplied notes about processing at the series and subseries level so that I can understand how to locate material within a collection.
- bugfix: correct RDFa relation between collection and creator of the collection.
1.6¶
- Remove dependency on RSS feeds for content pages, home page banners, and home page news items.
- Updated organizational logo in the header.
- bugfix: Loading the list of finding aid files from subversion in the tabbed display of the administrative interface is too slow.
- bugfix: display repeated <separatedmaterial> sections (formerly only the first one was being displayed)
- bugfix: if a login session times out, admin file tabs will not load; now redirects user to login again
1.5.1¶
- bugfix: configure celery task added in 1.5 for svn checkout so it is directed to findingaids queue so it gets picked up by the celery worker.
1.5¶
- A superuser can associate an archivist with one or more archives and their corresponding subversion repositories so that archivists can preview and publish for only the archives they are assigned.
- Superusers of the finding aids database can easily navigate between the django administrative module and the finding aids database administrative module.
- When an archivist associated with a single archive logs in to the admin site, they see a list of EAD documents in the associated subversion repository and are able to prep, preview, and publish finding aids from that repository.
- When an archivist associated with multiple archives logs in to the admin site, they see a tab for each corresponding subversion repository, each with a list of EAD documents in that subversion repo for prep, preview and publication.
- An archivist with permissions on multiple archives can choose a primary archive so they can view EAD documents from that repository by default when they log in.
- When an archivist preps an EAD document that requires modifications, they can click a button to accept and apply those changes so that updates are automatically committed to subversion on their behalf.
- Archivists can only prep, preview, and publish EAD documents from the subversion repositories they have been associated with in the admin site, for security.
- An archivist can only delete a published finding aid from the website if it is associated with an archive that they have permission to manage, for security reasons.
- When an archivist with permissions for multiple repositories selects a repository tab on the main admin page, that repository tab stays active until they select another repository or logout, in order to avoid confusion when prepping, previewing, and publishing content from a single repository.
- Deprecated and unused support for publish without preview has been removed.
- New celery task: when an archive is defined or updated, check out (or update) a local copy of the subversion repository.
- bugfix: add redirect for top-level /favicon.ico url
1.4.1¶
- bugfix to correct PDF generation (broken in some cases due to template changes relating to RDFa output)
1.4 - RDFa¶
- When a search engine accesses the web page for a finding aid, it can harvest semantic information about the finding aid document, so that the document can be related to other embedded semantic content.
- A system or technical user can view RDF XML based on the embedded RDFa in a finding aid page, in order to harvest RDF in a more standard format or to review the embedded data on the page.
- When a search engine accesses the web page for a finding aid with names tagged in the EAD, it can harvest semantic information about the originator of the finding aid from the collection description so the embedded data can become useful in another context.
- When a search engine accesses the Index of Selected Correspondents for a finding aid with names tagged in the EAD, it can harvest semantic information about correspondents with the originator of the finding aid so the embedded data can become useful in another context.
- When a search engine accesses a finding aid series describing correspondence for a finding aid with names tagged in the EAD, it can harvest semantic information about correspondents with the originator of the finding aid so the embedded data can become useful in another context
- When a search engine accesses the finding aid series for the Belfast Group Worksheets for a finding aid with names tagged in the EAD, it can harvest semantic information about participants in the group so the embedded data can become useful in another context.
- When a search engine accesses the finding aid series for the Belfast Group Worksheets for a finding aid with names tagged in the EAD, it can harvest group sheet titles in order so that title sequence can be preserved.
1.3¶
- When a researcher is viewing web pages or PDF documents for a finding aid with digital archival object references in the EAD document, they can see and click on links in order to access digital items associated with the collection.
- A researcher searching within a single finding aid can optionally restrict results to items that include digital objects in order to find digital content by keyword or all digital objects in one finding aid.
- A researcher searching all finding aids by keyword can optionally restrict results to collections with publicly accessible digital objects in order to make use of archival items available online.
- A researcher viewing the web page or PDF for a finding aid with digital archival object references in the EAD can see that the finding aid includes digital content by a header at the top of the page, so that they are aware some of the materials may be available online.
- Users can view PDF documents from within a browser so they can view, print, and save the entire finding aid quickly and easily.
- Configurable beta warning to be displayed in test/staging sites; turn on via ENABLE_BETA_WARNING setting.
1.2¶
- Updated to Django 1.5.
- When a user is viewing web pages for a finding aid with external references (extref tags) in the EAD document, they can see and click on links in order to access external webpages.
- When a user downloads the PDF for a finding aid with external references in the EAD document, they can click on links in case they want to access related content from the PDF.
- Automated tools, such as search engine robots or site crawlers, can find machine-readable site maps for findingaids and content pages, in order to improve search engine harvesting of finding aids site content.
- Bug fix: related material section should be displayed when present at series level
1.1¶
- When an admin is logged in, they will see a link to the admin page at the top of the left sidebar, so that they can always get back to the main admin page.
- A researcher viewing the HTML or PDF version of a finding aid can see the processing information from the EAD, so that they know who is responsible for the content.
- Updated to Django 1.4.2
- Updates to follow team best practices for Django project code organization:
- Moved media directory to top-level sitemedia directory
- Moved templates directory to top-level and moved app-specific templates into their respective apps.
- Renamed localsettings.py example from
localsettings-sample.py
tolocalsettings.py.dist
1.0 micro releases¶
1.0.12¶
- Catch exceptions when reloading cached content feed data.
1.0.11¶
- Adjust the XQuery for single-document searches to be more efficient for large documents, in order to address a time-out issue identified in SCLC1083.
1.0.10¶
- Better error-handling for empty list title in EAD when prepping for preview/load.
- Require eulxml 0.17.1 for improved xpath parser handling.
1.0.9¶
- Now compatible with Python 2.7
- Upgrade to Django 1.3 and the latest released versions of the
broken-out eulcore modules (
eulxml
,eulexistdb
, andeulcommon
). - Minor error-handling and search-engine optimization for the feedback page.
- Rewrite rule to handle non-existent URL (‘-Libraries-EmoryFindingAids’) that search engines follow from other Emory sites.
- Add a reset button to the advanced search form so that a selected repository can be unselected.
1.0.8¶
- bugfix: allow admin publication of documents with a
<title>
at the beginning of the document<unittitle>
- bugfix: Revised logic for celery PDF caching task, to ensure cache is cleared and reloaded with the new version of a published document.
- Plain HTML page with a list of all published findingaids, with a link to the full EAD xml for each, as a simple way to allow harvesting content.
1.0.6¶
- Newer version of
eulcore.existdb
that adds a configurable timeout on queries made to the eXist database.
1.0.5¶
- Fix response-time issue for series/subseries page with highlighted search terms.
- Rework admin site preview mode logic so site cannot get stuck in preview mode.
- Use pip+virtualenv to manage dependencies like eulcore.
1.0.4¶
- Fix preview subseries link so it stays in series mode
- Update to eulcore to try to improve xpath error reporting for errors that are being generated on the prodution site by web spiders.
1.0.3¶
Minor usability and display tweaks: * Show all alpha-browse page labels instead of only 9 * Brief search tips on the main page
1.0.2¶
- Fix character corruption issue in origination field on main finding aid page.
1.0.1¶
- Correct single-doucment search for simple finding aids with no series.
1.0 Site Design & Content¶
- Users can view additional pages maintained by the finding aids administrator which contain helpful information for regarding searching, defining terms, participating institutions, etc.
- User visiting the homepage sees one of several archivist-selected images (rotate randomly on page refresh) to market unique items in MARBL’s collections.
- A user visiting the Finding Aids home page will see the most recent archivist- entered/created announcement (if any), in order to receive up-to-date news about special events or notifications about site downtime.
- Researchers can submit feedback relating to the website site from the main homepage to help improve content and functionality.
- When a researcher is viewing a single finding aid, they can submit feedback to help correct typos and errors in the text or provide additional information which may be helpful to future researchers.
- Prospective visitors/researchers can submit a request for materials to facilitate retrieval prior to their arrival, which will be routed to the appropriate repository via email.
- Researchers can select a repository (other than ‘All’) on the advanced search form and submit the form without entering any other search terms, in order to browse all finding aids from a single repository.
- Users view html and PDF versions of finding aids that are consistently and cleanly formatted and displayed according to MARBL formatting requirements.
0.4.1 Unitid Identifiers¶
- Custom manage command to add machine-readable identifiers to the top-level unitid tag.
0.4 Persistent IDs¶
- A system administrator can run a command that will generate ARKs for all existing EAD documents that do not already have ARKs to update the documents and store the ARK in the appropriate eadid attribute.
- When an archivist runs the ‘prep’ step in the Finding Aid admin site, an ARK will be generated and added to the ‘prepared’ EAD.
- When an archivist runs the ‘prep’ step on a Finding Aid with no ARK stored in the EADID, but for which an ARK has already been generated, the existing ARK will be used and the archivist will see an explanatory message.
- When an archivist attempts to publish a Finding Aid without an ARK stored in the EADID, the document will not be published and the archivist will see an explanatory message.
- A researcher or search engine accessing a Finding Aid document has access to view and bookmark the permanent url for that document.
- When researchers try to use the Emory Finding Aids Database and it is down, they will see a message about the problem and who to contact.
0.3 Enhanced Search¶
- When viewing a finding aid after a search, a researcher can easily find search terms and exact phrases because they are highlighted.
- When viewing a finding aid after a search, a researcher sees an indicator of which sections of the finding aid include their search terms.
- A system administrator can run a script to migrate EAD files in the configured source directory from EAD DTD format to EAD XSD schema.
- When an admin cleans, publishes, or previews an schema-based EAD document, the application validates against the XSD schema.
- Researchers can retrieve an alphabetical browse list in less than 5 seconds, based on the first letter of a stakeholder specified field.
- Researchers receive their search results in less than 5 seconds.
- Researchers can see how many pages of search results there are, and jump to any section of search results from any page in the search results.
- When viewing a finding aid with series or sub-series, a researcher can use breadcrumbs to navigate within the hierarchy of the document.
- Researchers can search for an exact phrase in all indexed fields in the full text of the finding aid, to allow targeted discovery.
- Researchers can search using wildcards to match partial or variant words.
- Researchers can use grouping and boolean operators in the main search input, to generate very precise, relevant search results.
- Researchers find finding aids with matches in stake-holder specified fields at the top of search results.
- When viewing a finding aid, a researcher can search within that one document, to find relevant folder contents in a large finding aid.
- Researchers can click on a subject heading (any of the controlaccess terms) in a single finding aid to discover other finding aids with the same subject headings.
- When browsing finding aids by any first letter, a researcher can jump to alphabetical groupings within that letter, to enable identifying and accessing a particular portion of that browse listing (e.g., A-Ar, As-Ax, etc.).
- When viewing a finding aid found via search, a researcher can get back to the last page of search results they were on.
- Researchers can filter their search by repository (MARBL, Pitts, University Archives, etc.), to find resources available at a specific location.
- Users interact with a site that has a consistent look and feel across Emory Libraries websites.
Minor changes
- Pisa/ReportLab PDF generation has been replaced with XSL-FO and Apache FOP.
- Logging now available in runserver
- Clean urls for series/subseries/index (without redundant eadid)
- Includes a prototype version simplepages for editable site content
0.2 Data Preparation / Admin site¶
Replaces the legacy command-line ant process for validating EAD xml data and loading it to the eXist database.
- An authorized archivist can log in to an admin section of the finding aids site inaccessible to other users.
- Logged in admins can view a list of finding aid files recently modified on F:and ready for upload, sorted by last modified.
- Logged in admins can select files from the recently modified list for upload directly to publication.
- Logged in admins can select a file from the recently modified list for preparing, see a list of changes made, and optionally download the prepared version if changes were made, in order to safely prepare the canonical copy of the EAD xml files.
- Logged in admins can select files from the recently modified list for preview; multiple admins can preview different documents simultaneously.
- An admin previewing a finding aid can click a link (on any page in a multi-page finding aid) to publish that document.
- When an admin tries to publish or preview an invalid finding aid, the user sees a meaningful error message directing them how to fix it.
- When the web application is unable to save a finding aid, the user sees a meaningful message describing the problem and how to proceed.
- Logged in admins can view a minimal alphabetical list of published finding aids.
- Logged in admins can select a finding aid for deletion from the alphabetical list of published finding aids.
- When a collection is removed from the production site, patrons accessing their URLs are referred to MARBL staff for collection status.
- Researchers can receive a pdf of a finding aid in less than 10 seconds.
- A search engine or web crawler can harvest descriptive metadata based on the EAD contents along with the HTML data, to improve google-ability.
- A system administrator can run a command to prepare all or specified EAD xml files in the configured directory, in order to easily update all existing files to new standards.
- A system administrator can run a command to load all or specified EAD xml files in the configured source directory to the configured eXist collection, in order to easily populate a new eXist collection
0.1 Port to Django¶
Reimplementation of the functionality of the existing PHP Finding Aids site in django and eXist 1.4.
- Researchers can browse finding aids alphabetically by first letter of title.
- Researchers can click on the title of a finding aid in search or browse results to view more details about what resources are available in that collection.
- Researchers can search finding aids by keyword.
- Developers can access EAD XML objects in an eXist-backed Django Model workalike.
- Researchers can click ‘download PDF’ when viewing a single finding aid to download a PDF version of the entire finding aid.
- Researchers can navigate through finding aid site with the same look and feel of the library site.
- When a researcher clicks on an old link to a drupal or pre-drupal finding aid URL, they are automatically redirected to new finding aid URLs.


Getting Started
Claire and I are easily distrated by new work. With the tendency to be attracted to what's newest, it's easy to question whether we'll ever be content or satisfied. To make matters more complicated, we both came upon our occupations as hobbies. The line between work and pleasure has become rather thin. The thinner it gets, the more work has by necessety become a matter of passion.
So, here we are. Passion is what I call the line between work and pleasure. But, think of it not as a divider, but as a combiner.
In my mind, it's the only reason I feel a semplence of satisfaction at the end of the day. If passion has done its job, work and pleasure/hobbies worked hand in hand. It's how I feel rest and peace when I lay my head down at night
To be honest, it's a rare day that work lines up with personal projects or hobbies. But should it always be that way?
This project, Passionately, is about exploring the depth of what's behind great art. Can great art only be made by artists who are willing to step outside of what makes them happy? Must great artists feel lack of satisfaction in order to acheive success? Is the melenial dream of pleasure making money only a dream?
We're going to interview as many artists as time and money will allow between April and August of 2013 to try to find some answers.


An example of using
<content-editable>
:
I'm an editable text!
I'm a span that's also editable
I'm a link
An editable paragraph of text.
An div with text inside.


| has.OHLC {quantmod} | R Documentation |
A set of functions to check for appropriate OHLC and HLC column names within a data object, as well as the availability and position of those columns.
is.OHLC(x) has.OHLC(x, which = FALSE) is.OHLCV(x) has.OHLCV(x, which = FALSE) is.HLC(x) has.HLC(x, which = FALSE) has.Op(x, which = FALSE) has.Hi(x, which = FALSE) has.Lo(x, which = FALSE) has.Cl(x, which = FALSE) has.Vo(x, which = FALSE) has.Ad(x, which = FALSE) is.BBO(x) is.TBBO(x) has.Ask(x, which = FALSE) has.Bid(x, which = FALSE) has.Price(x, which = FALSE) has.Qty(x, which = FALSE) has.Trade(x, which = FALSE)
x |
data object |
which |
disply position of match |
Mostly used internally by quantmod, they can be useful
for checking whether an object
can be used in OHLC requiring functions like Op
,
OpCl
, etc.
Columns names must contain the full description of data,
that is, Open, High, Low, Close, Volume or Adjusted. Abbreviations
will return FALSE
(or NA
when which=TRUE). See
quantmod.OHLC
for details of quantmod
naming conventions.
is.OHLC
(and is.HLC, similarly) will only return TRUE is there are columns
for Open, High, Low and Close. Additional columns will not affect
the value.
A logical value indicating success or failure by default.
If which=TRUE, a numeric value representing the column position will be returned.
is.OHLC
and is.HLC
return a single value of TRUE or FALSE.
Jeffrey A. Ryan
quantmod.OHLC
,OHLC.Transformations
## Not run: getSymbols("YHOO") is.OHLC(YHOO) has.OHLC(YHOO) has.Ad(YHOO) ## End(Not run)


RT @MariBrighe: If you get punched by a raccoon, ICD10 has you covered.
9/10/2019, 1:48:26 PM
Favs: 0
Retweets: 0
link--- layout: tweet ---
RT @MariBrighe: If you get punched by a raccoon, ICD10 has you covered.
9/10/2019, 1:48:26 PM
Favs: 0
Retweets: 0
link


ASTableDataSource Protocol Reference
| Conforms to | ASCommonTableDataSource NSObject |
|---|---|
| Declared in | ASTableNode.h |
– numberOfSectionsInTableNode:
Asks the data source for the number of sections in the table node.
- (NSInteger)numberOfSectionsInTableNode:(ASTableNode *)tableNode
Declared In
ASTableNode.h
– tableNode:numberOfRowsInSection:
Asks the data source for the number of rows in the given section of the table node.
- (NSInteger)tableNode:(ASTableNode *)tableNode numberOfRowsInSection:(NSInteger)section
Declared In
ASTableNode.h
– tableNode:nodeBlockForRowAtIndexPath:
Asks the data source for a block to create a node to represent the row at the given index path. The block will be run by the table node concurrently in the background before the row is inserted into the table view.
- (ASCellNodeBlock)tableNode:(ASTableNode *)tableNode nodeBlockForRowAtIndexPath:(NSIndexPath *)indexPath
Parameters
tableNode |
The sender. |
|---|---|
indexPath |
The index path of the row. |
Return Value
a block that creates the node for display at this indexpath. Must be thread-safe (can be called on the main thread or a background queue) and should not implement reuse (it will be called once per row).
Discussion
Note: This method takes precedence over tableNode:nodeForRowAtIndexPath: if implemented.
Declared In
ASTableNode.h
– tableNode:nodeForRowAtIndexPath:
Asks the data source for a node to represent the row at the given index path.
- (ASCellNode *)tableNode:(ASTableNode *)tableNode nodeForRowAtIndexPath:(NSIndexPath *)indexPath
Parameters
tableNode |
The sender. |
|---|---|
indexPath |
The index path of the row. |
Return Value
a node to display for this row. This will be called on the main thread and should not implement reuse (it will be called once per row). Unlike UITableView’s version, this method is not called when the row is about to display.
Declared In
ASTableNode.h
– tableView:nodeForRowAtIndexPath:
Similar to -tableView:cellForRowAtIndexPath:.
- (ASCellNode *)tableView:(ASTableView *)tableView nodeForRowAtIndexPath:(NSIndexPath *)indexPath
Parameters
indexPath |
The index path of the requested node. |
|---|---|
tableNode |
The sender. |
Return Value
a node for display at this indexpath. This will be called on the main thread and should not implement reuse (it will be called once per row). Unlike UITableView’s version, this method is not called when the row is about to display.
Declared In
ASTableNode.h
– tableView:nodeBlockForRowAtIndexPath:
Similar to -tableView:nodeForRowAtIndexPath: This method takes precedence over tableView:nodeForRowAtIndexPath: if implemented.
- (ASCellNodeBlock)tableView:(ASTableView *)tableView nodeBlockForRowAtIndexPath:(NSIndexPath *)indexPath
Parameters
tableView |
The sender. |
|---|---|
indexPath |
The index path of the requested node. |
Return Value
a block that creates the node for display at this indexpath. Must be thread-safe (can be called on the main thread or a background queue) and should not implement reuse (it will be called once per row).
Declared In
ASTableNode.h
– tableViewLockDataSource:
Indicator to lock the data source for data fetching in async mode. We should not update the data source until the data source has been unlocked. Otherwise, it will incur data inconsistency or exception due to the data access in async mode. (Deprecated: The data source is always accessed on the main thread, and this method will not be called.)
- (void)tableViewLockDataSource:(ASTableView *)tableView
Parameters
tableView |
The sender. |
|---|
Declared In
ASTableNode.h
– tableViewUnlockDataSource:
Indicator to unlock the data source for data fetching in asyn mode. We should not update the data source until the data source has been unlocked. Otherwise, it will incur data inconsistency or exception due to the data access in async mode. (Deprecated: The data source is always accessed on the main thread, and this method will not be called.)
- (void)tableViewUnlockDataSource:(ASTableView *)tableView
Parameters
tableView |
The sender. |
|---|
Declared In
ASTableNode.h


This website does not store any information from you or your YNAB account on it's server. All data retrieved from the YNAB API is stored only in your browser's local storage and is never transmitted to any other location or third-party.


Reservation Calendar
Start Time
-
End Time:
-
Customer:
Your reservation can be publish?
Yes
No
Purpose:
username


Last updated: 2018-01-08
Code version: 327e818
See single sample T-test for \(d = X_1 - X_2\)
interval, continous
normal
\[t = \frac{\bar{Y_{1}} - \bar{Y_{2}}}{\bar{{SE}}_{difference}}\]
\(t\) follows Student’s T-distribution with n1+n2-2 degrees of freedom
\[{SE}_{difference} = s_{pooled}*\sqrt{\frac{1}{n_{1}}+\frac{1}{n_{2}}}\]
\[s_{pooled} = \frac{(n_{1}-1)*s_{1}^2 + (n_{2}-1)*s_{2}^2}{n_{1}+n_{2}-2}\]
t, follows Student’s T-distribution with df degrees of freedom
interval, continous
normal
interval, continous
normal
\[t = \frac{\bar{Y_{1}} - \bar{Y_{2}}}{\bar{{SE}}_{difference}}\]
\[{SE}_{difference} = \sqrt{\frac{s_1^2}{n_{1}}+\frac{s_2^2}{n_{2}}}\]
\[{df} = \frac{(\frac{s_{1}^2}{n_{1}} + \frac{s_{2}^2}{n_{2}})^2}{({s_{1}^2}/{n_{1}})^2/(n_{1}-1) + ({s_{2}^2}/{n_{2}})^2/(n_{2}-1)}\]
Calculate one-sided directly from binomial distribution for some \(H_0\) value of \(\pi =\pi_0\). \[P(R \geq r | \pi_0) = \sum_{k = r}^n{{n\choose{k}}\pi_0^k(1-\pi_0)^{n-k}}\] Where \(n\) the sample size.
\[\hat{\pi} \pm Z_{\alpha(2)}\sqrt{\frac{\hat{\pi}(1-\hat{\pi})}{n}}\]
Add 2 successess and 2 failures (for 95%-confidence interval)
\(r' = r + \frac{Z_{\alpha(2)}^2}{2}\); \(n' = n + Z_{\alpha(2)}^2\); \(\hat{\pi}' = r'/n'\)
Plug \(\pi\) in Wald formula This moves the estimate a little closer to 0.5. Most useful for extreme p-values.
Has better coverage. Report \(\pi\) and the confidence interval based on \(r'\) and \(n'\).
For paired data. Take discordant pairs only. \(H_0:\ \pi_0-\pi_1 = 0 \equiv H_0:\ \pi_0=0.5\)
\[p_{value} = 2*min \{ P(R \leq r|\pi_0=0.5),P(R \geq r|\pi_0=0.5)\}\]
Proportions in unpaired design
\[\hat{\pi_1}-\hat{\pi_2} \pm Z_{\alpha(2)}\sqrt{\frac{\hat{\pi_1}(1-\hat{\pi_1})}{n_1} + \frac{\hat{\pi_2}(1-\hat{\pi_2})}{n_2}}\]
Add 1 success and failure in each group for 95% confidence interval. \(r_i' = r_i + \frac{Z_\alpha(2)^2}{4}\); \(n_i' = n_i + \frac{Z_{\alpha(2)}^2}{2}\)
Use Wald formula with adjusted values for \(\pi_1\) and \(\pi_2\).
Is best way for determining confidence intervals for proportions. Single proportion: binom::binom.confint(..., method = "wilson")
Difference between proportions: Epi::ci.pd(..., method = "Nc")
Independence of categorical dimensions \(A\) and \(B\) \[H_0:\ P(A=A_i,B=B_j) = P(A=A_i)*P(B=B_j)\]
\[T = \sum_{i=1}^I\sum_{j=1}^J{\frac{(O_{ij}-E_{ij})^2}{E_{ij}}}\]
Where
\[E_{ij} = \sum^I_{k=1}{O_{ik}}*\sum_{l=1}^J{O_{lj}} = P(A=A_i)*P(B=B_i)\]
\(T\) is asymptotically chi-squared distributed with \((I-1)(J-1)\) degrees of freedom, where \(I\) and \(J\) are the numbers of discrete values that \(A\) and \(B\) can take.
\[S(t) = 1 - F(t)\]
Where \(F(t)\) the cumulative distribution function of survival times
\[h(t) = -S'(t)/S(t)\]
Defined as the instantanious death rate, the probability of dying at time \(t\), given that you have survived this far.
Divide survival time in intervals j
Number at risk at beginning of interval \(n_j\) Number of events within the interval \(d_j\) Proportion surviving interval j \(p_j = \frac{n_j-d_j}{n_j}\) A sensored participant is included up until and including the last preceding event.
\[S(t_j) = \prod_j{p_j} = \prod_j{\frac{n_j-d_j}{n_j}}\]
Or differently
\[S(t_j) = P(T>t_j) = P(Event | t = t_{1-j})*P(T = t_{1-j}) = (1-P(t_{j-1} < t < t_j | T > t_{j-1}))*P(T>t_{j-1})\]
\[SE[S(t)] = S(t)\sqrt{\sum_{j=1}^k{\frac{d_j}{n_j(n_j-d_j)}}}\]
Where \(t_k \leq t < t_{k+1}\) all intervals up to \(T=t\)
\(H_0:\) expected number of events in group \(i\) at \(t_j\) is given by: \[e_{ij} = n_{ij}*(d_j/n_j)\]
So the number of persons in the group, multiplied with the overall number of events during \(t_j\)
Let \(e_i = \sum_j{e_{ij}}\) and \(d_i = \sum_j{d_{ij}}\) for \(i = 1,2\) the groups. Then
\[T = \sum_{i=1,2}{\frac{(d_i-e_i)^2}{e_i}}\]
Is approximately chi-square distritubed with 1 degree of freedom.
Step 1: Estimate effect size.
\[\delta_0 = \frac{ln(p_1)}{ln(p_0)} = HR\] Where \(p_1\) and \(p_0\) the expected survival probabilities for two groups after some relevent follow-up time \(T\)
Hazard Ratio is the instatanious risk of dying for group 1, compared to the risk of dying for group 2, at some time-point t.
Step 2: number of events Estimate number of events \[d = (\frac{1+\delta_0}{1-\delta_0})^2(Z_{\alpha} + Z_{\beta})^2\]
Step 3: estimate total number of patients \[n \geq \frac{2d}{2-p_0-p_1}\]
Assuming * constant hazard ratio * inclusion time * follow-up time (no loss-to-follow up)
\[R_{adj}^2 = 1-(1-R^2)(\frac{n-1}{n-k-1})\]
\[w = logit(\pi) = ln(\frac{\pi}{1-\pi}) = \beta_0 + \sum_{i}{\beta_iX_i}\]
Or equivalently:
\[\pi(X) = \frac{1}{1+e^{-(\beta_0+\sum_i{\beta_iX_i})}}\]
\[L(\pi) = \prod_{i=1}^n[\pi^{Y_i}(1-\pi)^{1-Y_i}]\]
\[l(\pi) = ln(L(\pi)) = \sum_{i = 1}^n{[y_i*ln(\pi) + (1-y_i)*ln(1-\pi)]}\]
\[T = -2(l_1-l_0)\]
T is chi-square distributed with the difference in number of parameters between the model as degrees of freedom.
\[T = \frac{\beta}{se(\beta)}\] Is approximately standard normally distributed, \(T^2\) is approximately chi-square distributed with 1 degree of freedom
Ratio bewteen the first and second derivatives of the likelihood at \(\beta_1 = 0\). Oppriximately chi-square distributed.
sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS Sierra 10.12.6
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] backports_1.1.0 magrittr_1.5 rprojroot_1.2 tools_3.3.2
[5] htmltools_0.3.6 yaml_2.1.16 Rcpp_0.12.14 stringi_1.1.6
[9] rmarkdown_1.8 knitr_1.18 git2r_0.20.0 stringr_1.2.0
[13] digest_0.6.13 evaluate_0.10.1
This R Markdown site was created with workflowr


${fbSession.currentWarning}
${fbSession.title}
New Folder:
Upload File(s):
Overwrite Existing files:
Location:
${fbSession.currentFolder}
UP (${fbSession.upFolder.name})
UP
Name (
sort
)
Size (
sort
)
Last Modif. (
sort
)
Actions
${file.name}
${getTimeStamp(file)}
${file.name}
${file.name}
${getFileSize(file)}
${getTimeStamp(file)}
Rename
;

Delete
New Name:
Delete '${file.name}' ?
Delete recursively:


1.1 Web Caching
Web caching refers to the act of
storing certain web resources (i.e.,
pages and other data files) for possible future reuse. For example,
Matilda is the first person in the office each morning, and she likes
to read the local newspaper online with her wake-up coffee. As she
visits the various sections, the Squid cache on their office network
stores the HTML pages and JPEG images. Harry comes in a short while
later and also reads the newspaper online. For him, the site loads
much faster because much of the content is served from Squid.
Additionally, Harry's browsing
doesn't waste the bandwidth of the
company's DSL line by transferring the exact same
data as when Matilda viewed the site.
A cache hit occurs each
time Squid
satisfies an HTTP request from its cache. The cache hit
ratio, or cache hit rate, is the
percentage of all requests satisfied as
hits. Web caches typically achieve hit ratios between 30% and 60%. A
similar metric, the byte hit ratio, represents
the volume of data (i.e., number of bytes) served from the cache.
A cache miss occurs when
Squid
can't satisfy a request from the cache. A miss can
happen for any number of reasons. Obviously, the first time Squid
receives a request for a particular resource, it is a cache miss.
Similarly, Squid may have purged the cached copy to make room for new
objects.
Another possibility is that the resource is uncachable. Origin
servers can instruct caches on how to treat the response. For
example, they can say that the data must never be cached, can be
reused only within a certain amount of time, and so on. Squid also
uses a few internal heuristics to determine what should, or should
not, be saved for future use.
Cache validation is a
process that ensures Squid
doesn't serve stale data to the user. Before reusing
a cached response, Squid often validates it with the origin server.
If the server indicates that Squid's copy is still
valid, the data is sent from Squid. Otherwise, Squid updates its
cached copy as it relays the response to the client. Squid generally
performs validation using
timestamps. The origin
server's response usually contains a
last-modified timestamp. Squid sends the
timestamp back to the origin server to find if the original resource
has changed.
For a detailed treatment of web caching, have a look at my book
Web Caching, also by
O'Reilly.
|


{% extends "layout.html" %} {% block title %} You should log in{% endblock %} {% block body %}
There are no more available rooms for the game you asked.


Pakistan's ex-polio focal person seeks apology from UK daily over polio cover-up allegations
by News DeskThe country’s former focal person on polio eradication, Babar bin Atta, denounced a report published by British daily The Guardian and sought an apology over accusations of “covering-up” a polio outbreak.
Babar termed the allegations “baseless”.
“I wish, I just wish, if Guardian had contacted me even once for my version. They even don’t know that Pakistan reported a bigger VDPV outbreak in 2016/17 in Balochistan. Noone blamed Ayesha Raza (PMLN) & today the same Attention seeking Ayesha Raza is quoted by The Guardian,” he said in a tweet on Friday.
The former focal person further said that he would be writing to the publication to “correct the record and issue an apology”, otherwise he would pursue legal options.
According to the report in the British publication, a dozen children have been infected with the P2 strain of the virus, which causes paralysis and primarily affects those under the age of five.
Special Assistant to the Prime Minister on Health Dr Zafar Mirza also denied any no cover-up and said the “Sabin-like type 2 derived virus outbreak in Pakistan is vigilantly being monitored & appropriately responded”.
He further added that the “situation is under control” and a “full-fledged” campaign will start on November 11.
The report added that the P2 strain had previously been eliminated from Pakistan. Polio exists in three different strands, P1, P2, and P3, with P2 notorious as the most contagious and most vicious in its impact on those infected.
Alongside Afghanistan, Pakistan is one of only two countries in the world that has not entirely eradicated polio.


CD Detail
Back to cd list
Added to cart.
Oops, only
In Stock.
CD Title
Musician:
Company:
Release Date:
Language:
Category:
:
songs
Reference:
Shipping Weight:
ounces
Price:
$
Qty:
In Stock
Only
In Stock
Unavailable
Add to Cart


Brief Biography of @bring—Old Wu, or 老吴 in Chinese: Old Wu, male, born on Dec. 12, 1968, Chinese nationality, is a good man and has been studying English for over 30 years. He teaches himself English, and he does not believe in fate, so he keeps working hard. Humorous and optimistic, he will become your good friend.
First, Mount Tai and Taian City
As a Chinese, you may not know Taian, but you definitely hear of Mount Tai. In the program “Chinese attractions that foreigners are interested in “, Mount Tai ranks the second place after the Great Wall.
Mount Tai is located in the territory of Taian City, Shandong province. It is the world’s natural and cultural heritage, one of the ten famous mountains of China, the World Geopark and the national 5A tourist attractions. The name of Taian derives from the sentence: “The country is prosperous and the people are at peace”. Taian is the National Health City, the National Historical and Cultural City, the National Forest City, and has won the Chinese Habitat Environment Award.
By the way, the earliest Taian Railway Station is probably built by the Germans and then renamed the Taishan Railway Station; and the station at Beijing-Shanghai high-speed railway is the Tai’an station. Therefore, the outsiders to Tai’an must pay attention: Taishan Station and Taian Station are not the same place.
Second, Taishan Pancake
Located in the North Temperate Zone, Taian has a semi-moist warm temperate continental monsoon climate, characterized by four distinct seasons, rain and heat in the same quarter, cold and dry over the same period, adequate lighting, and concentrated rainfall. Sunshine, air temperature and precipitation of Taian City are very beneficial to the growth and development of crops and forest fruits. The main crops are wheat, corn, vegetables, silkworms, fruit trees and peanuts.
Growing a lot of corn, Taian has its food product: Taishan Pancake. With a history of over 1000 years, Taishan Pancak, made of corn, millet as raw material, is a famous traditional food product which is thin, crisp, ans shelf-stable. Pancake roll stuffed with scallions is a local cuisine.
Growing up by eating pancakes, I have a special feeling for pancakes. In Shandong the people of Linyi also eat pancakes, but Linyi pancakes are not the same as Tai’an pancakes. It looks like that Linyi pancakes are thick and wet, and Tai’an pancakes are thin and dry.
When I was a child, my father and other miners went to visit West Lake, bring some pancakes with them, and the people of the South were unable to understand: “Look, they are eating paper.”
 I had studied for four years in a secondary school in Weifang City, where people don’t eat pancakes. When I came back home to eat pancakes, I found it a hard thing.
Third, Process of Making Pancakes
a. Select the corn particle to remove the bad; b. Put corn in a bucket with water, usually for a night; c. Use stone mill to grind the corn into thick paste; d. Put the corn paste for a period of time to let it ferment, the degree of fermentation can determine the sweet and sour of the pancake； e. Wipe the griddle with soybean, put the corn paste on the heated griddle with a spoon; f. Extend the paste evenly on the griddle, slowly making circles to make it mature; g. Lift the pancake h. Stack and store.
The freshly made pancakes are the best to eat, and if they are stuffed with salted fish or shallots or sauces, that’s more delicious.
The pancakes are delicious, but making them is the most laborious.
Burn up and smoked, the maker suffers a lot, even in summer. It would cost at least three or four hours to finish a big pot of corn paste. I still remember when I was a child, my mother got up to make pancakes with the first cockcrew. Mother did not have the breakfast until she finished the work.
Now, even the pancake production is also mechanized, which is no longer so tiring. However people always feel that the hand-made pancakes are good to eat. It is said that on the top of Mt.Tai, a piece of pancake, stuffed with a shallot and a bit of sauce, is worth several yuan, or is it 10 yuan, and I’m not sure.
The pancakes bear the culture, and the memories of my childhood, where I can find happiness and the sweat and hardship of my mother.
When I pick up the pancakes, as if I am chewing the hard times, I seem to see my mother keeping making pancakes in smoke, backhunched…
This page is synchronized from the post: Taishan Pancake


{%extends 'base.html'%} {%block content%}
Join Trash Talk
Enter your information below
Record Volunteer Hours (optional)
Cancel
{%endblock%}


BREAKING General Election: A&E waiting times hit worst level in 15 years
We'll be bringing you the very latest updates, pictures and video on this breaking news story
by Dave BurkeNew figures reveal A&E waiting times have hit their worst levels on record, just weeks before voters go to the polls.
NHS England data shows just 83.6% of patients were admitted or treated within four hours.
We'll be bringing you the very latest updates, pictures and video on this breaking news story.
Please check back regularly for updates on this developing story here at Mirror.co.uk .
For the latest news and breaking news visit Mirror.co.uk/news .
Get all the big headlines, pictures, analysis, opinion and video on the stories that matter to you.
Follow us on Twitter @DailyMirror - the official Daily Mirror & Mirror Online Twitter account - real news in real time.
We're also on Facebook /dailymirror - your must-see news, features, videos and pictures throughout the day from the Daily Mirror, Sunday Mirror and Mirror Online.