Perhaps I've not searched thoroughly enough for a setting or workaround, but it would be nice to be able to browse to dashboards on my RPi without having to log in.
Is there a way to bypass username and password logon for emoncms on RPi?
Submitted by Guest on Sun, 12/01/2014 - 18:24»
Re: Is there a way to bypass username and password logon for emoncms on RPi?
Are you saying you'd just like to be logged in by default (essentially just permanently bypassing the authentication on emoncms)?
If so, could you get rid of the session stuff in index.php (around line 52) and replace with:
$session['userid'] = ** Your userid **
$session['read'] = 1;
$session['write'] = 1;
$session['admin'] = 1;
$session['editmode'] = TRUE;
$session['lang'] = "en";
Re: Is there a way to bypass username and password logon for emoncms on RPi?
Jeff, the facility to view dashboards without logging in already exists without amending the code/modules.
See my comments in http://openenergymonitor.org/emon/node/3594
Paul
Re: Is there a way to bypass username and password logon for emoncms on RPi?
Hi Paul,
Normally, I would login at :
"<IP_ADDRESS>/emoncms" from chrome in Windows 7
user name: admin
If I add /admin at the end as your post suggests "<IP_ADDRESS>/emoncms/admin"
I get:
"Notice: Undefined variable: result in /var/www/emoncms/Modules/admin/admin_controller.php on line 59"
Regarding browsing to a dashboard without logging in, I am unable to browse to a (public) dashboard that has been bookmarked.
any help would be appreciated.
best,
Jeff
(Edited by mod - Paul)
Re: Is there a way to bypass username and password logon for emoncms on RPi?
Jeff, I think that you are getting this error because you are using the username 'admin', (the system thinks that you are trying to call the admin module?)
If you go to your 'account' menu, and change your username to something else, it should work.
A further enhancement is to use alias's to access your dashboards;
Go to <IP_ADDRESS>/emoncms/dashboard/list or select the 'list' icon from your dashboards menu, and add 'alias' names to each of your dashboards.
So for example to get the 'Weather' dashboard, you would go to <IP_ADDRESS>/emoncms/loginname/weather
Just ensure that your feeds are set to Public, and your dashboards are set similar to mine.
Paul
Re: Is there a way to bypass username and password logon for emoncms on RPi?
Hi Paul,
Got public dashboards to work!
Thanks for your help.
Jeff
Re: Is there a way to bypass username and password logon for emoncms on RPi?
I know this is a bit of an old topic, but aside from making the dashboards public, I was really just wanting to not have to bother logging in to emoncms on my own computer.
I've read mdbigg's comment, and went to the index.php file. I didn't see anything resembling what was mentioned around line 52, which was "set charset to utf8" A few lines further is "//3) User sessions", which is probably the "sessions stuff" mentioned, but what is the next step? Do you delete everything in the //3) User sessions and replace with just the lines mentioned? Because it seems the language is then supposed to be set in the //4) Language section. So would you then eliminate the //4) section, or just leave it as the language is already set in //3) or do you put the one $session[lang] line in //4)? Also, I didn't see any mention of editmode in the entire index.php file, so that is confusing too.
And it seems like since the program only asks for the username and password, couldn't that be inserted into the index.php file somewhere and just let the program go on as if it had been entered?
Thanks for any help.


@Beta
public class IsolateUtil
extends java.lang.Object
Formally, given the application of a relational operator op1 (=, !=, <, >, <=, >=) t1 + ... + t_m op1 r_1 + ... + r_n and given a `linear generalized variable` V, derive a newequivalent application: V op2 Term/Alpha Where V does not appear on Term and Alpha, which must be normalized polynomials. Alpha may be 0 if V does not appear originally or disappears. V may disappear from the expression, for example, x = x becomes 0 = 0, also known as `true`. This still fits `Alpha*V op2 Term`, with Alpha being 0, but now `Alpha` cannot move to the right-hand side as a denominator. In these cases, we simply return `0 op2 Term`. op2 is the same as op1, if op1 is an equality (i.e. `=` or `!=`), otherwise it can be a `flipped` op1 if you multiply or divide the inequality by a negative (or swap the left and right hand sides). In the cases where it cannot be determined if `Alpha` is 0 or not, for example: y * x = 10 the resulting expression needs to test for `Alpha` being 0. With respect to the previous example we would get: if y != 0 then x = 10/y else 0 = 10 or more generally: if Alpha != 0 then V = Term/Alpha else 0 = Term Similarly, if the sign of `Alpha` cannot be determined and `op1` is an inequality (i.e. `<`, `<=`, `>=`, or `>`) then the general result will be of the form: if Alpha != 0 then if Alpha > 0 V op1 Term/Alpha else VTerm/Alpha else 0 op1 Term
| Constructor and Description |
|---|
IsolateUtil() |
| Modifier and Type | Method and Description |
|---|---|
static void |
assertSupportedOperator(Expression operator) |
static Expression |
flipIfRequired(Expression operator,
Polynomial alpha) |
static Expression |
flipInequalityOperator(Expression operator) |
static boolean |
isEquality(Expression operator) |
static Expression |
isolate(Expression inequality,
Expression linearVariableToIsolate) |
static Expression |
isolate(Expression leftExpression,
Expression operator,
Expression rightExpression,
Expression linearVariableToIsolate) |
static Expression |
isolate(Polynomial leftPolynomial,
Expression operator,
Polynomial rightPolynomial,
Expression linearVariableToIsolate) |
static Pair<java.util.List<Monomial>,java.util.List<Monomial>> |
splitIntoSimilarAndDissimilarTerms(Polynomial polynomial,
Expression generalizedVariable)
Separates the monomials of a given polynomial into those containing a given generalized variable (similar terms),
and those not containing the given generalized variable (dissimilar terms).
|
public static Expression isolate(Expression inequality, Expression linearVariableToIsolate)
public static Expression isolate(Expression leftExpression, Expression operator, Expression rightExpression, Expression linearVariableToIsolate)
public static Expression isolate(Polynomial leftPolynomial, Expression operator, Polynomial rightPolynomial, Expression linearVariableToIsolate)
public static Pair<java.util.List<Monomial>,java.util.List<Monomial>> splitIntoSimilarAndDissimilarTerms(Polynomial polynomial, Expression generalizedVariable)
polynomial
- generalizedVariable
- public static void assertSupportedOperator(Expression operator)
public static Expression flipIfRequired(Expression operator, Polynomial alpha)
public static boolean isEquality(Expression operator)
public static Expression flipInequalityOperator(Expression operator)


My first project was to make this website about me. For the programming I'm using HTML and CSS.
For our second project we had to make a web site but more professional looking. I used the same programs for coding but we added a new one called Bootstrap.
For my third project, I learned how to make my first game wich is called Snake
For my fourth project I made my second game called Mario


Manuel Gerardo Delatour Quintero
Front-end Developer and Desktop Programmer
Currently I am a Developer of Desktop and Front-End Applications, I have learned everything in a self-taught way and I am studying Basic Secondary "School Padre Luis Variara"
I consider myself a person, respectful, friendly, trustworthy, perfectionist and solidarity. I love teaching my knowledge to others and I am always constantly learning
My objective one day is to found my own Software Technology Solutions company
Respectful
I consider myself a respectful person, since I always treat my colleagues well and I have very good manners.
Friendly
I consider myself a respectful person, since I always treat my colleagues well and I have very good manners.
Trustworthy
I consider myself a respectful person, since I always treat my colleagues well and I have very good manners.
Perfectionist
I consider myself a respectful person, since I always treat my colleagues well and I have very good manners.
Technologies that I use
Actualmente me encuentro trabajando de manera autónoma y estoy desarollando varios proyectos y emprendimientos
<EMAIL_ADDRESS>+57<PHONE_NUMBER>


WITNESS LIST International Relations, Trade & Technology Committee May 7, 1997-1:00P HB 1487 Registering, but not testifying: On: Alvarado, Betty Jean (IRS), San Antonio Gunn,David (TX. Dept. of Lic. & Reg), Austin HB 1556 Registering, but not testifying: On: Cowan, Andrea (Comptroller's Office), Austin Poole, Clyde (Tx. Dept. of Info. Res.), Austin HB 1761 For: Arnold, Cynthia (La Mujer Obrera), El Paso Hinojosa, Walter (TX. AFL-CIO), Austin Against: Carlson, David (Ocsten Staffing Service), El Paso Darling, Gail (Gail Darling Temp.), El Paso Ellis, Cissy (TX. Assoc. of Staffing), Austin Flores, Rene (TX. Assoc. of Staffing), El Paso Thornberry, Kate (TX. Assoc. of Staffing), Houston Registering, but not testifying: Against: Escobar,Bonnie (Greater El Paso Chamber), El Paso Grieg, Brian (TX Assoc. of Staffing), Austin Martinez, Cyndi (TDY Temporaries), Austin On: Hale, William (TX Comm. on Human Right), Austin HB 1805 Registering, but not testifying: On: Cowan, Andrea (Comptroller's Office), Austin HB 1880 Registering, but not testifying: On: Key, Alicia (Attorney General), Austin


British Military Slang Or Phrases You Need To Know 3
Warning: Contains adult language unsuitable for younger audiences ...
by Julian PerreiraWarning: This content contains adult themes and strong language which some might find offensive.
This is our third feature in a series for anyone struggling to understand the military language that their son, daughter, mother, father or an ex-military colleague is using - with help from our go-to guide/dictionary of all the words and phrases that you will ever need to know.
Our previous list of phrases were so popular, we thought we'd share some more.
What would you add to the next list?
1. 'Block Rat'
Someone who never seems to go home on leave, and enjoys their time locked in their room playing online games thoroughout the night, while drinking copious amounts of energy drinks.
2. 'Racing Spoon'
Personal eating utensil, used at every meal time, usually licked clean between meals and kept in top left side pocket.
3. 'God Squad'
Name for any member of the Army Chaplain Service, usually the unit’s Padre. Although he/she is a welcome addition on exercise or operations, as they will come armed with a large bag of Haribo sweets.
4. 'Pan Bashing'
The name given for the punishment whereby a person is sent to help in the field kitchen, cleaning pots and pans for the Master Chef.
5. 'Drop Shorts'
Nickname given to members of the Royal Artillery Regiment by other units, as it is sometimes claimed their rounds are more likely to drop short of where they are needed (enemy position). The Royal Artillery Regiment will have a retort to this.
6. 'In Clip'
Self-diagnosis for anything from a seriously bad hangover, to sore muscles after a tough PT session. E.g ‘I’m in clip’.
7. 'Gen'
What I am about to tell you is definitely not a lie.
8. 'Eyebrows Gen'
What I am about to tell you is definitely NOT a lie.
9. 'Argos Gen'
What I am about to tell you is NOT a lie. However, if what I say turns out to be a lie, on pay day, I must go to the nearest Argos shop, type in a random set of digits, and whatever item appears on the screen, I must buy it. No matter how expensive it is.
10. 'Robo Gen'
What I am about to tell you is definitely NOT a lie. However, if what I say turns out to be bo****ks, then you can shave my head bald like in the movie RoboCop.
11. 'Slack'
The name given to a person whose soldiering abilities have been called into question, usually because a lack of self-discipline.
12.
13. 'Sucking Air From China'
Term to describe someone who is ‘hanging out’ during a PT session. Usually because they are trying to breathe in as much oxygen as they physically can.
14. 'Squared Away'
Something is sorted. Term used to refer to when military personnel would fold their bed blankets and sheets neatly into a square shape prior to room inspections.
15. 'The Badge'
The name sometimes given to the senior soldier within the regiment – The Regimental Sergeant Major, called 'The Badge' … this is due to the ridiculously large badge of rank that they wear.
16.
17. 'You’re In Your Own Time Now'
Hurry up and finish this task, I want to go to the Mess for a beer … you are most definitely not in your own time.
18. 'Donkey Wolloper'
Nickname given to soldiers within the British Household Cavalry regiment, or the Royal Horse Artillery.
19. 'No Duff'
An incident that is not a drill or training exercise. A real emergency.
20. 'Wriggly Tin'
Corrugated metal sheets used during construction of defences or when digging trenches.
21. 'Bulling'
Military word used to describe polishing equipment, using a cloth in circular motion. The term 'bulling' is typically used when shining boots.
22. 'Glasshouse'
The name of the British military prison in Colchester. The name 'Glasshouse' originated from the former military prison in Aldershot, which had a glazed roof. Often used to mean any military prison.
23. 'Crow Canon'
The name of the Lightweight Support Weapon (LSW) usually given to the newest member (CROW) of the regiment, because nobody else wants to carry it.
24.  Usually before dawn or just after sunset.
25. 'D & V'
Diarrhoea and vomiting stomach bug, usually contracted when deployed on operations and living in less than sanitary conditions.
If you found this list helpful, why not share this and spread the word?
What would have made it onto your list?
* Cover image: LSwanPhotography.


ABRT\-ACTION\-SAVE\-
Section: ABRT Manual (5)Updated: 06/22/2012
Index Return to Main Contents
NAME
abrt-action-save-package-data.conf - Configuration file for abrt-action-save-package-dataDESCRIPTION
The abrt-action-save-package-data tool queries RPM database and saves package and component name to problem data.
The configuration file consists of items in the format "Option = Value". A description of each item follows:
OpenGPGCheck = yes | no
- When set to yes, abrt will report crashes only in executables which belong to an installed package with valid GPG signature. The default is no. The files containing trusted GPG keys which are used to verify the signatures are listed in the gpg_keys configuration file.
BlackList = packageName, packageName ...
-
abrt will ignore packages in this list and will not handle their crashes.
BlackListedPaths = /path/to/ignore/*, */another/ignored/path* ...
-
abrt will ignore crashes in executables whose absolute path matches one of specified patterns.
ProcessUnpackaged = yes | no
- When set to yes, abrt will catch all crashes in the system. When set to no, it will catch crashes only in executables which belong to an installed package. The default is no.
SEE ALSO
abrt.conf(5)
AUTHORS
- * ABRT team
Index
This document was created by man2html, using the manual pages.
Time: 05:33:59 GMT, December 24, 2015


"Architect, what does X table/column do in the DB?"
"Developer, what is the datatype of X column?"
"DBA, can we add X to the DB?"
Problem
We often can't answer those questions without hopping into the database with our laptop which may not be with us at that moment.
What if we could answer those questions
using any device,wherever we want(like in meetings or during development),with mininal input?
DOUMI
Korean word for "Helper"
(Thank you, Seung!)
DOUMI Benefits
Fuzzy search for DB objects
Displays table structures
Pulls sample data
Provides DB Object Documentation
Runs on laptops and smartphones
Mockups
Architecture
Ionic - Hybrid framework that allows us to target desktop, android, and iOS!"Use any device"
Angular - "Minimize input"using components
API Abstraction - "Use any where"over HTTPS
Methodology
Design and Planning
Install and Learn Tools
Ionic, angular, Node, npm, Ruby, Sinatra, Sequel
Build MVP Style
How far did we get? What's Next?
Nearly Complete - Fuzzy search for DB objects
Complete - Displays table structures
Partial Completion - Pulls sample data
Partial Completion - Provides DB Object Documentation
Partial Completion - Runs on laptops and smartphones


[1] POI with Shortest distance it from their web software .Now my device you wish to retain this system you what its supposed to Toronto during Black Friday 250.oo.
[2] web software .Now my device you wish to Kimberly ,WI the FM Transmitter for extra 65 miles the 'poi' database maybe they will contnue support ,buy thier maps - i updated the speech recognition feature lacks laccurate local info..


{% trans "Test Sending in Progress" %}
{% blocktrans %}Please do not leave the page while the test mail is being sent. This process might take several minutes.{% endblocktrans %}
{% extends "admin/change_form.html" %} {% load i18n admin_modify %} {% block extrahead %}{{ block.super }} {% endblock %} {% block bodyclass %}change-form{% endblock %} {% block breadcrumbs %}{% endblock %} {% block content %}
{% blocktrans %}Please do not leave the page while the test mail is being sent. This process might take several minutes.{% endblocktrans %}


This documentation is intended to help you properly install and configure the template to your needs. If you have any questions, please contact me via email or leave a comment on my profile.
Before you start
Links That May Be Useful:
http://docs.ghost.org/
https://en.ghost.org/forum/
Instalation.
- Open to your FTP client (or explore your ghost directory if you are hosted ghost locally).
- Navigate to
ghost/content/themes
. - Paste main phantom folder.
- Restart ghost and log in to your dashboard.
- In settings under
themes
selectphantom
from fropdown menu. * Clicksave
. That's all, now time to configure your theme.
Configuration.
For a full functionality of this template, you have to go the setup process. It will not take too long.
General functionality
- Go to your dashboard and open settings page.
- There fill inputs like blog name, description etc.
- On
blog logo
you can add your logo, to be showed in footer and single post page. - On
blog cover
upload your image to be showed as your homepage cover photo. I used one of 4 covers included to this template.
Remember! If you wannt your homepage to look pretty, make sure that your cover image has no less than 1920px width and 700px height.
- Into user section, fill areas and add your avatar. That info's and your avatar will be showed in a footer of every single post u write.
That's all from the front-end side. Now you only need to write some javascript's. (just kidding ;))
Features functionality
There are some features like second logo addicted to cover image, like you see phantom logo looks well in footer, but imagine how it will look on a cover image.
Featured Images
If you wannt to add featured background to single post, just write this code:
![cover]
. Now, on preview side of your post will you see special place to with caption: "Add image for cover". Click on that or just drag image to this place from your local files.
Remember!
Remember! Always put ![cover]
after first paragraph, this first paragraph is automatically used on homepage as post excerpt. Quick Example from this post:
This documentation is intended to help you properly install and configure the template to your needs. If you have any questions, please contact me via email or leave a comment on my profile.
![cover]
You don't need to always add that image, if you do not do this, phantom will fill empty space with nice color ;)
Video
Just put your iframe code everywhere u wannt. Easy!
Footer social links
As you see in footer, there are 3 random buttons. If you wanntt to add social media to your theme, do it in order:
* Open phantom/partials/footer.hbs
* Overwrite #
with links to appropriate url's i.e if you wann't a link to your dribbble account, it should look like this:

There are many link you can add, twitter, facebook, linkedin, everything you need is correct icon. Full list of icons included to this template you can find here.
Wann't facebook button? Let me explain in a example:

as you see i overwrited hashtag #
and added special facebook icon icon-facebook
from included icons list.
That's all. Your Theme is installed and configured. It's not? Something is not working properly? Feel free to ask me anything via e-mail<EMAIL_ADDRESS>or write a comment in phantom item page on themeforest.
Credits.
http://www.unsplash.com/ - image resources.
http://www.ianlunn.co.uk/plugins/jquery-parallax/ - jQuery Parallax
http://fortawesome.github.io/Font-Awesome/ - Font Awesome Icons


scikits.learn.grid_search.GridSearchCV¶
- class scikits.learn.grid_search.GridSearchCV(estimator, param_grid, loss_func=None, score_func=None, fit_params={}, n_jobs=1, iid=True, refit=True, cv=None, verbose=0)¶
Grid search on the parameters of a classifier
Important members are fit, predict.
GridSearchCV implements a “fit” method and a “predict” method like any classifier except that the parameters of the classifier used to predict is optimized by cross-validation
Parameters : estimator: object type that implements the “fit” and “predict” methods :
A object of that type is instanciated for each grid point
param_grid: dict :
a dictionary of parameters that are used the generate the grid
loss_func: callable, optional :
function that takes 2 arguments and compares them in order to evaluate the performance of prediciton (small is good) if None is passed, the score of the estimator is maximized
score_func: callable, optional :
function that takes 2 arguments and compares them in order to evaluate the performance of prediciton (big is good) if None is passed, the score of the estimator is maximized
fit_params : dict, optional
parameters to pass to the fit method
n_jobs: int, optional :
number of jobs to run in parallel (default 1)
iid: boolean, optional :
If True, the data is assumed to be identically distributed across the folds, and the loss minimized is the total loss per sample, and not the mean loss across the folds.
cv : crossvalidation generator
see scikits.learn.cross_val module
refit: boolean :
refit the best estimator with the entire dataset
verbose: integer :
Controls the verbosity: the higher, the more messages.
Notes
The parameters selected are those that maximize the score of the left out data, unless an explicit score_func is passed in which case it is used instead. If a loss function loss_func is passed, it overrides the score functions and is minimized.
Examples
>>> from scikits.learn import svm, grid_search, datasets >>> iris = datasets.load_iris() >>> parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]} >>> svr = svm.SVR() >>> clf = grid_search.GridSearchCV(svr, parameters) >>> clf.fit(iris.data, iris.target) GridSearchCV(n_jobs=1, verbose=0, fit_params={}, loss_func=None, refit=True, cv=None, iid=True, estimator=SVR(kernel='rbf', C=1.0, probability=False, ... ...
Methods
- __init__(estimator, param_grid, loss_func=None, score_func=None, fit_params={}, n_jobs=1, iid=True, refit=True, cv=None, verbose=0)¶
- fit(X, y=None, **params)¶
Run fit with all sets of parameters
Returns the best classifier
Parameters : X: array, [n_samples, n_features] :
Training vector, where n_samples in the number of samples and n_features is the number of features.
y: array, [n_samples] or None :
Target vector relative to X, None for unsupervised problems


This week, we learned all about Arduino. It is microcontroller platform that is easy to use for electronics projects.
It allows us to connect electronics through its pins so we can control things.
We finished every single project on the Sparkfun Inventors Kit Guide book, it took quite long but it is interesting to see how Arduino works.
We did projects like blink LED, photoresistor, buzzer, the game simon says, distance sensor, LCD display screen, motor and even an autonomous robot at last
Then, I processed to play around with the code myself, like changing it and using different variables.
So here is one of the changed code:
#include
LiquidCrystal lcd(13, 12, 11, 10, 9, 8);
void setup() {
lcd.begin(10, 4);
lcd.clear(); }
void loop() {
lcd.setCursor(0,0);
lcd.print("I FEEL ALIVE!!!");
lcd.setCursor(0,3);
lcd.print(millis()/1500); }


Specialties: Node.js, JavaScript, React.js, MongoDB, MySQL, Firebase, HTML, CSS, Bootstrap, APIs, JSON, REST, AJAX.
About MeI have a passion for running, exploring new technology and coding - let's connect & talk about technology!
Contact Me!Searching for a Full Stack Web Developer? That's great! Give me a call or send me an email and I will get back to you as soon as possible!


"Kristofor is a great person who works hard and stuff!" - Coworker 1
"He works hard, and plays harder. Definitely a must-have on any team" - Coworker 2
URL linking to github project would go here


PERLFORK
Section: Perl Programmers Reference Guide (1)Updated: 2009-02-12
Index Return to Main Contents
NAME
perlfork - Perl's fork() emulationSYNOPSIS
NOTE: As of the 5.8.0 release, fork() emulation has considerably matured. However, there are still a few known bugs and differences from real fork() that might affect you. See the "BUGS" and "CAVEATS AND LIMITATIONS" sections below.
Perl provides a fork() keyword that corresponds to the Unix system call of the same name. On most Unix-like platforms where the fork() system call is available, Perl's fork() simply calls it.
On some platforms such as Windows where the fork() system call is not available, Perl can be built to emulate fork() at the interpreter level. While the emulation is designed to be as compatible as possible with the real fork() at the level of the Perl program, there are certain important differences that stem from the fact that all the pseudo child ``processes'' created this way live in the same real process as far as the operating system is concerned.
This document provides a general overview of the capabilities and limitations of the fork() emulation. Note that the issues discussed here are not applicable to platforms where a real fork() is available and Perl has been configured to use it.
DESCRIPTION
The fork() emulation is implemented at the level of the Perl interpreter. What this means in general is that running fork() will actually clone the running interpreter and all its state, and run the cloned interpreter in a separate thread, beginning execution in the new thread just after the point where the fork() was called in the parent. We will refer to the thread that implements this child ``process'' as the pseudo-process.To the Perl program that called fork(), all this is designed to be transparent. The parent returns from the fork() with a pseudo-process ID that can be subsequently used in any process manipulation functions; the child returns from the fork() with a value of 0 to signify that it is the child pseudo-process.
Behavior of other Perl features in forked pseudo-processes
Most Perl features behave in a natural way within pseudo-processes.- $$ or $PROCESS_ID
- This special variable is correctly set to the pseudo-process ID. It can be used to identify pseudo-processes within a particular session. Note that this value is subject to recycling if any pseudo-processes are launched after others have been wait()-ed on.
- %ENV
- Each pseudo-process maintains its own virtual environment. Modifications to %ENV affect the virtual environment, and are only visible within that pseudo-process, and in any processes (or pseudo-processes) launched from it.
- chdir() and all other builtins that accept filenames
- Each pseudo-process maintains its own virtual idea of the current directory. Modifications to the current directory using chdir() are only visible within that pseudo-process, and in any processes (or pseudo-processes) launched from it. All file and directory accesses from the pseudo-process will correctly map the virtual working directory to the real working directory appropriately.
- wait() and waitpid()
- wait() and waitpid() can be passed a pseudo-process ID returned by fork(). These calls will properly wait for the termination of the pseudo-process and return its status.
- kill()
- kill() can be used to terminate a pseudo-process by passing it the ID returned by fork(). This should not be used except under dire circumstances, because the operating system may not guarantee integrity of the process resources when a running thread is terminated. Note that using kill() on a pseudo-process() may typically cause memory leaks, because the thread that implements the pseudo-process does not get a chance to clean up its resources.
- exec()
-
Calling exec() within a pseudo-process actually spawns the requested
executable in a separate process and waits for it to complete before
exiting with the same exit status as that process. This means that the
process ID reported within the running executable will be different from
what the earlier Perl fork() might have returned. Similarly, any process
manipulation functions applied to the ID returned by fork() will affect the
waiting pseudo-process that called exec(), not the real process it is
waiting for after the exec().
When exec() is called inside a pseudo-process then DESTROY methods and END blocks will still be called after the external process returns.
- exit()
- exit() always exits just the executing pseudo-process, after automatically wait()-ing for any outstanding child pseudo-processes. Note that this means that the process as a whole will not exit unless all running pseudo-processes have exited. See below for some limitations with open filehandles.
- Open handles to files, directories and network sockets
- All open handles are dup()-ed in pseudo-processes, so that closing any handles in one process does not affect the others. See below for some limitations.
Resource limits
In the eyes of the operating system, pseudo-processes created via the fork() emulation are simply threads in the same process. This means that any process-level limits imposed by the operating system apply to all pseudo-processes taken together. This includes any limits imposed by the operating system on the number of open file, directory and socket handles, limits on disk space usage, limits on memory size, limits on CPU utilization etc.Killing the parent process
If the parent process is killed (either using Perl's kill() builtin, or using some external means) all the pseudo-processes are killed as well, and the whole process exits.Lifetime of the parent process and pseudo-processes
During the normal course of events, the parent process and every pseudo-process started by it will wait for their respective pseudo-children to complete before they exit. This means that the parent and every pseudo-child created by it that is also a pseudo-parent will only exit after their pseudo-children have exited.A way to mark a pseudo-processes as running detached from their parent (so that the parent would not have to wait() for them if it doesn't want to) will be provided in future.
CAVEATS AND LIMITATIONS
- BEGIN blocks
-
The fork() emulation will not work entirely correctly when called from
within a BEGIN block. The forked copy will run the contents of the
BEGIN block, but will not continue parsing the source stream after the
BEGIN block. For example, consider the following code:
BEGIN { fork and exit; # fork child and exit the parent print "inner\n"; } print "outer\n";
This will print:
inner
rather than the expected:
inner outer
This limitation arises from fundamental technical difficulties in cloning and restarting the stacks used by the Perl parser in the middle of a parse.
- Open filehandles
-
Any filehandles open at the time of the fork() will be dup()-ed. Thus,
the files can be closed independently in the parent and child, but beware
that the dup()-ed handles will still share the same seek pointer. Changing
the seek position in the parent will change it in the child and vice-versa.
One can avoid this by opening files that need distinct seek pointers
separately in the child.
On some operating systems, notably Solaris and Unixware, calling "exit()" from a child process will flush and close open filehandles in the parent, thereby corrupting the filehandles. On these systems, calling "_exit()" is suggested instead. "_exit()" is available in Perl through the "POSIX" module. Please consult your systems manpages for more information on this.
- Forking pipe open() not yet implemented
-
The "open(FOO, "|-")" and "open(BAR, "-|")" constructs are not yet
implemented. This limitation can be easily worked around in new code
by creating a pipe explicitly. The following example shows how to
write to a forked child:
# simulate open(FOO, "|-") sub pipe_to_fork ($) { my $parent = shift; pipe my $child, $parent or die; my $pid = fork(); die "fork() failed: $!" unless defined $pid; if ($pid) { close $child; } else { close $parent; open(STDIN, "<&=" . fileno($child)) or die; } $pid; } if (pipe_to_fork('FOO')) { # parent print FOO "pipe_to_fork\n"; close FOO; } else { # child while (<STDIN>) { print; } exit(0); }
And this one reads from the child:
# simulate open(FOO, "-|") sub pipe_from_fork ($) { my $parent = shift; pipe $parent, my $child or die; my $pid = fork(); die "fork() failed: $!" unless defined $pid; if ($pid) { close $child; } else { close $parent; open(STDOUT, ">&=" . fileno($child)) or die; } $pid; } if (pipe_from_fork('BAR')) { # parent while (<BAR>) { print; } close BAR; } else { # child print "pipe_from_fork\n"; exit(0); }
Forking pipe open() constructs will be supported in future.
- Global state maintained by XSUBs
- External subroutines (XSUBs) that maintain their own global state may not work correctly. Such XSUBs will either need to maintain locks to protect simultaneous access to global data from different pseudo-processes, or maintain all their state on the Perl symbol table, which is copied naturally when fork() is called. A callback mechanism that provides extensions an opportunity to clone their state will be provided in the near future.
- Interpreter embedded in larger application
- The fork() emulation may not behave as expected when it is executed in an application which embeds a Perl interpreter and calls Perl APIs that can evaluate bits of Perl code. This stems from the fact that the emulation only has knowledge about the Perl interpreter's own data structures and knows nothing about the containing application's state. For example, any state carried on the application's own call stack is out of reach.
- Thread-safety of extensions
- Since the fork() emulation runs code in multiple threads, extensions calling into non-thread-safe libraries may not work reliably when calling fork(). As Perl's threading support gradually becomes more widely adopted even on platforms with a native fork(), such extensions are expected to be fixed for thread-safety.
BUGS
- *
- Having pseudo-process IDs be negative integers breaks down for the integer "-1" because the wait() and waitpid() functions treat this number as being special. The tacit assumption in the current implementation is that the system never allocates a thread ID of 1 for user threads. A better representation for pseudo-process IDs will be implemented in future.
- *
- In certain cases, the OS-level handles created by the pipe(), socket(), and accept() operators are apparently not duplicated accurately in pseudo-processes. This only happens in some situations, but where it does happen, it may result in deadlocks between the read and write ends of pipe handles, or inability to send or receive data across socket handles.
- *
- This document may be incomplete in some respects.
AUTHOR
Support for concurrent interpreters and the fork() emulation was implemented by ActiveState, with funding from Microsoft Corporation.This document is authored and maintained by Gurusamy Sarathy<EMAIL_ADDRESS>SEE ALSO
``fork'' in perlfunc, perlipc
Index
This document was created by man2html, using the manual pages.
Time: 05:29:08 GMT, December 24, 2015


Ionosphere inference
MASS - Similar motifs
Ionosphere inference is a extension of Ionosphere specifically for the purpose of making use of MASS (Mueen’s Algorithm for Similarity Search) for searching time series sub-sequences under z-normalized Euclidean distance for similarity. In simple terms it finds similar patterns.
Note that the terms motif, shape, shapelet and subsuqeunces are somewhat synonymous in this regard. HOWEVER their use in relation to Skyline has distinctions made:
shapelet,
subsequence
or shape: is a snippet of a timeseriesmotif
: is shapelet that has be discovered based on distance.matched_motif
: is a motif identified as matching some additional criteria (other than distance). A matched_motif can represent asimilar
,exact
ornot_similar
motif.
The initially Ionosphere implemented similarity search solely with the use of feature profiles created with tsfresh, which works very well, however introducing the use of MASS makes Ionosphere work even better. MASS compliments the original SITTCA algorithm, in terms of MASS is very good at shorter pattern similarity search and SITTCA is very good at long pattern similarity search.
It is important to know that MASS only finds similar shapes (values are not taken into account). So a metric that was running steadily at 10 and then jumps to 100 and runs steadily at 100, will have the same shape as a metric that was running at 10000 and then jumps to 100000. However MASS makes it possible to discover similar motifs very efficiently, these discovered motifs can then be validated based not only on their distances but also on their similar range and similarity in their areas under the curve as well. These multiple conditional verifications ensure that the method being used attempts to achieve 0 false negatives.
It is important to state that MASS is being used with very strict default allowed maximum distances and areas so that the outcomes are correct for purpose. The purpose being to find very similar motifs in trained data.
A note on batch_size vs. time series length, even if a batch_size is less than
the length of the time series Skyline only runs a batch_size
if the time series at least 10% larger than the batch_size. Although it is
generally possible to run a batch_size of very similar size on a time series,
errors occur when a time series that has some missing data points is used.
In testing a match was never found on data with a batch_size of 1440 and
settings.FULL_DURATION
time series.
Consider the system flags a potentially anomaly and using the subsequence of the last 180 data points of the potentially anomalous time series to create a shapelet.
Using mass-ts the system then searches all the available trained data for the metric to find similar motifs within the trained data. It then filters out similar motifs that are not similar enough based on:
their distance scores
the similar motif not being within the same values range as potentially anomalous motif.
the areas under the curve not being similar enough
If there are similar motifs remaining after the filtering, this is deemed as a match and the instance is considered normal and not anomalous. These examples just show the last 10 data points, in practice the minimum batch size should not be less than 180 data points so as to represent a large enough sample of the data to exclude repetitive volatility shifts.
2 matches are visualised below:
To demonstrate what inference is filtering out, the following are similar
motifs that are filtered out as they are a similar shape, but they are not
similar enough in terms of the values range (based on a
settings.IONOSPHERE_INFERENCE_MOTIFS_RANGE_PADDING
of 10)
The method is based on the best match found wins, not the first match found wins. The first matched found wins method is more efficient in terms of potential compute time, but more importantly the method wants to ensure 0 false negatives.
With Ionosphere inference enabled the analysis pipeline is as follows, best match found wins:
Ionosphere finding similar matching motifs for existing
SECOND_ORDER_RESOLUTION_SECONDS
features profiles time series andsettings.FULL_DURATION
features profilesIonosphere comparison to Mirage
SECOND_ORDER_RESOLUTION_SECONDS
features profiles and minmax scaled features profilesIonosphere echo comparison to Mirage
settings.FULL_DURATION
features profilesIonosphere comparison to Mirage
settings.FULL_DURATION
minmax scaled features profilesIonosphere layers comparisons
Testing
It is possible to run inference in a testing mode, where all analysis is done and results recorded, but the results do not have any bearing on the normal anomaly detection and alerting, meaning that if inference finds a similar motif that matches, it does not feedback as not anomalous, it just records the results for evaluation and validation, but do note that results will be stored in the database as if it were live.


FIFA Chief Urges Amakhosi To Start Women’s Team
by Soccer LadumaAt the CAF Women’s Football Strategy Taskforce Workshop in Cairo, FIFA’s Chief Officer for Women’s Football has urged Kaizer Chiefs to form a female team at Naturena.
Read: Major Blow For Chiefs Clash
Chiefs’ Marketing Director, Jessica Motaung, spoke at the workshop as women from across the continent discussed plans to take the women’s game forward in Africa, and Sarai Bareman was full of praise for Motaung, the Amakhosi and what the club can do for the cause.
The Glamour Boys are the most successful team in South African football in terms of their trophy count over the last 50 years, but as yet there are no ‘Glamour Girls’ for their female fans to cheer on – though that could potentially change in the future.
As quoted by on their official website, Bareman said, “We have experts from all around Africa, like Jessica, whose expertise in marketing, commercial and knowledge of the game is useful, and we have used this knowledge to build a robust strategy.
“The great thing about clubs like Chiefs is that they are well established and have good governance structures in place and have big fan bases. A club such as Chiefs can use the resources they have to start establishing women’s team.
Watch: Bafana See Stars In Ghana
“Using the same resources, fan base, and good governance it means there is an investment from the top down and there is a ripple effect. When people see big name clubs like Kaizer Chiefs having a women’s team, they see it is as important and fans also then support the team. This certainly helps and creates a good image for Women’s football.”
Download The Soccer Laduma Social Media App & Earn Airtime for Selfies!


📝 Note Taker
Add Note
My Notes
Add Note
Title
Content
Add Note
Note Added
×
Your notes have been updated
Note
Title
Content
Update Note
Note Updated
×
Your notes have been updated


Graph for Multiple Storms in a Single Location
Event Type 1:
Event Type 2:
Event Type 3:
Event Type 4:
Event Type 5:
Event Type 6:
Event Type 7:
Event Type 8:
Event Type 9:
Event Type 10:
State:
County/Zone:
Start Year:
End Year:
Submit
State
County
Begin Time
End Time


[1]
i think management could invest in the future.
[2]
i think management could invest in some staff having language issues.


| From |
{{txn.from}} |
{{txn.from | knownMiners: 0}} |
| To |
Contract Executiontransform{{txn.to}} |
{{txn.to | knownMiners: 0}} |
| Amount |
{{txn.value | fromWei}} UBQ |
${{txn.value | fromWei | toUSD}} |
{{txn.value | fromWei | toBTC}} BTC |
| Block Hash |
{{txn.blockHash}} |
| Received |
{{blockInfo.timestamp | relativetime }}
|
{{blockInfo.timestamp+"000" | date:'yyyy-MM-dd HH:mm:ss'}} |
| Block |
{{txn.blockNumber}} |
| Gas |
{{txn.gas}} |
| Gas Price |
{{txn.gasPrice}} |
| Previous Transactions |
{{txn.nonce}} |
| Transaction Cost |
{{txn.txprice | fromWei}} |
| Data |
{{txn.input}} |


|
|
def | __init__ |
| |
|
def | result |
| |
|
def | set_result |
| |
|
def | clear_result |
| |
|
def | has_result |
| |
|
def | chosen_task_name |
| |
|
def | set_chosen_task_name |
| |
|
def | clear_chosen_task_name |
| |
|
def | has_chosen_task_name |
| |
|
def | MergeFrom |
| |
|
def | Equals |
| |
|
def | IsInitialized |
| |
|
def | ByteSize |
| |
|
def | ByteSizePartial |
| |
|
def | Clear |
| |
|
def | OutputUnchecked |
| |
|
def | OutputPartial |
| |
|
def | TryMerge |
| |
|
def | __str__ |
| |
|
|
| has_result_ |
| |
|
| result_ |
| |
|
| has_chosen_task_name_ |
| |
|
| chosen_task_name_ |
| |
|
|
int | has_result_ = 0 |
| |
|
int | result_ = 0 |
| |
|
int | has_chosen_task_name_ = 0 |
| |
|
string | chosen_task_name_ = "" |
| |
The documentation for this class was generated from the following file:
- code/googleappengine-read-only/python/google/appengine/api/taskqueue/taskqueue_service_pb.py


{% verbatim %}
Your session will expire in
60
seconds, would you like to continue?
×
Set how many days of data should be retained.
Please enter an integer
that is not negative
that is lower than 9999
.
For facts collected older than the time period specified, save one fact scan (snapshot) per time window (frequency). For example, facts older than 30 days are purged, while one weekly fact scan is kept.
CAUTION: Setting both numerical variables to "0" will delete all facts.
Select a time period after which to remove old facts
Please enter an integer
that is not negative
that is lower than 9999
.
Select a frequency for snapshot retention
Please enter an integer
that is not negative
that is lower than 9999
.
working...
{% endverbatim %}


Let's say, by some miraculous turn of events, that you have a problem which can be modelled as a graph! And let's say this problem involves finding the shortest distance between two vertices on this graph. Then a good place to start would be with a Breadth-First Search, or BFS.
BFS is one of the simplest graph algorithms, but understanding it is important since it is a conceptual stepping-stone to more intricate ones.
The idea behind BFS is that it starts at some initial vertex in the graph and works its way out from this vertex to all others, level by level. That is, all vertices of distance d from the start vertex will get visited before any vertices of distance d + 1. By working its way out level by level, BFS is guaranteed to find, for every vertex, the shortest possible path from the starting point. Another way to think about this level-by-level traversal of the graph is that it expands the width of the search as far as possible before expanding the depth of the search. This is where the name comes from!
The basic pseudocode for this algorithm is as follows:
1 bfs( (V, E), s ):
2 for each vertex v in V:
3 mark v as not visited
4 end for
5 mark s as visited
6 add s to queue
7 while queue is not empty:
8 remove a vertex v from queue
9 for each vertex u adjacent to v:
10 if u is not visited:
11 mark u as visited
12 add u to queue
13 end if
14 end for
15 end while
Lines 2 to 4 serve only to set up the starting conditions for the search; before the search begins, each vertex must not be considered as visited, until it actually has been visited.
Lines 5 and 6 add the starting vertex to the queue and mark it as visited. Vertices are marked as visited when they are added to the queue. This is done so that the algorithm does not add a vertex to the queue more than once.
The interesting part happens on line 7 and afterwards. The loop will iterate until queue is empty. At the beginning, the queue has only the starting vertex, s. So on line 8, s gets removed from the queue and assigned to v. Now the loop on line 9 iterates through each vertex that is connected to v (i.e. s, at this point). Each of these vertices is added to the queue and marked as visited. When the loop has finished, or terminated, all of the vertices adjacent to s have been added to the queue, and line 7 gets executed again. This time, on line 8, the first vertex that was adjacent to s gets removed and the process gets repeated.
What is crucial to the understanding of how the algorithm works is an understanding of how queues work. After the first iteration of the while loop, when the starting vertex was processed, all of the vertices adjacent to the starting vertex are in the queue. This means that all vertices added to the queue later will be processed last. This is the fundamental property that makes the BFS work. Because of this property, we can see that each 'level' of the graph gets traversed before the next 'level'.
The above pseudocode gives the structure for BFS, but notice that it does not actually do anything. Each vertex is traversed in the order required by the BFS but nothing is done when the vertex is visited. So! Let's remedy this.
1 bfs( (V, E), s ):
2 for each vertex v in V:
3 mark v as not visited
4 distance[v] = 0
5 end for
6 mark s as visited
7 add s to queue
8 while queue is not empty:
9 remove a vertex v from queue
10 for each vertex u adjacent to v:
11 if u is not visited:
12 distance[u] = distance[v] + 1
13 mark u as visited
14 add u to queue
15 end if
16 end for
17 end while
The differences are on lines 4, 12, and 18. The purpose of these additions is to calculate the distance of each vertex from s. The important bit happens on line 12. This does what we want it to because of the properties that we established above; namely, the property that each level of the graph gets traversed before the next. Since each level has the same distance from s, this means that all the nodes that are of distance d from s are processed before any nodes of distance d + 1. So when line 12 is reached for some vertex u, we know two things: that u has not yet been added to the queue, and that u is farther away from s than v. So when distance[u] is assigned the value of distance[v] + 1, we know that this is the shortest possible distance from s to v!
But what if this isn't enough for us? We don't want to know only the distance from s, we want to know how to get from s to the vertex! Another small change to the algorithm will let us do this as well.
1 bfs( (V, E), s ):
2 for each vertex v in V:
3 mark v as not visited
4 distance[v] = infinity
5 predecessor[v] = null
6 end for
7 mark s as visited
8 add s to queue
9 distance[s] = 0
10 while queue is not empty:
11 remove a vertex v from queue
12 for each vertex u adjacent to v:
13 if u is not visited:
14 distance[u] = distance[v] + 1
15 predecessor[u] = v
16 mark u as visited
17 add u to queue
18 end if
19 end for
20 end while
The differences here are on lines 5 and 14. Line 14 assigns v to be the predecessor of u. The predecessor of some vertex u is the vertex that we would come from to get to u from s. It may not be obvious why this helps us find paths in the graph yet. To find the paths, remember that since the predecessor is the vertex we came from to get to u, we must have gotten to the predecessor somehow, so it has its very own predecessor too! Then we can travel backwards from some vertex u to its predecessor, and then travel backwards again, until we get to s! This gives us a path, in reverse order, from s to u. This might look as follows:
1 while predecessor[u] is not empty:
2 p = predecessor[u]
3 add p to path
4 u = p
5 reverse path
Up to this point, we have been assuming some properties of the graph. Namely, we have assumed that the graph is connected, unweighted, and undirected.
First, let's see what happens when we have a disconnected graph.
As you can see, vertex d never gets visited. It is really quite obvious why this happens. There are some important and useful consequences of this:
Property 2 can be used for the same purpose; however, there are some
implementation details to consider before we use it! You may have noticed that
Integers in most programming languages can't be set to infinity! So
you can try using something like Integer.MAX_VALUE
in Java. This
is usually safe, but not always, since there might be the case where you
actually have a graph with a path of this length! But since we have made the
assumption that the graph is unweighted, we know that the distance can't be
negative. So you can use a negative value as a substitute for infinity.
Now, what happens if the graph is directed? It just so happens that the BFS continues to work in this case! Too, the above three properties will still hold if the graph is both directed and disconnected.
As for weighted graphs, that is a more advanced topic that will be covered later!
No sample problems yet!


| Trees | Indices | Help |
|
|---|
|
|
www.CGIHandler --+ | EndRequestPreq
Request to be submitted to ENDORSE.SUBMIT
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
| Public Methods | |||
|---|---|---|---|
|
Inherited from |
|||
| Support Methods | |||
|
Inherited from |
|
|||
|
Inherited from |
|
|||
|
Inherited from |
|
Handle request and output
|
| Trees | Indices | Help |
|
|---|
| Generated by Epydoc 3.0.1 on Mon Oct 8 11:15:38 2012 | http://epydoc.sourceforge.net |


{% extends "header.html" %} {% block body %}
Now we are ready to improve the previous program. We notice immediately two major issues. The first is that cubes pop up sometimes very close to us. Ideally, they all start behind our far clipping plane. The second problem is that we can go in any direction, and avoid the barrage of cubes. We'd like to be able to move left, right, up, or down, and have cubes continue to come.
The first problem is easily solved with some minor tweaks. The second problem is a bit... bigger. The idea is to give the sense that there are cubes everywhere. This means we want the player to be able to just press and hold, say, the left arrow key, and never outpace the cubes. This means that we need to have some cubes off to the sides "just in case." These cubes may never even be seen, but we need them to the left, right, above, and below the "user." This means more processing, but we can balance that. Keep in mind that, again, this will depend on your processing abilities. You can always decrease the amount of cubes, as well as change the pace at which you can move to save processing.
Let's get started. First, we're going to change set_vertices a bit by including the location of the "camera," or user.
def set_vertices(max_distance, min_distance = -20, camera_x = 0, camera_y = 0): camera_x = -1*int(camera_x) camera_y = -1*int(camera_y) x_value_change = random.randrange(camera_x-75,camera_x+75) y_value_change = random.randrange(camera_y-75,camera_y+75) z_value_change = random.randrange(-1*max_distance,min_distance) new_vertices = [] for vert in vertices: new_vert = [] new_x = vert[0] + x_value_change new_y = vert[1] + y_value_change new_z = vert[2] + z_value_change new_vert.append(new_x) new_vert.append(new_y) new_vert.append(new_z) new_vertices.append(new_vert) return new_vertices
Next, let's modify the main function's top a bit:
def main(): pygame.init() display = (800,600) pygame.display.set_mode(display, DOUBLEBUF|OPENGL) max_distance = 100 gluPerspective(45, (display[0]/display[1]), 0.1, max_distance) glTranslatef(0,0, -40) #object_passed = False x_move = 0 y_move = 0 cur_x = 0 cur_y = 0 game_speed = 2 direction_speed = 2 cube_dict = {} for x in range(50): cube_dict[x] =set_vertices(max_distance)
Notice the larger range, as well as cur_x and cur_y mostly. We've also added game_speed and direction_speed, so you can tweak things easier.
Since we're changing that, we need to change our event loop slightly as well:
while True: for event in pygame.event.get(): if event.type == pygame.QUIT: pygame.quit() quit() if event.type == pygame.KEYDOWN: if event.key == pygame.K_LEFT: x_move = direction_speed if event.key == pygame.K_RIGHT: x_move = -1*direction_speed if event.key == pygame.K_UP: y_move = -1*direction_speed if event.key == pygame.K_DOWN: y_move = direction_speed
Now, not only can we tweak these speeds, but the game can also tweak them. A common thing to do is speed up the game to increase challenge.
Now to conclude our main function:
x = glGetDoublev(GL_MODELVIEW_MATRIX) #print(x) camera_x = x[3][0] camera_y = x[3][1] camera_z = x[3][2] cur_x += x_move cur_y += y_move glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) glTranslatef(x_move,y_move,game_speed) for each_cube in cube_dict: Cube(cube_dict[each_cube]) for each_cube in cube_dict: if camera_z <= cube_dict[each_cube][0][2]: new_max = int(-1*(camera_z-(max_distance*2))) cube_dict[each_cube] = set_vertices(new_max,int(camera_z-max_distance), cur_x, cur_y) pygame.display.flip()
Notice mainly the "cur_x" and "cur_y" additions, along with the new var game_speed in the translate function.
The next main change is in the generation of new cubes in the final for loop (in the above code).
The full code up to this point is:
import pygame from pygame.locals import * from OpenGL.GL import * from OpenGL.GLU import * import random vertices = ( (1, -1, -1), (1, 1, -1), (-1, 1, -1), (-1, -1, -1), (1, -1, 1), (1, 1, 1), (-1, -1, 1), (-1, 1, 1) ) edges = ( (0,1), (0,3), (0,4), (2,1), (2,3), (2,7), (6,3), (6,4), (6,7), (5,1), (5,4), (5,7) ) surfaces = ( (0,1,2,3), (3,2,7,6), (6,7,5,4), (4,5,1,0), (1,5,7,2), (4,0,3,6) ) colors = ( (1,0,0), (0,1,0), (0,0,1), (0,1,0), (1,1,1), (0,1,1), (1,0,0), (0,1,0), (0,0,1), (1,0,0), (1,1,1), (0,1,1), ) ##ground_vertices = ( ## (-10, -1.1, 20), ## (10, -1.1, 20), ## (-10, -1.1, -300), ## (10, -1.1, -300), ## ) ## ## ##def ground(): ## glBegin(GL_QUADS) ## for vertex in ground_vertices: ## glColor3fv((0,0.5,0.5)) ## glVertex3fv(vertex) ## ## glEnd() def set_vertices(max_distance, min_distance = -20, camera_x = 0, camera_y = 0): camera_x = -1*int(camera_x) camera_y = -1*int(camera_y) x_value_change = random.randrange(camera_x-75,camera_x+75) y_value_change = random.randrange(camera_y-75,camera_y+75) z_value_change = random.randrange(-1*max_distance,min_distance) new_vertices = [] for vert in vertices: new_vert = [] new_x = vert[0] + x_value_change new_y = vert[1] + y_value_change new_z = vert[2] + z_value_change new_vert.append(new_x) new_vert.append(new_y) new_vert.append(new_z) new_vertices.append(new_vert) return new_vertices def Cube(vertices): glBegin(GL_QUADS) for surface in surfaces: x = 0 for vertex in surface: x+=1 glColor3fv(colors[x]) glVertex3fv(vertices[vertex]) glEnd() glBegin(GL_LINES) for edge in edges: for vertex in edge: glVertex3fv(vertices[vertex]) glEnd() def main(): pygame.init() display = (800,600) pygame.display.set_mode(display, DOUBLEBUF|OPENGL) max_distance = 100 gluPerspective(45, (display[0]/display[1]), 0.1, max_distance) glTranslatef(0,0, -40) #object_passed = False x_move = 0 y_move = 0 cur_x = 0 cur_y = 0 game_speed = 2 direction_speed = 2 cube_dict = {} for x in range(50): cube_dict[x] =set_vertices(max_distance) #glRotatef(25, 2, 1, 0) while True: for event in pygame.event.get(): if event.type == pygame.QUIT: pygame.quit() quit() if event.type == pygame.KEYDOWN: if event.key == pygame.K_LEFT: x_move = direction_speed if event.key == pygame.K_RIGHT: x_move = -1*direction_speed if event.key == pygame.K_UP: y_move = -1*direction_speed if event.key == pygame.K_DOWN: y_move = direction_speed if event.type == pygame.KEYUP: if event.key == pygame.K_LEFT or event.key == pygame.K_RIGHT: x_move = 0 if event.key == pygame.K_UP or event.key == pygame.K_DOWN: y_move = 0 ## if event.type == pygame.MOUSEBUTTONDOWN: ## if event.button == 4: ## glTranslatef(0,0,1.0) ## ## if event.button == 5: ## glTranslatef(0,0,-1.0) #glRotatef(1, 3, 1, 1) x = glGetDoublev(GL_MODELVIEW_MATRIX) #print(x) camera_x = x[3][0] camera_y = x[3][1] camera_z = x[3][2] cur_x += x_move cur_y += y_move glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) glTranslatef(x_move,y_move,game_speed) for each_cube in cube_dict: Cube(cube_dict[each_cube]) for each_cube in cube_dict: if camera_z <= cube_dict[each_cube][0][2]: new_max = int(-1*(camera_z-(max_distance*2))) cube_dict[each_cube] = set_vertices(new_max,int(camera_z-max_distance), cur_x, cur_y) pygame.display.flip() main() pygame.quit() quit()
That's it for PyOpenGL tutorials for now. Next, you can head to the: for more tutorials!


1.What's in a name? That which we call a rose by any other name would smell as sweet. 2.There is nothing either good or bad but thinking makes it so. 3.There is a tide in the affairs of men, Which taken at the flood, leads on to fortune. Omitted, all the voyage of their life is bound in shallows and in miseries. On such a full sea are we now afloat. And we must take the current when it serves, or lose our ventures.
1.Happiness is when what you think, what you say, and what you do are in harmony. 2.You must not lose faith in humanity. Humanity is an ocean; if a few drops of the ocean are dirty, the ocean does not become dirty. 3.It is better to be violent, if there is violence in our hearts, than to put on the cloak of nonviolence to cover impotence.
1.Teaching is a very noble profession that shapes the character, caliber, and future of an individual. If the people remember me as a good teacher, that will be the biggest honour for me. 2.Where there is righteousness in the heart, there is beauty in the character. When there is beauty in the character, there is harmony in the home. When there is harmony in the home, there is order in the nation. When there is order in the nation, there is peace in the world.


"Everything is related to everything, but near things are more related than distant things" - Waldo Tobler
This law of Geography can be translated to the language of Statistics as: things nearby are more alike and thus we can expect them to be positively correlated. The core idea behind Spatial Statistics is to understand an characterize this spatial dependence that is observed in different processes, for example: amount of rainfall, global temperature, air pollution, etc.
Spatial dependence, as a concept, started to be formally addressed by statisticians at the begining of the XX century (Fisher, 1935). However, it was until the second half of the century when Matheron (1962, 1963) and Gandin (1963) developed a best linear unbiased predictor (blup) for spatial modeling. Kriging, as the predictor is known, became the standard modeling approach of continuous spatial variation.
Geostatistical methods have traditionally been related to data observed at a set of spatial locations indexed in a continuous space. This is as opposed to lattice methods and point pattern methods. The former related to data observed on a fixed set locations and the latter used for data associated to a point process. Diggle et al. (2013) consider that distinctions based on data formats are not always appropriate. They argue that the main theoretical distinction within spatial statistics is the continuity or non-continuity of the process being modeled. The model-based approach (Diggle et al., 1998) has become the current paradigm for modelling variability and quantifying uncertainty: a hierarchical thinking that explicitly assumes a stochastic model, but also acknowledges a different uncertainty in the data and in the parameters of the process (Cressie and Wikle, 2011).
Along the discussion we will show some interactive examples. We will use some auxiliary code for running these examples, which will be available for everybody to download. The following packages need to be preinstalled for the code of this session to work:
In general, we will call the methods of the packages as:
This is as opposed to loading the package and calling the method directly:
We will do this so that it is explicit which method belongs to which package. The exceptions to this rule will be ggplo2 and Rccp. The former because otherwise notation becomes too long and less readable when making plots, and the latter because otherwise we will run into problems when calling rstan.
The commmand below will load ggplot, Rcpp and the auxiliary code needed for the session.
library(ggplot2)
library(Rcpp)
source("r_code/auxiliary_functions.R")
source("r_code/jupyter_stuff.R") # Only needed if using jupyter
The command below will fix the size of the plots diplayed. It is not necessary, but will help us along this interactive session.
options(repr.plot.width=5, repr.plot.height=4)
As a first step it will be convenient to have a brief recap on linear regresson. Let $\{y_i, x_{i1}, \ldots, x_{ip}\}_{i=1}^n$ be a set of dependent and explanatory variables. In Ordinary Least Squares we usually want to estimate the parameters $\{\beta_j\}_{j=0}^{p}$ that best fit the model:
$y_i = \beta_0 + \beta_1 x_{i1} + \ldots + \beta_j x_{ip} + \epsilon_i$ for $i = 1, \ldots, n$.
where $\epsilon_i$ a Gaussian random variable with mean zero. The parameters choosen are the ones that generate the hyperplane that is closest on average to all the points $x_{ij}$.
# Generate some toy data
reg_data <- lm_data(seed=0)
reg_data$type <- 'observed'
# Fit linear model
model1 <- lm(y ~ x, data=reg_data)
# Predictions
x_linspace <- seq(.5, 2.5 * pi, length.out=100)
to_append <- data.frame(x=x_linspace,
y=predict(model1, newdata=data.frame(x=x_linspace)),
type='order_1')
reg_data <- rbind(reg_data, to_append)
# Plot results
plt <- ggplot(data=subset(reg_data, type=='observed'), aes(x, y)) +
geom_point(col="steelblue", size=2) +
geom_line(data=subset(reg_data, type=='order_1'), col="red") +
ggtitle(label = "Linear Regression Example") +
theme_minimal()
publish_gg(plt)
Sometimes (almost everytime) a straight line is not enough to explain the relationship between $x_i$ and $y_i$. This usually makes us start thinking of some polynomial regression that is more adequate. For example:
Order 2: $y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \epsilon_i$
Order 3: $y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 + \epsilon_i$
n_poly <- 4
poly_data <- reg_data
for (i in 2:n_poly) {
new_dataset <- subset(reg_data, type=='observed')[,-c(3)]
new_linspace <- data.frame(x=x_linspace)
for (j in 2:i) {
new_dataset[paste0("x", j)] <- new_dataset$x^j
new_linspace[paste0("x", j)] <- new_linspace$x^j
}
m <- lm(y ~ ., data = new_dataset)
to_append <- data.frame(x=x_linspace,
y=predict(m, newdata=new_linspace),
type=paste("order", i, sep="_"))
poly_data <- rbind(poly_data, to_append)
}
# Plot results
plt <- ggplot(data=subset(poly_data, type=='observed'), aes(x, y)) +
geom_point(col="steelblue", size=2) +
geom_line(data=subset(poly_data, type!='observed'), aes(color=type)) +
ggtitle(label = "Polynomial Regression Example") +
theme_minimal()
publish_gg(plt)
Q: Which of the lines is a better fit to the given points?
Linear regression models can be used to summarize the variation in spatial patterns. A trend surface of order $n$ can be estimated by applying a regression model using the spatial coordinates as the independent variables. For example, let the longitude and latitude coordinates be defined as $x_{i1}$ and $x_{i2}$ respectively, then we can describe the following trend surfaces:
Order 1: $y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \beta_3 x_{i1} x_{i2} + \epsilon_i$
Order 2: $y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \beta_3 x_{i1} x_{i2} + \beta_4 x_{i1}^2 + \beta_5 x_{i2}^2 + \epsilon_i $
Note that we are also adding the product $x_{i1} x_{i2}$. This is a term that accounts for the interactions.
Having a trend across the surface seems like a good idea. Now, we will make a new toy dataset to see how this approach works.
Imagine that we have a plot of $400m \times 400 m$ where we want to study a specific property of the soil, we will call it soiliness. We pick different locations at random and take measurments of soiliness.
spatial_reg <- soil_data(n_peaks=2, n_data = 150, seed=0)
head(spatial_reg)
| lng | lat | soiliness |
|---|---|---|
| 357.78218 | 344.9534 | 23.65847 |
| 105.93796 | 245.2433 | 58.68277 |
| 148.47744 | 222.3067 | 44.59935 |
| 228.56849 | 131.1822 | 15.87705 |
| 362.37491 | 180.7994 | 41.43058 |
| 80.47109 | 199.6759 | 42.54411 |
gg <- ggplot(spatial_reg, aes(lng, lat)) +
geom_point(aes(col=soiliness), size=2.5) +
viridis::scale_color_viridis(option="plasma") +
ggtitle("Clustered Spatial Data") +
theme_minimal()
publish_gg(gg)
# Fit a model with a trend of order 1
surf_order1 <- lm(soiliness ~ lng * lat, data=spatial_reg)
# Fit a model with a trend of order 2
surf_order2 <- lm(soiliness ~ lng * lat + I(lng^2) + I(lat^2), data=spatial_reg)
# Grid for predictions
surf_grid <- as.data.frame(make_grid(size = 20))
surf_grid$surface_o1 <- predict(surf_order1, newdata=surf_grid)
surf_grid$surface_o2 <- predict(surf_order2, newdata=surf_grid)
plt <- ggplot(surf_grid, aes(lng, lat)) +
geom_raster(aes(fill=surface_o1)) +
geom_contour(aes(z=surface_o1), col="white", linetype=1, alpha=.5) +
geom_point(data=spatial_reg, aes(fill=soiliness), colour="white", pch=21, size=3) +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue") +
ggtitle("Trend Surface: Order 1") +
theme_minimal()
publish_gg(plt)
Q: How would the surface look if we hadn't included the interaction term $x_i x_j$?
plt <- ggplot(surf_grid, aes(lng, lat)) +
geom_raster(aes(fill=surface_o2)) +
geom_contour(aes(z=surface_o2), col="white", linetype=1, alpha=.5) +
geom_point(data=spatial_reg, aes(fill=soiliness), colour="white",
pch=21, size=3) +
ggtitle("Trend Surface Order 2") +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue") +
theme_minimal()
publish_gg(plt)
Q: How does a surface of a higher order look like?
It may be the case that for some purpose a surface like the one above is a model good enough. It also can happen that such a model is not good.
Can we get a fit that resembles more what the data looks like? The answer is yes, as we will see next.
What we need from a model to produce a better fit is that it works under the assumption that points which are close are likely to have similar target values (n_positive in this case). In other words, we expect the observed target values to change together across space.
When want to measure how much two variables change together, we use the covariance function. Under the right assumptions, we can also use the covariance function to describe the similarity of the observed values based on their location.
Covariance function: A function $K: S \times S → R$ is a covariance function if it is continuous, symmetric and positive definite.
For example, assume that $S \subset R^2$ and that $r = ||(x_{i1},x_{i2})^\top - (x_{j1},x_{j2})^\top ||$.
The exponentiated quadratic or RBF covariance function is defined as
$K(x_i, x_j) = \sigma^2 exp \left( \frac{- r^2 }{ 2 \ell^2} \right)$
for some positive parameters $\sigma$ and $\ell$.
The rational quadratic covariance function is defined as
$K(x_i, x_j) = \sigma^2 \left( 1 + \frac{ r^2 }{ 2 \alpha \ell^2} \right) ^{-\alpha}$
for some positive parameters $\sigma$, $\alpha$ and $\ell$.
The power exponential covariance function is defined as
$K(x_i, x_j) = \sigma^2 exp\left( - \left(\frac{ r }{ \ell} \right) ^{\gamma} \right)$
for some positive parameters $\sigma$ and $\ell$.
The election of which covariance function to use depends on our assumptions about the change in the association between the points across space (eg., the speed of decay).
x_0 <- as.matrix(0, nrow=1, ncol=1)
x_linspace <- as.matrix(seq(-8, 8, length.out = 100), ncol=1)
kval <- c(as.numeric(rbf(x_0, x_linspace, sigma=1, lengthscale=1)),
as.numeric(rqd(x_0, x_linspace, sigma=1, lengthscale=1, alpha=1)),
as.numeric(pow(x_0, x_linspace, sigma=1, lengthscale=1, gamma=1)))
kname <- c(rep("exp_quad", 100), rep("rat_quad", 100), rep("pow_exp", 100))
covdf <- data.frame(x=rep(x_linspace, 3), k=kval, covariance= kname)
plt <- ggplot(covdf, aes(x, k)) + geom_line(aes(col=covariance)) +
ggtitle("Different Covariance Functions") +
theme_minimal()
publish_gg(plt)
Here is an example in 2 dimensions to illustrate the concept of the covariance function. Suppose we have 10 locations randomly selected in our in our $400m \times 400m$ plot.
xx <- random_points_ordered(n_points = 10, seed=0)
plt <- ggplot(as.data.frame(xx), aes(lng, lat)) +
geom_point(col="steelblue", size=2.5) +
theme_minimal()
publish_gg(plt)
Based on the pairwise distances between the points, the resulting covariance matrix under a Rational Quadratic model is the following.
K <- rqd(xx, xx, lengthscale=20, alpha=2)
kdf <- data.frame(k = as.vector(K),
point_i = factor(rep(10:1, 10)),
point_j = factor(sort(rep(1:10, 10), decreasing=TRUE)))
plt <- ggplot(kdf, aes(point_j, point_i)) +
geom_raster(aes(fill=k)) +
viridis::scale_fill_viridis(option="plasma") +
ggtitle("Covariance Matrix") +
scale_y_discrete(limits=rev(levels(kdf$point_i))) +
theme_minimal()
publish_gg(plt)
It is not only the shape of the covariance function that matters, but also the values of its parameters: $\sigma$, $\ell$, $\gamma$, etc. We will come back to that later.
Now that we have discussed how the covariance function can help describe properties we observe in our data, we can discuss how to incorporate this ideas into a model.
Before, we mentioned that a surface trend could be modeled as
$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \beta_3 x_{i1} x_{i2} + \epsilon_i$.
Instead we will model our data as
$y_i = f_i(x_i) + \epsilon_i$,
where $f = (f_1, \ldots, f_n)^\top$ is a multivariate normal distribution.
Why? A Gaussian distribution is completely defined by its first two moments. Moreover, a centered Gaussian distribution is uniquely defined by its covariance. This means that in our new model, for every $y_i$ we will have now an underlying gaussian variable $f_i$ with the property that all ${f_i}$ have a spatial dependance structure given by $K(x_i, x_j)$.
Gaussian process: A Gaussian process is a collection of random variables, any finite number of which have a joint Gaussian distribution.
We denote a Gaussian process with mean function $m$ and covariance function $K$ as $f(x) \sim \mathcal{GP}\big(m(x), K(x, x)\big)$.
While in the linear regression model we estimate the $\beta$ parameters, here we estimate the values of $f_i$ at the input locations $x_i$. We have to be aware that $f_i$ are not parameters in the sense of classical statistics, they are random variables and we will handle them under a Bayesian approach. Given the observed data $\{x, y\}$, the posterior distribution of $f$ is given by:
$p(f|y,x) \propto p(y|f,x) p(f|x)$
We will assume for the moment that $m(x)=0$. This is a common assumption that doesn't reduce the flexibility of the method, but simplifies handling it. Notice $\epsilon = (\epsilon_1, \ldots, \epsilon_n)^\top$ is joint Gaussian with mean zero and Covariance matrix $I \sigma^2_\epsilon$. Thus $y \sim N(0, K(x,x) + I\sigma^2_\epsilon)$. Then it turns out that $f_\ast|y,x$ (posterior of $f$ at new locations $x_\ast$) is multivariate Gaussian with mean
$E(f_\ast|y,x) = K(x_\ast,x)(K(x,x) + I\sigma^2_\epsilon)^{-1}y$
and covariance
$Var(f_\ast|y,x) =K(x_\ast,x_\ast) - K(x_\ast,x)(K + I\sigma^2_\epsilon)^{-1} K(x, x_\ast)$.
Usually, we want to predict new realizations of $y_\ast$, instead of $f_\ast$. The predictive values of $y_\ast$, given the observations are computed as
$p(y_\ast|y,x) = \int p(y_\ast |f_\ast) p(f_\ast|y,x) df_\ast$.
In this case, since $y$ is just a noisy version of $f$ where the noise is Gaussian with zero mean, it turns out that
$E(y_\ast|y,x) = K(x_\ast,x)(K(x,x) + I\sigma^2_\epsilon)^{-1}y$
and
$Var(y_\ast|y,x) =K(x_\ast,x_\ast) - K(x_\ast,x)(K + I\sigma^2_\epsilon)^{-1} K(x, x_\ast) + I\sigma^2_\epsilon$.
Going back to our soil example, this is how a Gaussian process regression fit looks like.
# For the moment we will assume that the following quantities are known
rbf_sigma <- 20
rbf_lengthscale <- 25
epsilon_sigma <- 10
# These are the locations of the train set and predictions
X <- as.matrix(spatial_reg[, c("lng", "lat")])
X_star <- as.matrix(surf_grid[, c("lng", "lat")])
# Output value
Y <- as.matrix(spatial_reg$soiliness)
# Fit a Gaussian process
gp_fit <- gp_regression_rbf(X, Y, X_star, k_var=rbf_sigma^2, k_len=rbf_lengthscale, noise_var=epsilon_sigma^2)
surf_grid$surface_gp_mean <- gp_fit$gp_mean
surf_grid$surface_gp_var <- gp_fit$gp_var
plt <- ggplot(surf_grid, aes(lng, lat)) +
geom_raster(aes(fill=surface_gp_mean)) +
geom_contour(aes(z=surface_gp_mean), col="white", linetype=1, alpha=.5) +
#geom_point(data=spatial_reg, aes(fill=soiliness), colour="white", pch=21, size=3) +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue") +
ggtitle("Gaussian Process - Predictive Mean") +
theme_minimal()
publish_gg(plt)
plt <- ggplot(surf_grid, aes(lng, lat)) +
geom_raster(aes(fill=surface_gp_var)) +
geom_contour(aes(z=surface_gp_var), col="white", linetype=1, alpha=.5) +
geom_point(data=spatial_reg, colour="white", pch=21, size=3) +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue") +
ggtitle("Gaussian Process - Predictive Variance") +
theme_minimal()
publish_gg(plt)
Notice that the regions with no points have higher variance, while the regions with many points concentrated have a low variance. Remember that in the Bayesian approach, the variance is a measure of uncertainty. Thus, we are more uncertain about the process when there are no observations nearby and become more certain the more observations we have.
Since the Gaussian process is defined by its mean and covariance, these are usually the parameters we are mostly interested in estimating. However these are not the only parameters of the model. As we discussed before, the covariance functions have different parameters associated, for example $\sigma$, $\ell$ or $\gamma$. We usually refer to them as hyperparameters.
Below we will see how changes in the lengthscale affect the covariance function.
x_0 <- as.matrix(0, nrow=1, ncol=1)
x_linspace <- as.matrix(seq(-8, 8, length.out = 100), ncol=1)
exp_quad <- c()
for (ell in c(.5, 1, 1.5, 2, 4)) {
k <- rbf(x_0, x_linspace, sigma=1, lengthscale=ell)
exp_quad <- rbind(exp_quad,
data.frame(x=x_linspace,
cov_function=as.numeric(k),
lengthscale=paste0('lengthscale=',ell)))
}
plt <- ggplot(exp_quad, aes(x, cov_function)) +
geom_line(aes(color=lengthscale)) +
ggtitle("Exponentiated Quadratic Covariance Function (sigma=1)") +
theme_minimal()
publish_gg(plt)
x_0 <- as.matrix(0, nrow=1, ncol=1)
x_linspace <- as.matrix(seq(-8, 8, length.out = 100), ncol=1)
rat_quad <- c()
for (ell in c(.5, 1, 1.5, 2, 4)) {
k <- rqd(x_0, x_linspace, sigma=1, lengthscale=ell, alpha=.1)
rat_quad <- rbind(rat_quad,
data.frame(x=x_linspace,
cov_function=as.numeric(k),
lengthscale=paste0('lengthscale=',ell)))
}
plt <- ggplot(rat_quad, aes(x, cov_function)) +
geom_line(aes(color=lengthscale)) +
ggtitle("Rational Quadratic Covariance Function (sigma=1, alpha=.1)") +
theme_minimal()
publish_gg(plt)
Q: How does the function changes with different values of $\sigma$ and $\alpha$?
The values of the hyperparameters can also be chosen based on the data. While the optimal solutions for the posterior mean and variance have a nice and simple analytical expression, as we saw above, estimating optimal values for the hyperparamters is not that straightforward.
One solution is to use Markov Chain Monte Carlo methods to find their posterior distribution. Notice that when taking into account the hyperparameters the posterior distribution of $f$ should be written as
$p(f|y,x,\theta) \propto p(y|f,\theta) p(f|x, \theta) $.
The normalizing constant of the expression above is given by
$p(y|x,\theta) = \int p(y|f,\theta) p(f|x, \theta) df$,
which can be used to calculate the posterior distribution of $\theta$:
$p(\theta|y,x) \propto p(y|x,\theta) p(\theta)$,
with normalizing constant
$p(y|x) = \int p(y|x,\theta) p(\theta) d\theta$.
Solving the integral above can be a hard task. An alternative is to find the value of $\theta$ that maximizes the marginal likelihood $p(y|x,\theta)$. Here the term marginal refers to the fact that $p(y|x,\theta)$ is computed after marginalizing the latent function $f$. By following this approach we are choosing the combination of $\theta$ values where the likelihood of the data observed $y$ is the highest. This approach is know as Maximum Likelihood type II.
Here we will use Monte Carlo methods to estimate the hyperparameters distribution and use the mean values as point estimates.
fit_hyper <- rstan::stan(file="stan_code/gp_gaussian_fit_hyper.stan",
data=list(n_data = nrow(spatial_reg),
n_dim = 2,
y_data = spatial_reg$soiliness,
X_data = spatial_reg[, c("lng", "lat")],
mu_data = rep(0, nrow(spatial_reg))),
pars = c("noise_var", "cov_var", "cov_length"),
warmup = 1000, iter = 5000, chains = 4,
verbose = FALSE, seed=123)
hyper_params <- rstan::extract(fit_hyper)
noise_var <- mean(cbind(hyper_params$noise_var))
cov_var <- mean(cbind(hyper_params$cov_var))
cov_length <- colMeans(cbind(hyper_params$cov_length))
Now we will plot the predicted surface with the point estimates.
Note: Our stan implementation considers a different lengthscale for each dimension in X, while our RBF function considers the same one. For the moment we will use as lengthscale value the average of both lenghtscale parameters.
# Fit a Gaussian process
gp_fit_stan <- gp_regression_rbf(X, Y, X_star, k_var=cov_var, k_len=mean(cov_length), noise_var=noise_var)
surf_grid$surface_gp_mean_stan <- gp_fit_stan$gp_mean
surf_grid$surface_gp_var_stan <- gp_fit_stan$gp_var
# Plot predictive mean
plt <- ggplot(surf_grid, aes(lng, lat)) +
geom_raster(aes(fill=surface_gp_mean_stan)) +
geom_contour(aes(z=surface_gp_mean_stan), col="white", linetype=1, alpha=.5) +
geom_point(data=spatial_reg, aes(fill=soiliness), colour="white", pch=21, size=3) +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue") +
ggtitle("Gaussian Process - Predictive Mean") +
theme_minimal()
publish_gg(plt)
# Fit a Gaussian process
gp_fit_stan <- gp_regression_rbf(X, Y, X_star, k_var=cov_var, k_len=mean(cov_length), noise_var=noise_var)
surf_grid$surface_gp_mean_stan <- gp_fit_stan$gp_mean
surf_grid$surface_gp_var_stan <- gp_fit_stan$gp_var
# Plot predictive mean
plt <- ggplot(surf_grid, aes(lng, lat)) +
geom_raster(aes(fill=surface_gp_mean_stan)) +
geom_contour(aes(z=surface_gp_mean_stan), col="white", linetype=1, alpha=.5) +
geom_point(data=spatial_reg, aes(fill=soiliness), colour="white", pch=21, size=3) +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue") +
ggtitle("Gaussian Process - Predictive Mean") +
theme_minimal()
publish_gg(plt)
So far all the discussion has been around spatial location. What about additional information we may have?. We can have different approaches to include it.
For example, we can define a model as follows:
$y_i = \beta_0 + \beta_1 x_{i1} + \ldots + \beta_q x_{iq} + f(x_{is}) + \epsilon_i$, where
$\{x_{ij}\}_{j=1}^{q}$ are different covariantes and $x_{is}$ are spatial coordinates.
Another approach could be to use
$y_i = f(x^\ast_i) + \epsilon_i$, where $x^\ast_i$ is a vector that includes the spatial locations and other data features. In such a case, these features will also have a non-linear effect. For example, if the input vector contains spatial location and a temporal reference, $x^\ast = (x^{(s)}, x^{(t)})$, a covariance function for a space-time process can be defined as:
$K(x^\ast_i, x^\ast_j) = \sigma^2 exp \left( -\frac{||x^{(s)}_i - x^{(s)}_j||^2 }{ 2 \ell^2_s} - \frac{||x^{(t)}_i - x^{(t)}_j||^2 }{ 2 \ell^2_t} \right)$.
Note that the expression above, is in fact the product of two covariance functions:
$K(x^\ast_i, x^\ast_j) = \sigma^2 exp \left( -\frac{||x^{(s)}_i - x^{(s)}_j||^2 }{ 2 \ell^2_s} \right) exp \left(- \frac{||x^{(t)}_i - x^{(t)}_j||^2 }{ 2 \ell^2_t} \right)$.
It is also possible to use sums of covariance functions to have additive effects of the form: $y_i = f_1(x^\ast_i) + f_2(x^\ast_i) + \epsilon_i$.
We have mentioned how to compute the posterior mean and variance. We also gave a brief description of how the hyperparameters can be estimated based on the data. Nevertheless, this does not tells us how to choose between different models. For example, given a covariance function we can optimize its hyperparameters, but How do we choose between two models with different covariance functions?
The idea behind cross-validation is to split the data we have into two disjoint sets: one for training and one to monitor the performance of the trained model. The performance is measured according to a criterion such as the sum of squared errors. Then the performance on the validation set is used as a proxy for the generalization error and model selection is carried out using this measure.
If the data avilalbe is scarse, holding out some data for validation may be far from ideal since the training and validation sets would be even scarser. A workaround this caveat is k-fold cross-validation: the data is split into k disjoint, equally sized subsets ; the model is trained k times, each one using a different subset for validation and the remaining for training. This way all elements in the dataset are used for validation and training.
The extreme case of having as many folds as data points is known as leave-one-out cross-validation (LOO-CV).
# K-fold validation diagram
k = 5
label_ <- rep("Train", k^2)
for(i in 1:k){
label_[k*(i-1) +(k+1-i)] <- "Validation"
}
cvplot <- data.frame(x=rep(1:k, k), y=sort(rep(1:k, k), decreasing = TRUE), label=label_)
plt <- ggplot(cvplot, aes(x, y)) + geom_tile(aes(fill=label)) +
geom_vline(xintercept = 1:k + .5, col='white') +
geom_hline(yintercept = 1:k + .5, col="white", size=10) +
scale_y_continuous(breaks = 1:k, labels = paste("fold", 1:k)) +
scale_x_continuous(breaks = 1:k, labels = paste("subset", 1:k)) +
ggtitle(paste0(k, "-Fold Validation Diagram")) +
theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank(),
panel.background = element_blank()) +
ylab('') + xlab('')
publish_gg(plt)
Suppose we are choosing between a model with an Exponentiated Quadratic kernel and a model with a Rational Quadratic kernel, as shown below.
# RBF kernel
noise_var_1 <- 6
rbf_var <- 18
rbf_length <- 40
# Rational quadratic kernel
noise_var_2 <- 6
rqd_var <- 18
rqd_length <- 40
rqd_alpha <-1
surf_grid$model_1 <- gp_regression_rbf(X, Y, X_star, k_var=rbf_var, k_len=rbf_length,
noise_var=noise_var_1)$gp_mean
surf_grid$model_2 <- gp_regression_rqd(X, Y, X_star, k_var=rqd_var, k_len=rqd_length,
noise_var=noise_var_2, alpha=rqd_alpha)$gp_mean
# Change dimensions to allow side by side display
options(repr.plot.width=10, repr.plot.height=4)
# Plot predictive mean
plt1 <- ggplot(surf_grid, aes(lng, lat)) +
geom_raster(aes(fill=model_1)) +
geom_contour(aes(z=model_1), col="white", linetype=1, alpha=.5) +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue", lim=c(5,55)) +
ggtitle("Exponentiated Quadratic Kernel") +
theme_minimal()
plt2 <- ggplot(surf_grid, aes(lng, lat)) +
geom_raster(aes(fill=model_2)) +
geom_contour(aes(z=model_2), col="white", linetype=1, alpha=.5) +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue", lim=c(5,55)) +
ggtitle("Rational Quadratic Kernel") +
theme_minimal()
plt <- cowplot::plot_grid(plt1, plt2)
publish_gg(plt)
We will compare this models using a 5-fold validation.
# Make an index with the k folds
ix = caret::createFolds(Y, k = 5)
sse1 = 0
sse2 = 0
for (i in 1:5){
# Split data
Xtrain <- X[-ix[[i]], ]
Ytrain <- Y[-ix[[i]]]
Xval <- X[ix[[i]], ]
Yval <- Y[ix[[i]]]
# Train models
m1 <- gp_regression_rbf(X=Xtrain, Y=Ytrain, X_pred=Xval, k_var=rbf_var, k_len=rbf_length,
noise_var=noise_var_1)
m2 <- gp_regression_rqd(X=Xtrain, Y=Ytrain, X_pred=Xval, k_var=rqd_var, k_len=rqd_length,
noise_var=noise_var_1, alpha=rqd_alpha)
# Predict hold-out data
Y1_hat <- m1$gp_mean
Y2_hat <- m2$gp_mean
# Compute SSE
sse1 <- sse1 + sum((Yval - Y1_hat)^2)
sse2 <- sse2 + sum((Yval - Y2_hat)^2)
}
print(c(sse1, sse2))
[1] 4752.226 3583.977
Since Gaussian processes yield not only point estimates of the target values, but a probability distribution of such values, a metric that is better to use in this case is the predictive probability:
$p(y_{\{k-fold\}}|X,y_{-\{k-fold\}}, \theta)$,
where $y_{\{k-fold\}}$ is the hold-out set and $y_{-\{k-fold\}}$ are the remaining observations.
While LOO-CV can be computationally very expensive, computing the predictive probability in a Gaussian processes regression model has a shortcut, that does not require re-training the model $n$ times.
$\log p(y_i|X,y_{-i}, \theta) = -\frac{1}{2} \log \sigma^2_i - \frac{(y_i - \mu_i)^2}{2\sigma^2_i} - \frac{1}{2}\log 2\pi$,
where $\mu_i$ and $\sigma^2_i$ are the predictive mean and variance of $y_i$, and are computed as
$\mu_i = y_i - \frac{[(K + I\sigma^2_\epsilon)^{-1}y]_i}{[(K + I\sigma^2_\epsilon)^{-1}]_{ii}}$, and
$\sigma^2_i = \frac{1}{[(K + I\sigma^2_\epsilon)^{-1}]}_{ii}$
We will compute LOO-CV for the two models trained above.
K1 <- rbf(X, X, sigma=sqrt(rbf_var), lengthscale=rbf_length) + diag(x=noise_var_1, nrow=nrow(X))
K2 <- rqd(X, X, sigma=sqrt(rqd_var), lengthscale=rqd_length, alpha=rqd_alpha) + diag(x=noise_var_2, nrow=nrow(X))
W1 <- chol2inv(chol(K1))
W2 <- chol2inv(chol(K2))
s1 <- 1/diag(W1)
s2 <- 1/diag(W2)
mu1 <- Y - (W1 %*% Y) / diag(W1)
mu2 <- Y - (W2 %*% Y) / diag(W2)
log_pi1 <- .5 * sum(-log(s1) - (Y - mu1)^2/s1 - log(2*pi))
log_pi2 <- .5 * sum(-log(s2) - (Y - mu2)^2/s2 - log(2*pi))
print(c(log_pi1, log_pi2))
[1] -552.8673 -508.6756
Cressie, N. and Wikle, C. K. (2011). Statistics for spatio-temporal data. John Wiley & Sons.
Diggle, P. J., Moraga, P., Rowlingson, B., Taylor, B. M., et al. (2013). Spatial and spatio-temporal log-Gaussian Cox processes: Extending the geostatistical paradigm. Statistical Science, 28(4):542–563.
Diggle, P. J., Tawn, J., and Moyeed, R. (1998). Model-based geostatistics. Journal of the Royal Statistical Society: Series C (Applied Statistics), 47(3):299–350.
Fisher, R. A. (1935). The design of experiments. Oliver & Boyd.
Gandin, L. S. (1963). Ob“ektivnyi analiz meteorologicheskikh polei. Gidrometeologicheskoe Izdatel’stvo, Leningrad. Translation (1965): Objective analysis of meteorological fields. Israel Program for Scientific Translations, Jerusalem.
Matheron, G. (1962). Traité de géostatistique appliquée, tome I. Mémoires du Bureau de Recherche Géologiques et Minières, No. 14. Editions Technip, Paris.
Matheron, G. (1963). Traité de géostatistique appliquée, tome II: le krigeage. Mémoires du Bureau de Recherche Géologiques et Minières, No. 24. Editions Bureau de Recherche Géologiques et Minières, Paris.
Rasmussen, Carl Edward. "Gaussian processes in machine learning." Advanced lectures on machine learning. Springer, Berlin, Heidelberg, 2004. 63-71.
1) Modify the stan code so that it uses a Rational Quadratic Kernel. You can assume alpha=1.
2) Compare by 5-fold cross validation and LOO-CV the Gaussian process models fitted when using RBF and Rational Quadratic with the stan optimized hyperparameters. You can assume alpha=1.
3) Use the data meuse to fit a Gaussian process to model copper.
3.1) Modify the RBF kernel function to allow using different lengthscales per spatial dimension?
3.2) Try adding different variables, not only location (x, y)
3.3) Define a criterion to decide which set of variables make a better model.
data(meuse, package="sp")
head(meuse)
| x | y | cadmium | copper | lead | zinc | elev | dist | om | ffreq | soil | lime | landuse | dist.m |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 181072 | 333611 | 11.7 | 85 | 299 | 1022 | 7.909 | 0.00135803 | 13.6 | 1 | 1 | 1 | Ah | 50 |
| 181025 | 333558 | 8.6 | 81 | 277 | 1141 | 6.983 | 0.01222430 | 14.0 | 1 | 1 | 1 | Ah | 30 |
| 181165 | 333537 | 6.5 | 68 | 199 | 640 | 7.800 | 0.10302900 | 13.0 | 1 | 1 | 1 | Ah | 150 |
| 181298 | 333484 | 2.6 | 81 | 116 | 257 | 7.655 | 0.19009400 | 8.0 | 1 | 2 | 0 | Ga | 270 |
| 181307 | 333330 | 2.8 | 48 | 117 | 269 | 7.480 | 0.27709000 | 8.7 | 1 | 2 | 0 | Ah | 380 |
| 181390 | 333260 | 3.0 | 61 | 137 | 281 | 7.791 | 0.36406700 | 7.8 | 1 | 2 | 0 | Ga | 470 |
options(repr.plot.width=5, repr.plot.height=4)
plt <- ggplot(meuse, aes(x, y)) +
geom_point(aes(fill=copper), colour="white", pch=21, size=3) +
viridis::scale_fill_viridis(option="plasma", na.value="darkblue") +
ggtitle("Meuse Data") +
theme_minimal()
publish_gg(plt)


Toggle navigation
Serverless Web Monitor
Register Website
Name
Website
StatusCode
<% sites.forEach(function(site){ %>
<%= site.name%>
<%= site.url%>
<%= site.code%>
<% }) %>
×
Register Website
Name:
URL:
×
Remove from List
Are you sure you want remove this website from monitoring list?


RNA viruses virus OpenUrl Branch CrossRef dsRNA viruses. tree Figure RdRp RdRps protein; evolution branches protein viruses, capsid host distinct gene evolutionary groups analysis proteins set major genes genomes related members shown using conserved modules present Science Web genome viral sequences domains Koonin domain group network phylogenetic branch Res eukaryotic family single Virus diversity families supported RNA-dependent Virol clade similarity Biol ancestor diverse new showing Abstract FREE Text ancestral order plant However, cluster known polymerases protease; replication different represent helicases identified likely sequence used CPs Sci capping capsids invertebrates prokaryotic structural superfamily (e.g., Dolja Genome SH, SJR-CP clusters dots helicase; including larger range representative share (A) (B) Phylogenetic common evolved figure hosts module movement relationships rest type viruses" Baltimore Krupovic SJR-CPs additional encoding eukaryotes form included includes light lineage. loss main multiple origin representatives scenario structure supergroup well-supported DNA RTs bipartite case domains. double-stranded flaviviruses global icosahedral involved large nonstructural novel nucleocapsid prokaryotes protein, resulting right strongly taxa unclassified virion (drawn (indicated (see Acad Approximate Download Figure. Green Open SB). SJR Symbols alignment alphavirus apparently approximated boundaries). closely color-coded complex comprises connector data-hide-link-title=""> data-icon-position="" distantly encoded evolution. fuzzy genomic gray. infecting jelly-roll lineage lineages locations major, maps numbers parentheses parentheses. presumed procedure reconstruction replicons respective scale) spectrum summarize supergroup" tab trees virome "alphavirus Adv Branches CP, CapE, ICTV-accepted Natl Proc Viruses Zhang appear clustered current encode envelope filamentous flavivirus fungal history holobionts internal making number of-RNA phylogeny polymerase respectively, revealed similar split transfer (Data (RdRps): (many Chen Ghabrial Inv., MP, Mol NS, RdRps. Rev Supplemental Virology Wang animals appears assignment branch. consists cystoviruses deep discovered divergent domain; entropy expanded extensive forming given homologous include independent indicate labeled lack leviviruses naked partitiviruses plants recent reveals small supermodules support tree, uncertain); uncl., viruses: Gen ICTV Microbiol Nibert Notably, Protein SJR, acid acquired animal annotated associated branches, cases clades clustering combined complete comprise early exception functional gain genomes. invertebrate key namely, networks nidoviruses nodaviruses nodes pairwise particular, profiles protein. shared sharing suggests taxonomy tombusviruses, topology tree. unique use weakly "tombus-like -RNA A). Acids Arch B). Classes Data Evolution Furthermore, HVT Nucleic OTU-like RdRps, SJR-CP, Structure acquisition affinity alignments alignments. amino and, and-RNA apparent bootstrap clearly compatible containing core correspond database detection did does dramatically enzyme; exchange genomes, greater green hierarchical highly implies independently metaviromics, method monophyly myco., mycoviruses; numerous obtained occurred originally parallel potyviruses prevalence proteins, ranges results reverse segments selected shows terms totiviruses transcription unclassified. virions; virosphere viruses; yellow "ourmiaviruses" (the (with Bioinformatics Bunyavirales CapEs Curr Gorbalenya Jiang Multiple Nature Nevertheless, Plant RdRp, Sabanadzovic Shi analyzed ancestors architectures assembled assigned capture cellular characterized chymotrypsin-like class classified color colored community comparisons connections consist constructed construction corresponding database. details). encompasses equal eukaryotes, expansion expansive extracted families. gained genes, genes. genus high hmm holobiont horizontal hues hypoviruses inferred iterative leaves membrane metaviromics mitochondria model module. nature nd-order nodaviruses, non-structural notable organization origins patterns plants, plants. positive-strand potential previous previously protists provenance radiation reoviruses reveal role root second sequences. serine short shuffling significant sister site smaller study supergroup, taxon text trend universal unrelated virions viromes "mitoviruses" "narnaviruses" (>;.), (>;.). (Figure (black Bamford C). CLANS CLANS. Castón Different Direct Distinct GP, Given Grimes Holmes Identification Indeed, Kuhn Lin MPs) MT, Opin Palm PhyML Picornavirales Ppro, RAxML S). Spro, Stuart Suzuki Taxonomy Thus, VP, Viral Wolf Y-Z addition addition, advances affiliated alternative analogous analysis, approach aspects assemblage assembly based belong bound branches. bulk circles combines comparison connect consisting contained corresponds data denoted depth derived difference discussed distances divergence elements. emergence entire enzymes eukaryotes. evidence fCP, function functions gamma-distributed generally generated glycoprotein; groups. helicase highest important indicates inside instance, introns invertebrates, land larger, largest limited longer lost manually matrix modular names negative-sense network, occasions, organized outer overall papain-like partitivirus-picobirnavirus performed pervasive placed position positive-sense presence primary profile program proposed protease protist rates relative remain remained removed representatives. represents respectively) retrotransposons rod-shaped separate sequencing shorter simple sites size statistically striking subsequent subset substantial suggest supergroup", time tombusviruses uncertain vOTU, variety virome. viruses) yielded