The purpose of this lab is to configure and test a IDaaS SAML Identity
Provider. Students will configure a IDaaS based SAML Identity Provider
(in this case OKTA) and import and bind to a SAML Service Provider and
test IdP-Initiated and SP-Initiated SAML Federation.
Lab 2 Tasks:
TASK 1: Sign Up for OKTA Developer Account
Refer to the instructions and screen shots below:
Note: The following steps provide instruction for setting up an OKTA developer account.
If you already have one, you may elect to use that account. Understand, however, that the
instructions below may need to be modified to match your environment.
|
Sign Up for an OKTA developer account by navigating to:
https://developer.okta.com/signup/ and using a VALID email and click Get Started
Upon registration, you will be directed to a hyperlink (hostname) for your developer
account. This link should be saved for future use.
Additional instructions will be sent to the email address provided during account setup.
|
|
Following the instructions received from the generated email, sign into the OKTA
development environment with your provided, temporary password.
|
|
- Enter a New Password and the Repeat New Password
- Use the drop down to select a Forgot Password Question and provide the Answer
- Click a Security Image
- Click Create My Account
|
|
TASK 2: OKTA Classic UI
Refer to the instructions and screen shots below:
For the purposes of the lab and SAML development, we will be using the OKTA Classic UI
which provides access to SAML configurations. (Note: At lab publication, the Developer
Console did not have SAML resources.)
In the top, left hand corner click the <> & select Classic UI from the drop down.
|
|
TASK 3: Enable OKTA Multi-Factor Authentication [OPTIONAL]
Refer to the instructions and screen shots below. This task will require a mobile app to enable a second factor.
[OPTIONAL]
Note: Enabling MFA will require a Smart Device with the appropriate OKTA client for your OS
The step can be skipped if you prefer to just use UserID/Password
- Click Security from the top navigation, then click Multifactor
|
|
[OPTIONAL]
- Under OKTA Verify, change the dropdown from Inactive to Active
- Click the Edit button next to *OKTA Verify Settings
|
|
[OPTIONAL]
- Check Enable Push Verification
- Check Require TouchID for OKTA Verify (optional)
- Click Save
|
|
TASK 4: Build SAML Application - OKTA
Refer to the instructions and screen shots below:
In the main menu, click Applications, and Applications from the dropdown in the
top navigation.
|
|
- Click Add Application in the Applications dialogue window.
|
|
- Click Create New App in the Add Application Menu
|
|
In the Create a New Application Integration dialogue box, select Web from the
drop down for Platform.
Select the SAML 2.0 radio button for Sign on Method and click Create.
|
|
- In the Create SAML Integration screen, enter app.f5demo.com for the App Name.
- Leave all other values as default and click Next.
|
|
- In the Create SAML Integration screen, enter the following values
- In the SAML Setting section
- Single Sign on URL: https://app.f5demo.com/saml/sp/profile/post/acs
- Audience URI (SP Entity ID): https://app.f5demo.com
- Leave all other values as default and click Next.
|
|
In the Create SAML Integration screen, select the:
“I’m an OKTA customer adding an internal app” radio button for
Are you a customer or partner?
In the resulting expanded window, select:
“This is an internal app that we have created” for App Type
and click Finish.
|
|
In the resulting application screen for app.f5demo.com, navigate to the
SAML 2.0 section.
Right Click the Identity Provider Metadata hyperlink and click Save Link As …
Save the metadata.xml to your jumphost desktop. We will be using it in a later step
in the Lab.
|
|
TASK 5: Add User to SAML Application
Refer to the instructions and screen shots below:
Within the app.f5demo.com application screen, Click Assignments then Assign
and then Assign to People from the dropdown.
|
|
In the Assign app.f5demo.com to People dialogue box, select your User ID, click
Assign, then Done.
|
|
- Click Save and Go Back.
|
|
- Click Done.
|
|
TASK 6: Add Multi-Factor Authentication Sign-On Policy [OPTIONAL]
Refer to the instructions and screen shots below. This section requires that Task 3 be completed.
[OPTIONAL]
- Within the app.f5demo.com application screen, Click Sign On
|
|
[OPTIONAL]
- Scroll down to the Sign On Policy section and click Add Rule
|
|
[OPTIONAL]
- In the Add Sign On Rule dialogue box, enter MFA for the Rule Name.
- Scroll down to the Actions section.
- In the Actions section, under Access, check the box for Prompt for factor.
- Ensure Every Sign On radio button is selected.
- Click Save.
|
|
TASK 7: Create the External IDP Connector
Refer to the instructions and screen shots below:
Login to your lab provided Virtual Edition BIG-IP
Begin by selecting: Access -> Federation -> SAML Service Provider ->
External IdP Connectors.
|
|
In the External IdP Connectors screen, click the downward arrow next to the word
Create on the Create button (right side)
Select From Metadata from the drop down menu
|
|
In the Create New SAML IdP Connector dialogue box, use the Browse button to
select the metadata.xml from the desktop (created in Task 4).
Name the Identity Provider Name: OKTA_SaaS-iDP.
Click OK.
|
|
TASK 8: Change the SAML SP Binding
Refer to the instructions and screen shots below:
Begin by selecting: Access -> Federation -> SAML Service Provider ->
Local SP Services
Select the checkbox next to app.f5demo.com and click Bind\UnBind IdP Connectors
|
|
- Check the existing binding and click Delete.
|
|
- Click Add New Row and use the following values
- SAML IdP Connectors: /Common/OKTA_SaaS-iDP
- Matching Source: %{session.server.landinguri}
- Matching Value: /*
- Click Update then OK.
|
|
TASK 9: Apply Access Policy Changes
Refer to the instructions and screen shots below:
- Click the Apply Access Policy link in the top left corner of the Admin GUI
|
|
- Ensure app.f5demo.com-policy is checked and click Apply
|
|


Alicia Fox finally goes public with the issue that has affected her WWE career
by Lennard SurraoA lot has been said about Alicia Fox's WWE status and the problems that she has been facing in the past few months, and now, the former Divas Champion has finally addressed her issues with alcoholism in a lengthy Instagram post.
Fox revealed she had come to grips with her difficulties with alcohol and that she was grateful to be a few months sober.
Fox shared her experience, strength and hope to all the recovering alcoholics that are currently battling their own demons. The WWE veteran revealed that she had learned new skills to cope with her struggles and was thankful that she could finally get this message off her chest.
Here's what Alicia Fox wrote in a very positive Instagram post:
For a WHILE I thought I could avoid addressing my #recovery of alcohol. I am human, and realized it was time to be #transparent and #honest w you #socialmedia now written on #cyberconcrete 👀💕 TODAY, I'm a few months #sober, I’ve creatively navigated my emotions, snagged some new skills AND I WANT TO HELP!!!
I've learned that alcoholism is life threatening situation and support is available!! I figured I can share my insight and BE of SERVICE to anyone finding it hard to believe that ‘YOU ARE IMPORTANT’ and deserve an opportunity to learn how to live a clean and sober life... it would be an honor to share my #hope, #strength, and #experience 🙌🏾✨ SNAIL MAIL ME at @resilienceamongmankind 3948 3rd Street South suite #442; Jacksonville Beach 32250 🌻💕
I’m inviting everyone to a safe social place to discuss and expand understanding of our transforming minds 🙏✨ #fancyasfox 🦊#foxyladies are in #recovery too!! Ok! Thx for allowing me get that off my chest!! 🐏 💨✨#rambrand #resilienceamongmankind V. III. x.
Alicia Fox has had a tough year. The Superstar reportedly arrived drunk and was allowed to wrestle at a live event in February. Arn Anderson, who was the producer of her match, was shown the door for permitting her to compete while being intoxicated.
She wrestled her last match on the Main Event taping in April and has since been moved to the alumni section of the WWE website after she was spotted with the WWE Legends during the RAW Reunion show.
It has been widely reported that the 33-year-old's in-ring career could be over. It was also reported that the company reached out and tried to help her deal with her alcoholism.
Reports stated that she went AWOL when WWE first tried to provide her with the opportunity to get her life back on track. However, it seems Fox accepted the help that was required and is now in recovery.
We're glad that Fox has taken the steps to come out of the deadly clutches of alcoholism and convey all the good wishes as she continues to be on the path of sobriety.
Now you can rate RAW, SmackDown and NXT matches on Sportskeeda !


o_O overview
1. Simple Binding
2. Auto updating
ms since 1st Jan 1970:
3. Binding to inputs
(try entering new name)
4. Checkbox
5. Select
6. Complex dependencies
Lists


Before Change
if not os.path.exists(args.embedding_path):
print("word embedding model file was not found")
exit()
if args.restore is not None and not os.path.exists(args.restore):
print("restore model file was not found")
exit()
After Change
if args.embedding_path is not None and not os.path.exists(args.embedding_path):
print("word embedding model file was not found")
sys.exit(0)
if args.restore is not None and not os.path.exists(args.restore):
print("restore model file was not found")
sys.exit(0)
dataset = SNIPS(sentence_length=args.sentence_length,


Ace Illusions Customs
-How our builds are made-
-Services-
Interior
Our interior designers have a wide selection of world's finest quality materials as well all the latest technology.
Exterior
Vehicle Modifications give a new lease of life, whether it's updating an older model or a brand new car, it creates something completely unique.
Bespoke
If you can't find what you want, our designers can build you a body kit from the ground up.


W. Wayt Gibbs
"When computers are embedded in light switches, you've got to get the software right the first time because you're not going to have a chance to update it" [credited to Mary M Shaw]
"The amount of code in most consumer products is doubling every two years" [credited to Remi H. Bourgonjon]
Unread


Todd takes lead at RSM Classic, goes for 3rd straight win
by EndPlayST. SIMONS ISLAND, Ga. (AP) - Brendon Todd made six birdies on the front nine to seize control and shot 8-under 62 on Saturday at Sea Island to take a two-shot lead into the final round of the RSM Classic as he goes for his third straight PGA Tour victory.
Todd, who overcame a case of the full yips that nearly drove him from the game, is coming off victories in the inaugural Bermuda Championship and the Mayakoba Golf Classic.
He was at 18-under 194, two shots ahead of Webb Simpson (63) and Sebastian Munoz (66).
Tyler Duncan, who started with a two-shot lead, made 18 pars for a 70 and fell four shots behind.
Todd is trying to become the first player since Dustin Johnson in 2017 to win three consecutive starts.
___
For more AP golf coverage: https://apnews.com/apf-Golf and https://twitter.com/AP_Sports


| T |
| public nothrow | WeakCount() |
| public nothrow | WeakCount(WeakCount&& that) |
| public nothrow | WeakCount(const SharedCount<T>& that) |
| public nothrow | WeakCount(const WeakCount& that) |
| public | ~WeakCount() |
| public nothrow Counter<T>* | GetCounter() const |
| public nothrow int | GetUseCount() const |
| public nothrow void | Swap(WeakCount& that) |
| public nothrow void | operator=(WeakCount&& that) |
| public nothrow void | operator=(const SharedCount<T>& that) |
| public nothrow void | operator=(const WeakCount& that) |
| private typedef CounterPtrType | Counter<T>* |
| private typedef Self | WeakCount |
| private Counter<T>* | counter |


0001 function [ridge_markers] = PSzz_components_to_markers(ridge_components,input_image);
0002
0003
0004
0005
0006
0007
0008
0009 ridge_markers = zeros(size(input_image));
0010
0011
0012 ridge_components = keep_points(ridge_components,find(ridge_components.lst>4));
0013 for k=1:length(ridge_components.lstrings),
0014 mean_ener(k) = mean(ridge_components.attribs{k}.ener);
0015 mean_scale(k)= mean(ridge_components.attribs{k}.scl);
0016 end
0017 ridge_components = keep_points(ridge_components,find((mean_ener>.1)&(mean_scale>2)));
0018
0019
0020 for k= 1:length(ridge_components.lstrings),
0021 ridge_markers(ridge_components.lstrings{k}(3:end-2)) = k;
0022 end


The guiding principle of my life is to have passion for my work. I strongly believe in undertaking activities that I enjoy, so that I can channelise all my energies in one direction and persevere in a dedicated manner.
I believe in a life by choice, not by chance.
I describe myself as an initiator and a dynamic person with a great curiosity for all those things which revolve around new technologies. My major area of interest is centred around Android/iOS Application Development. I have equal interest in backend development and UI/UX.
Also, I love to explore new technology and learn new things. I am looking for an exciting opportunity to hone my skills in a dynamic organization where I can contribute towards developing something which makes a difference to the society.
Selected in CERN’s top 40 Openlab summer intern program amongst 3000 applicants around the globe.


Before Change
self._loop.run_until_complete(self._attempt_connection())
self._loop.close()
self.unregister()
threading.Thread(target=connection_thread, daemon=False).start()
@_assert_singleton_async
async def _attempt_connection(self):
Tries to establish a connection to the proxy (launching the
After Change
self._loop.close()
"""
self.unregister()
t = threading.Thread(target=connection_thread)
t.daemon = False
t.start()
"""
@_assert_singleton_async
async def _attempt_connection(self):
Tries to establish a connection to the proxy (launching the


Hack The Solution is a student-run virtual event consisting of a 36-hour hackathon with webinars and workshops by GDEs and professional experts and fun challenges. It will bring student developers to compete and develop creative solutions to real world problems based on United Nations’ Sustainable goals. The hackathon is designed to solve local community problems, spark innovation, attract and educate new talent in accordance with the 2021 Solution Challenge.
Whether you’re writing your first lines of HTML, building an ML app or a full-stack project, Hack the solution is just the place for you. We will have workshops and mentors to help make your idea happen. You will be able to exhibit your skills without any worries. We will take care of everything :)
Interested in becoming a sponsor? Email us at<EMAIL_ADDRESS>At Developer Student Clubs, we strive to bring together developers to participate in the hackathon irrespective of gender, race, sexual orientation, socioeconomic background or ethnicity. We usually try to keep things simple. Read our FAQs to phase out ambiguity.
Did we miss something?


| line {stats} | R Documentation |
Fit a line robustly as recommended in Exploratory Data Analysis.
Currently the initial median-median line is not iterated (as opposed to Tukey's “resistant line” in the references).
line(x, y, iter = 1)
x, y |
the arguments can be any way of specifying x-y pairs. See
|
iter |
positive integer specifying the number of
“polishing” iterations. Note that this was hard coded to
|
Cases with missing values are omitted.
Contrary to the references where the data is split in three (almost)
equally sized groups with symmetric sizes depending on n and
n %% 3
and computes medians inside each group, the
line()
code splits into three groups using all observations
with x[.] <= q1
and x[.] >= q2
, where q1, q2
are
(a kind of) quantiles for probabilities p = 1/3 and p = 2/3
of the form (x[j1]+x[j2])/2
where j1 = floor(p*(n-1))
and j2 = ceiling(p(n-1))
, n = length(x)
.
Long vectors are not supported yet.
An object of class "tukeyline"
.
Methods are available for the generic functions coef
,
residuals
, fitted
, and print
.
Tukey, J. W. (1977). Exploratory Data Analysis, Reading Massachusetts: Addison-Wesley.
Velleman, P. F. and Hoaglin, D. C. (1981). Applications, Basics and Computing of Exploratory Data Analysis, Duxbury Press. Chapter 5.
Emerson, J. D. and Hoaglin, D. C. (1983). Resistant Lines for y versus x. Chapter 5 of Understanding Robust and Exploratory Data Analysis, eds. David C. Hoaglin, Frederick Mosteller and John W. Tukey. Wiley.
Iain M. Johnstone and Paul F. Velleman (1985). The Resistant Line and Related Regression Methods. Journal of the American Statistical Association, 80, 1041–1054. doi: 10.2307/2288572.
lm
.
There are alternatives for robust linear regression more robust and
more (statistically) efficient,
see rlm()
from MASS, or
lmrob()
from robustbase.
require(graphics) plot(cars) (z <- line(cars)) abline(coef(z)) ## Tukey-Anscombe Plot : plot(residuals(z) ~ fitted(z), main = deparse(z$call)) ## Andrew Siegel's pathological 9-point data, y-values multiplied by 3: d.AS <- data.frame(x = c(-4:3, 12), y = 3*c(rep(0,6), -5, 5, 1)) cAS <- with(d.AS, t(sapply(1:10, function(it) line(x,y, iter=it)$coefficients))) dimnames(cAS) <- list(paste("it =", format(1:10)), c("intercept", "slope")) cAS ## iterations started to oscillate, repeating iteration 7,8 indefinitely


Family: Louis Dantès (father)
Occupation: Sailor
A young sailor on the Pharaon from Marseilles. His employer is Pierre Morrel, owner of the firm Morrel & Son. Following his promotion to Captain, he is betrayed by his crewmate Danglars and his friends Fernand Mondego and Gaspard Caderousse who sent an anonymous tip of a Bonapartist plot to Gérard de Villefort, the public prosecutor. Villefort sentences Edmond to a lifetime in prison without parole at the Château d'If. There, he befriends Abbé Faria. He suffers in prison for 13 years before escaping.
Upon escaping, he uncovers the great treasure left for him by Faria. With his newfound wealth, Edmond assumes the persona of The Count of Monte Cristo and travels to Paris to seek revenge.
 He bequeathed his fortune to Maximilien Morrel and Valentine de Villefort.


TecFit allows you to improve your bodybuilding
performance!
Create your personalized schedule - Improve the execution of the exercises - Choose the right diet
ACHIEVE YOUR GOALS
BY GIVING YOUR MAXIMUM DURING
TRAINING THANKS TO A HEALTHY DIET!


Welcome to Kavindu Gems
Best Gem dealer in Rathnapura Sri Lanka
Front time immemorial gems hav e been used by people in different ways. It has curativ e power as well as the power to pour in fortune and drive away the ev il of our planetary position. Certain gems and stones hav e magical powers of healing and fortune giv ing. Gems hav e made life happier to many indiv iduals.
These are widely used to get benefit from the planets as these I ittle gems cony nev erending source of rays of the planet and continue ev en after years. As per astrology and numerology gems counteract the bad effects of planets end gores luck, fortune, prosperity, status, knowledge, concentration, good health, name and fame.
Facebook Page
How we are?
This is why we are the best in the best Gem sellers in Sri Lanka
Service
Make a lasting first impression... Beauty is an art...ev en the most undesirable object can be conv erted toe piece of glamour, if we cam to dig deep, right into the core.
Profile
Make a lasting first impression... Beauty is an art...ev en the most undesirable object can be converted too piece of glamour, if we care to dig deep, sight into the core.
About Gems
Make a lasting first impression... Beauty is an an.. ev en the most undesirable object con be converted ton piece of glamour
Minutes
Customers
Gems
Awwards


Description
In practice, not all architectures and compilers can be made to support
all of the interlocked portion of this library. While many common systems
may support all of the functionality described in documentation, others
may support none, or a certain subset. Because of this, it is important
to be able to check at preprocessor time if certain functionality is supported,
allowing the implementation to branch off accordingly. This can be accomplished
by using the BOOST_ACT_INTERLOCKED_HAS
macro.
Notation
-
name
-
The name of an
Interlocked Operation
inBoost.Act
. -
S
-
An
Interlocked Semantic Type
, as specified without qualification and with the exact name as listed in documentation (not a typedef name). As an exception,default_
,default
, and_
may each be used when specifying default semantics.
Expression Requirements
Table 10. Interlocked Operation Requirements
|
Expression |
Pre/Post-Condition |
|---|---|
|
|
Yields the literal |
Headers
// Brings in the `BOOST_ACT_INTERLOCKED_HAS` macro. #include <boost/act/config/interlocked/has.hpp> // Note: No semantic headers have to be included in order to use // BOOST_ACT_INTERLOCKED_HAS as it relies entirely on the preprocessor.
Example
ToDo: Put a code example here.
In a future release, another macro, BOOST_ACT_INTERLOCKED_HAS_LOCK_FREE
will likely be introduced, allowing one to check if an operation is
supported natively without locking as opposed to being simulated with
locks.


ARTICLE V [OLD]
PROTECTION OF PERSONS UNDER DISABILITY AND
THEIR PROPERTY--REPEALED
PART 1. [OLD] GENERAL PROVISIONS--REPEALED
§§560:5-101 to 560:5-105 REPEALED. L 2004, c 161, §37.


VeriFast Help > Glossary
Term
Symbolic store
Definition
A function mapping local variables to their symbolic values. The symbolic value of a variable is a first-order term.
Shown in the Locals pane in the top right corner of the VeriFast window.
Notes
- To show the symbolic store at a given symbolic execution step, select the step in the Steps pane.
- An entry "x=t" in the symbolic store indicates that the local variable x has symbolic value t.
- The initial value of a variable or function parameter "x" is the first-order variable "x".
- The local variable "currentThread" is a ghost variable that represents the id of the current thread.
- The path condition, the symbolic store, and the symbolic heap together constitute the symbolic state of the symbolic execution.


Java Runtime Environment Download
Download Java Runtime Environment 7 32 Bit For Pc
Java Runtime Environment 7 8 9 10 11 Jre Free Download
Java Runtime Environment 7 Update 45 Neowin
Java Runtime Environment Version 8 0 Update 181 Appdetails
How To Download And Install Java Jre Java Runtime Environment On Windows 10 Youtube
Java Runtime Environment 1 1 7 0 Download Free Leshara
Java 6 Update 29 Offline Installer Omahaeasysite
Java Se Runtime Environment Download Windows 7 Comphedenabri Blogcu Com
Java Runtime Environment Jre 64 Bit Free Download
Java Jre 8u281 Download For Windows
Java Runtime Environment Jre X86
Download Java Runtime Environment 64 Bit Majorgeeks
Java Download Install Jdk 1 7 On Windows Codenotfound Com
Sql Server 2017 Oracle Jre 7 Update 51 64 Bit Or Higher Is Required For Polybase Sqlterritory Com
Java Runtime Environment Jre 64 Bit Setup For Windows 10 8 7 Get Pc Apps
Java Control Panel
Sql Server 2019 Oracle Jre 7 Update 51 64 Bit Or Higher Is Required Sql Authority With Pinal Dave
Java Runtime Environment 7 Update 40 Download
Free Full Java Offline For Windows 32 Bit Java Free Download For Windows 10 64 Bit Offline Installer Latest You Con Download This Pc Salawase
40501 Sas 9 2 Requires Jre 1 6 0 14 32 Bit On Microsoft Windows 7 System And Jre 1 6 0 23 32 Bit On Microsoft Windows Server 2008 R2 System
Java 1 8 Download Deltabrown
Java Jre Latest Version Download Install 32 Bit 64 Bit
Download Java Runtime Environment 1 5 0 11 For Pc
Applets And Java Web Start Applications Troubleshooting Guide For Java Se 7 Desktop Technologies
Java Runtime Environment 7 0 Update 7 Final Windows 32 Bit Free Download 32 Bit Java Environment
Java Runtime Environment For Mac Download 2021 Latest Version
Java Download Version 6 Update 37


Mu ji zhe \ Who Killed Cock Robin?
Mu ji zhe is a movie starring Wei-Ning Hsu, Kaiser Chuang, and Chia-Yen Ko. The story about an ambitious journalist who eagerly pursues a long-forgotten accident. When the sole survivor of the accident suddenly disappears, he...


What can I say? If our story was made into a movie, it would be a rather short one. Short and sweet. We both felt that if we were to date it should lead somewhere with the goal of marriage. We didn’t want to be drifting off into nowhere wondering what’s next. So with that in mind we wanted to be intentional and purposeful. The word love doesn’t even make sense without God, and our relationship could have never survived without our relationship with Christ. Annalynn and I are two broken people. Yes, we are broken. If we waited to be perfect before we started dating…well, you wouldn’t be reading this right now. The grace of God abounds in every step we take, so we don’t have to be perfect, Christ is. We have made our first relationship with Christ and that is our foundation!
The first time I met Lui was after church one evening. I remember thinking he was pretty handsome! He introduced himself....and compared himself to a coconut! His words were, "I'm brown on the outside, but white on the inside!" We clicked right away and I remember thinking, "I better leave before he thinks I'm flirting with him." (Which of course I was!) For a couple weeks, my parents called him "Coconut" until we officially started dating. Lui and I talked more during a church service project while we scraped old paint off a house. He was easy to talk to, open with his life, and it didn't even bother me that his paint scraps flew in my hair! On our first date, Lui took me on a walk at the Katy Trail and told me that he wanted to date with a purpose. Our second date was not as romantic but it took most of the day because we both went to a defensive driving class for speeding tickets! Since then, we have learned that pretty much anything from grocery shopping to going to a nice restaurant can be a fun as long as we're together. I met Lui's entire family on Georgina's, Lui's sister, birthday. Talk about trying to make a good impression! They made me feel welcomed and part of the family immediately! When Lui met my family, he and my dad walked for 7 miles (3 hours) just talking. I knew that if that didn't scare him off, he was keeper. What I respect about Lui is that he never played with my heart or kept me guessing about his intentions. Lui asked my dad for my hand in marriage on New Year's Eve. On January 2nd, 2015, Lui took me to a "work meeting to meet his boss". When I opened the door, I walked into a room covered in hearts and flowers with chocolate covered strawberries and champagne! Lui got down on one knee and asked me to marry him! I said "Yes!" about 10 times to make sure he heard me! I am so happy that I get to be his wife! We are so thankful for our church, friends, and family that have encouraged us and given us wisdom along the way! Thank you for your continued prayers as we join our lives together before God.


Loading...
Info
Value
Owner
Beneficiary
Start Date
Cliff
End Date
Total Vesting
Released
Releasable
Revocable


Syntax:
loop-finish <no arguments> =>|
Description:
The loop-finish macro can be used lexically within an extended loop form to terminate that form ``normally.'' That is, it transfers control to the loop epilogue of the lexically innermost extended loop form. This permits execution of any finally clause (for effect) and the return of any accumulated result.
Examples:
;; Terminate the loop, but return the accumulated count. (loop for i in '(1 2 3 stop-here 4 5 6) when (symbolp i) do (loop-finish) count i) => 3 ;; The preceding loop is equivalent to: (loop for i in '(1 2 3 stop-here 4 5 6) until (symbolp i) count i) => 3 ;; While LOOP-FINISH can be used can be used in a variety of ;; situations it is really most needed in a situation where a need ;; to exit is detected at other than the loop's `top level' ;; (where UNTIL or WHEN often work just as well), or where some ;; computation must occur between the point where a need to exit is ;; detected and the point where the exit actually occurs. For example: (defun tokenize-sentence (string) (macrolet ((add-word (wvar svar) `(when ,wvar (push (coerce (nreverse ,wvar) 'string) ,svar) (setq ,wvar nil)))) (loop with word = '() and sentence = '() and endpos = nil for i below (length string) do (let ((char (aref string i))) (case char (#\Space (add-word word sentence)) (#\. (setq endpos (1+ i)) (loop-finish)) (otherwise (push char word)))) finally (add-word word sentence) (return (values (nreverse sentence) endpos))))) => TOKENIZE-SENTENCE (tokenize-sentence "this is a sentence. this is another sentence.") => ("this" "is" "a" "sentence"), 19 (tokenize-sentence "this is a sentence") => ("this" "is" "a" "sentence"), NIL
Side Effects:
Transfers control.
Affected By: None.
Exceptional Situations:
Whether or not loop-finish is fbound in the global environment is implementation-dependent; however, the restrictions on redefinition and shadowing of loop-finish are the same as for symbols in the COMMON-LISP package which are fbound in the global environment. The consequences of attempting to use loop-finish outside of loop are undefined.
See Also:
loop, Section 6.1 (The LOOP Facility)
Notes:


fun getSelectedChildrens(): Map<Parent, Set<Child>>
Get the map of the currently selected ParentItem to Set of ChildItem
Return The map of the currently selected ParentItem to Set of ChildItem


THE PROPOSITION
Reviewed by Harvey Karten, Ph.D. PolyGram Filmed Entertainment/ Interscope Director: Lesli Linka Glatter Writer: Rick Ramage Cast: Kenneth Branagh, Madeleine Stowe, William Hurt, Neil Patrick Harris, Robert Loggia, Josef Sommer, Blythe Danner, David Byrd
Not all propositions are indecent proposals, but cinema today has less use than ever for affairs freely entered between consenting adults, the sorts of liaisons that do not hurt anyone. In "Indecent Proposal" (1993) Robert Redford proposed to give one million dollars for a night to remember with Demi Moore and the audience wondered why HE would have to pay HER. In "The Proposition," the issue--one which would be considered vulgar by some, respectable by others--is more complex and with a better script could have been more involving. Arthur Barret (William Hurt), a fabulously wealthy lawyer, is sterile, unable to present the wife he adores, Eleanor (Madeleine Stowe) with the child she so intensely craves. Through an intermediary, Hannibal Thurman (Robert Loggia), Arthur hires a bright young stud, Roger Martin (Neil Patrick Harris), offering $25,000 if he would spend a night with his wife so that she would have what she so desperately wants. Would you believe that he at first refuses, even after seeing the beautiful Eleanor? But hormones triumph over ethics, and when the youthful Harvard graduate falls in love with his conquest in just one night threatening to expose his benefactor for the pimp he considers him to be, the boy must be eliminated.
"The Proposition" is an uneasy mix of soap opera and feminism with an occasional burst of clever humor. Situated in the Boston of 1935, the tale is told in flashback by Father Michael McKinnon (Kenneth Branagh), whose role in the melodramatic yarn is not entirely priestly. In typical potboiler mode, the story focuses on the growing affection between Eleanor and the priest who turns out to be the nephew of the affluent Arthur, a man who entered the priesthood to get back at his father--who was materialistic almost to the point of treason. Suggesting Kennedy-style with specific reference to Joseph Kennedy, Lesli Linka Glatter's production displays the pomp and palatial splendor of the Barret estate, complete with a head servant, Syril (Blythe Danner), who is dressed as elegantly as the woman of the house and who has more than employee-employer feelings for her boss, adding to the class politics of the drama.
For feminist resonance, Rick Ramage's screenplay portrays Eleanor as a novelist in the style of Virginia Woolf, a woman who despite her love for her husband has a separate room-- all the more convenient for both the welcome and the unwelcome trysts. Despite some droll moments, particularly in the lovemaking scene between the callow Roger and the sophisticated and understanding beauty, the performers take themselves entirely too seriously. Though he hired the young man to perform studly services, Barret conspicuously scowls from his car at the Don Juan has created and later admits that he hated the boy. Even worse, Father Michael is insufferably solemn throughout, displaying his displeasure at the follies of moneyed aristocrats, particularly when they invest their money in Germany at a time that Americans are out of work in appalling numbers. Despite his rugged good looks, you wonder how Eleanor could become so animated over this priggish man of the cloth whose hormones do battle with his oath to the church, the victor never in doubt.
David Brisbin's production design is impressive, featuring the stately autos of the 1930s and Anna B. Sheppard's costume design seems right on the money. But the drama becomes too fogged by suds to offer the clarity of a well- developed fable. Rated R. Running time: 114 minutes. (C) Harvey Karten 1998
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


A potentiometer is a variable resistor that is typically used on a robot to measure shaft angle or number of turns of a shaft. The potentiometer has a resistance that is proportional to the angle of the shaft on the device. There are several styles of potentiometers available. The potentiometers used on robots should be linear taper (not logarithmic as used in audio equipment for volume controls).
Typically a potentiometer shaft can rotate about 270 degrees before hitting the hard stop. It is important to make sure that the mechanical system the potentiometer is connected to doesn't exceed the maximum rotation of the potentiometer or it will likely become damaged. For applications requiring more rotations, multi-turn potentiometers are available. Using a 10-turn potentiometer often simplifies the mechanical design of a system such as an elevator where the rotations translate into elevator height.


Florian HuetiOS Developer - Game Developer - Game designer
Examples
Game engine - Unreality3D
Unreality3D is a six person project, the purpose was to create a GameEngine that allows us to
do a video game project being a part of our school program. You can find the GameEngine
here on GitHub. We managed to create a Gauntlet with it, you can find some footage on the Facebook page
School project. The goal was to make a game using C++ and Ogre3D, for that we developped
our own game engine Unreality3D. We made a Gauntlet
like game, 4 players over network, 4 classes and proceduraly generated dungeons.
Toolbox for unity3d. Contains homemade
2d methods, 2d level generation/drawing plugin and a 2d dungeon generator using pcg. I use it during game jams, makes me save a lot of time. Work still in progress ...
School project. Subject : Making a program that creates synthetic images according to the
raytracing method from a configuration file. Said method of ray tracing is the following
: For each pixel ( image point) , a color is determined according to the intersection between
one of the traced rays and one of the objects in the scene which is observed from a point
thereof . This color is then influenced by the added effects such as lighting , shadows ,
transparency ...
Global Game Jam 2016 Entry, subject : Ritual. QTE based 2 players game. You have
to be faster than your opponent if you want to use your full potential and defeat him.
Link
Mobile game project. Halfway between Tap games and RPG, you incarn Dee who crashed on a
planet and have to survive until he gets rescued. Work in progress ...
Excessive daytime somnolence (EDS) is defined as the inability to stay awake in daily life
activities. Several scales have been used to diagnose excessive daytime sleepiness, the most
widely used being the Epworth Sleepiness Scale (ESS).
Sleep disorders and EDS are very common in the general population. It is therefore important
to be able to screen patients for this symptom in order to obtain an accurate diagnosis of
sleep disorders. Embodied Conversational Agents (ECA) have been used in the field of affective
computing and human interactions but up to now no software has been specifically designed
to investigate sleep disorders.
We created an ECA able to conduct an interview based on the ESS and compared it to an interview conducted by a sleep specialist.
We recruited 32 consecutive patients and a group of 30 healthy volunteers free of any sleep
complaints.
The ESS is a self-administered questionnaire that asks the subject to rate (with a pen and
paper paradigm) his or her probability of falling asleep. For the purpose of our study, the
ECA or real-doctor questionnaire was modified as follows: Instead of the ‘‘I’’ formulate,
questions were asked as ‘‘Do you.’’ Our software is based on a common 3D game engine and
several commercial software libraries. It can run on standard and affordable hardware products.
The sensitivity and specificity of the interview conducted by the ECA were measured. The
best results (sensibility and specificity >98%) were obtained to discriminate the sleepiest
patients (ESS 16) but very good scores (sensibility and specificity >80%) were also obtained
for alert subjects (ESS).
ESS scores obtained in the interview conducted by the physician were significantly correlated
with ess scores obtained in the interview the eca conducted. most of the subjects had a positive
perception of the virtual physician and considered the interview with the eca as a good experience.
sixty-five percent of the participants felt that the virtual doctor could significantly help
real physicians. our results show that a virtual physician can conduct a very simple interview
to evaluate eds with very similar results to those obtained by a questionnaire administered
by a real physician. the expected massive increase in sleep complaints in the near future
likely means that more and more physicians will be looking for computerized systems to help
them to diagnose their patients.
Cognitive Overload is a immersive game, where you must destroy all the oponents that come to you from three different directions.
Left, right and in front of you. As further you go, numerous are the enemies.The
purpose of this game was to be as progressive as possible to mesure the cognitive activity
of the patient. More different actions done at the same time, more the cognitive charge is
high.
The Aniballs like to play hide and seek with their friends, find them all ! Inspired by the famous "Where is Waldo" by Martin
Handford and designed for toddlers and children (2 - 5 years old). AniBall Finding is a game
in which the kids will have to find the hidden animal amongst the others.
Here on PlayStore
My Ludum dare 35 entry. The goal of this game is to find the 3 keys to get out. You
have to gather energy so you can change your shape and fight/escape your foes.
Link


--- layout: archive title: Zulip Chat Archive permalink: /stream/116395-maths/topic/int.2Elt_succ_iff_le.html ---
lemma int.lt_succ_iff_le {m n : ℤ} : m < n + 1 ↔ m ≤ n :=
This doesn't seem to be in mathlib. split; intro; linarith
proves it but slowly...
Is it called int.lt_add_one
?
Aah, nice lt_add_one_iff
I first had the sides swapped, and then library_search
didn't find it. I thought you fixed that @Scott Morrison didn't you?
lemma int.lt_succ_iff_le {m n : ℤ} : m < n + 1 ↔ m ≤ n := begin symmetry, library_search end
Really? Ugh, I should fix that.
ohh... I see, yeah, library search can't do that out of the box. If you split; library_search
it should succeed, however.
Aha, I see
It's a slight hack to do this, but I can add it.
Maybe it can try symmetry
before applying...?
But maybe it costs too much
I mean, symmetry
probably costs exactly one more library_search_core
invocation?
Last updated: Apr 24 2020 at 19:02 UTC


{% extends 'index.html' %} {% block content %}
(Disclaimer: The following are not all inclusive instructions for every navy command and their local instructions take precedent over any information presented here)
Navigating the TA process and completing all of the steps to enroll in college to start your journey through higher education can be intimidating. Hopefully this consolidated list of instructions, links, and accompanying checklist can assist you in getting through the process.
First stop in your journey should be the Navy College website (https://www.navycollege.navy.mil/).
A site you should become intimately familiar with is the MyNavy Education website (https://myeducation.netc.navy.mil/webta/home.html#nbb). This site requires login with either DEERS account information or a CAC. This page shows you your eligibility for TA, as well as being the site you use to submit your electronic TA requests and receive your approved TA vouchers.
Obtaining your Joint Services Transcript (JST) is relatively simple. Head to the site (https://jst.doded.mil/jst/), log in and get an electronic copy of your unofficial transcript. If your school requests an official transcript be sent, you can also conduct that transaction here
If you have CLEP, AP, DSST, SAT, or ACT scores you need to submit, you can access those from the College Board (https://www.collegeboard.org/). If you are interested in taking CLEP or DSST examinations, those are discussed more in depth in the Alternative College Credit section of this website.
| Complete Once During your Navy Career: | |
|---|---|
| TA/NCPACE DL Policy and Procedures Training. | |
| Navy Virtual Counseling 101 training IMI. | |
| Schedule 1st virtual counseling with NCVEC. | |
| Selecting a College and Degree Plan: | |
| Select multiple colleges you are interested in. | |
| Select multiple degree plans from all chosen colleges. | |
| E-Mail advisors from selected schools. Tell them which degree plans you are interested in. | |
| Send your JST and any other transcripts you have to your advisors for evaluation. | |
| Next Steps: | |
| Schedule second counseling with NCVEC to review degree plan. | |
| Enroll with chosen school. | |
| Register for classes. | |
| Route TA request at least 14 days before class start date. | |
| Sign NCVEC approved voucher | |
| Submit approved and signed TA voucher through your school's process | |
| START CLASSES! |


{% extends 'base.html'%} {% import 'macros.html' as macro%} {% import "bootstrap/wtf.html" as wtf %} {% block content %}
Add your Blog here..


Single cell alignment
Table of Contents
Introduction
Single cell alignment is the problem of identifying common biological variation between different single cell measurements. The goal of alignment is to jointly analyze multiple data sets, maximizing power to detect biological differences by increasing sample sizes.
One obvious application of alignment is to perform batch correction, where each data set corresponds to a batch. With the advent of large atlases of single cells (Zheng et al 2016, Human Cell Atlas), a more pressing application is to allow researchers to quickly align novel single cell experiments with millions of previously assayed cells.
Butler et al 2018 propose a two-step approach to solve the single cell alignment problem. First, they project all of the data into a common basis using generalized canonical correlation analysis (Hotelling et al 1936, Kettenring 1971). Second, they align cells across data sets to each other along the canonical correlation vectors using dynamic time warping (Berndt and Clifford 1994). DTW is a special case of sequence alignment (Needleman and Wunsch 1970), where mismatches/indels are not penalized, and the substitution cost is Euclidean distance.
Here, we develop an alternative approach based on adversarial training of a deep generative model. Our main contributions are:
- We propose the negative cross-entropy loss as a quantitative metric for the quality of the alignment. Intuitively, after aligning the data (in low dimensional space), it should not be possible to successfully classify points as coming from different data sets.
- We propose an adversarial auto-encoding architecture (Makhzani et al. 2015) which simultaneously: (1) explains the observed data (by maximizing the evidence lower bound) and removes data-set specific differences (by maximizing the cross-entropy loss), (2) learns a separate generative model of the data to allow incremental alignment of novel datasets without using the original training data.
- We show that our method performs well using the alignment score proposed by Butler et al 2018, and outperforms it in terms of negative cross-entropy loss.
- We show our method is robust to non-overlapping cell subpopulations.
- We demonstrate our method scales by training models for >450,000 human peripheral blood mononuclear cells and >1,000,000 mouse brain cells, and then aligning held out datasets to them. We provide our pre-trained models as the basis for other researchers to align novel experimental data.
Methods
Variational auto-encoder
Suppose we want to fit a latent variable model:
\[ \mathbf{x}_i \mid \mathbf{z}_i, \theta \sim g(\mathbf{z}_i, \theta) \]
\[ \mathbf{z}_i \sim N(0, \sigma^2 \mathbf{I}) \]
where \(\mathbf{x}_i \in \mathbb{R}^p\), \(\mathbf{z}_i \in \mathbb{R}^d\), \(d \ll p\).
Assuming \(g(\mathbf{z_i}) = \mathbf{W z}_i\) and marginalizing over \(\mathbf{z}_i\), we recover PPCA (Tipping 1999).
Here, we instead pursue an approach where \(g\) is parameterized by a neural network (Kingma and Welling 2014, Rezende et al 2014, Titsias and Lázaro-Gredilla 2014). There are several reasons to do this:
- We want to learn a nonlinear embedding of the data. This is achieved by replacing the affine transform \(\mathbf{W Z}\) with a feed forward neural network, which is a recursive generalized linear model (Mohammed 2015).
- We want to operate on data which cannot fit in memory. This is achieved by using an inference network (Dayan et al 1995). The key idea is that we replace the \(n \times d\) matrix \(\mathbf{W}\) and \(d \times p\) matrix \(\mathbf{Z}\) with a neural network. The parameters of the neural network take the place of \(\mathbf{W}\), and the output of the neural network is \(\mathbf{z}_i\). Then, the size of the network is constant in the size of the data, and we can optimize the loss function using stochastic optimization over minibatches of data.
- We want to impose non-trivial constraints on the posterior. In our setting, we assume we have labels \(y_i\), and the goal is to impose the constraint that the points with different labels are not distinguishable from each other. This is not readily expressed as a prior distribution on \((\mathbf{z}_i, y_i)\).
We assume the following generative model:
\[ x_{ij} \mid \lambda_{ij} \sim \mathrm{Poisson}(\lambda_{ij}) \]
\[ \lambda_{ij} \mid z_i \sim [\pi(\mathbf{z}_i)]_j \delta_0(\cdot) + (1 - [\pi(\mathbf{z}_i)]_j)[\mu(\mathbf{z}_i)]_j \]
where \(\pi(\cdot), \mu(\cdot)\) are \(p\)-vector outputs of a neural network, termed the decoder.
This is a zero-inflated Poisson variational autoencoder (ZIPVAE). Previous studies have instead chosen the zero-inflated negative binomial likelihood, yielding a ZINBVAE (Lopez et al 2018, Grønbech et al 2018). This choice assumes unexplained technical heterogeneity beyond Poisson sampling noise, and needs to be justified from the data.
The inference goal of the fitted model is to estimate \(p(\mathbf{z}_i \mid \mathbf{x}_i)\). This posterior is non-conjugate to the likelihood, so we use variational inference to estimate an approximate posterior. Importantly, the model was parameterized using a neural network, so we need to also parameterize the approximate posterior using a neural network.
\[ q(\mathbf{z}_i \mid \mathbf{x}_i) = \mathrm{N}(\mu(\mathbf{x}_i), \sigma^2(\mathbf{x}_i) \mathbf{I}) \]
where \(\mu(\cdot), \sigma^2(\cdot)\) are \(d\)-vector outputs of a neural network, termed the encoder.
The evidence lower bound (ELBO) is:
\[ p(\mathbf{X}) \geq \sum_i \mathbb{E}_q[\ln p(x_i \mid z_i)] - \mathcal{KL}\left(q(z_i \mid x_i) \Vert p(z_i)\right) \]
We use the reparameterization trick to rewrite the expectations over \(q\) as sums over samples from standard Gaussians, yielding a stochastic objective function. We optimize the objective using gradient descent.
The resulting model is known as a variational autoencoder (VAE). In the case where both the encoder and decoder are affine transforms, we recover PPCA (as described above). In the case where the encoder is affine but the decoder is a neural network, we recover Robust PCA (Candés et al. 2009, Dai et al 2017).
Adversarial alignment
We want to enforce the constraint that data which reflect the same underlying biological state are not distinguishable on the basis of which data set they came from. This means the model must simultaneously learn the underlying biological state as well as “forget” systematic differences between different data sets.
To achieve this goal, we observe that the VAE described above learns a non-linear mapping from observed data \(\mathbf{x}_i\) to a low dimensional latent variable \(\mathbf{z}_i\) describing the biological state. We need the encoder to not use systematic differences between data sets in this mapping.
This constraint is not readily expressible as a prior on \(\mathbf{z}_i\), but it is expressible as the opposite of a classification problem on \(\mathbf{z}_i\). In essence, rather than minimizing the cross-entropy loss:
\[ l(\mathbf{z}_i, y_i) = y_i \ln(f(\mathbf{z}_i)) \]
where \(f\) is the classifier, we want to maximize it. This is related to maximum entropy (Jaynes 1957a, Jaynes 1957b) and minimum discrimination information (Kullback and Liebler 1951).
This suggests jointly optimizing a combined loss function:
\[ \mathcal{L} = \sum_i \mathrm{ELBO}(\mathbf{x}_i) - l(\mathbf{z}_i, y_i) \]
There are two major challenge in optimizing this loss function:
- We need to jointly train the classifier \(f\) and the the VAE \(p, q\). In principle, this could be done by simply performing gradient descent on the full objective function wrt. the parameters of \(f, p, q\). This approach is related to adversarial autoencoders (Makhzani et al. 2015), where an adversarial discriminator network is used to enforce constraints on a deep generative model.
- We need to balance the two parts of the objective function. The optimal way to fool the discriminator is to randomly place points in the low dimensional space, but the optimal way to describe the data is to model the systematic technical differences between data sets. In previous work, this was done by training each component alternately.
Incremental training
One further challenge is how to update the model with new data (e.g., a new experimental data set). After successfully fooling the adversary, the resulting low dimensional representation is “one dataset”, reducing the problem into aligning two data sets. However, all of the original data would be required to train the model from scratch.
Instead, our key idea is to add to use a generative adversarial network (GAN; Goodfellow 2014) in the architecture. In a GAN, a generator network \(G\) and discriminator network \(D\) are trained to optimize a minimax game:
\[ \min_G \max_D E_{\hat{p}} [\ln D(\mathbf{x})] + E_p [\ln(1 - D(G(\mathbf{z})))] \]
where \(\hat{p}\) is the empirical distribution of the data, and \(p\) is the distribution of data generated by \(G\). Typically, \(G\) maps isotropic Gaussian noise to generated data points. The inference goal is to learn \(G\), and the adversary \(D\) is used to push it towards the distribution \(\hat{p}\).
Here, we use the encoder network of the VAE as the generator, and use the dataset of origin as the labels for the adversary, yielding a simple adversarial autoencoder (Makhzani et al 2015). We can then train the model in phases in each minibatch:
- Train the encoder and decoder to learn the low dimensional manifold describing the data
- Train the adversary to classify the latent points based on the labels
- Train the encoder to fool the adversary
Simultaneouly, we can train a generator network which can generate latent samples matching the encoder. To train this network, we add a further discriminator network which tries to distinguish real outputs from the encoder network \(q(\mathbf{z} \mid \mathbf{z})\) from the generator outputs.
With the trained generator network, we can incrementally train the model on new data sets without needing the original data.
Results
ZIPVAE sanity check
First, check that the ZIPVAE works. Simulate some low-rank Poisson data, thin the counts to produce training and validation data, and evaluate the validation set log likelihood.
\[ \mathbf{l}_{ik} \sim \mathcal{N}(0, \sigma^2) \] \[ \mathbf{f}_{kj} \sim \mathcal{N}(0, \sigma^2) \] \[ \ln\lambda_{ij} = l_{ik} f_{kj} \] \[ x_{ij} \sim \mathrm{Poisson}(\lambda_{ij}) \] \[ y_{ij} \sim \mathrm{Binomial}(x_{ij}, 0.5) \] \[ \tilde{y}_{ij} = x_{ij} - y_{ij} \]
def simulate_pois(n, p, rank, eta_max=None, seed=0): np.random.seed(seed) l = np.random.normal(size=(n, rank)) f = np.random.normal(size=(rank, p)) eta = l.dot(f) if eta_max is not None: # Scale the maximum value eta *= eta_max / eta.max() x = np.random.poisson(lam=np.exp(eta)) return x, eta def train_test_split(x, p=0.5): train = np.random.binomial(n=x, p=p, size=x.shape) test = x - train return train, test def pois_llik(lam, train, test): lam *= test.sum(axis=0, keepdims=True) / train.sum(axis=0, keepdims=True) return st.poisson(mu=lam).logpmf(test).sum() def generalization_score_oracle(train, test, eta): return pois_llik(np.exp(eta), train, test) def generalization_score_zipvae(train, test, *args): import scaa import torch import torch.utils.data n, p = train.shape training_data = torch.utils.data.DataLoader(torch.tensor(train, dtype=torch.float), batch_size=25, shuffle=False) with torch.cuda.device(0): model = scaa.modules.ZIPVAE(p, 10).fit(training_data, lr=1e-2, max_epochs=10, verbose=False) lam = model.denoise(training_data) return pois_llik(lam, train, test) def evaluate_generalization(num_trials): result = dict() for method in ['oracle', 'zipvae']: result[method] = [] for trial in range(num_trials): x, eta = simulate_pois(n=500, p=1000, rank=3, eta_max=3, seed=trial) train, test = train_test_split(x) score = globals()[f'generalization_score_{method}'](train, test, eta) result[method].append(score) result = pd.DataFrame.from_dict(result) result.index.name = 'trial' return result
<<imports>> <<zipvae-sanity-check>> res = evaluate_generalization(num_trials=100) res.to_csv('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/alignment/generalization.txt.gz', sep='\t', compression='gzip')
sbatch --partition=gpu2 --gres=gpu:1 --time=60:00 #!/bin/bash source activate singlecell python /project2/mstephens/aksarkar/projects/singlecell-ideas/code/zipvae-generalization.py
Submitted batch job 51311689
Read the results.
res = pd.read_table('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/alignment/generalization.txt.gz', index_col=0)
Plot the results.
methods = ['oracle', 'zipvae'] plt.clf() plt.gcf().set_size_inches(3, 3) for j, method in enumerate(methods): y = res[method] f = st.gaussian_kde(y) py = f(y) x = j + .2 / py.max() * np.random.uniform(-py, py) plt.scatter(x, y, c='k', s=4) plt.xticks(range(len(methods)), methods) plt.xlim(-.5, len(methods) - .5) plt.xlabel('Method') _ = plt.ylabel('Validation set log likelihood')
ZIPVAE on known mixture
Embed a known mixture of sorted cells, then look at whether they are linearly separable in the embedding space.
cd8 = scipy.io.mmread('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/10xgenomics/cd8+_cytotoxic_t_cells/filtered_matrices_mex/hg19/matrix.mtx.gz').tocsr() cd19 = scipy.io.mmread('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/10xgenomics/cd19+_b_cells/filtered_matrices_mex/hg19/matrix.mtx.gz').tocsr()
Mix the cells and filter lowly expressed genes.
mix = ss.hstack((cd8, cd19)).tocsr() mix = mix[((mix > 0).mean(axis=1) >= 0.25).A.ravel()].T.astype(np.int) mix.shape
(20294, 404)
Load the data.
training_data = torch.utils.data.DataLoader( scaa.dataset.SparseDataset(mix), batch_size=100, num_workers=1, pin_memory=True, shuffle=True, ) eval_data = torch.utils.data.DataLoader( scaa.dataset.SparseDataset(mix), batch_size=100, num_workers=1, pin_memory=True, shuffle=False, )
Fit the model and get the embedding.
with torch.cuda.device(0): model = scaa.modules.ZIPVAE(mix.shape[1], 10).fit(training_data, max_epochs=2, verbose=True) with torch.set_grad_enabled(False): z = torch.cat([model.encoder.forward(x)[0] for x in eval_data]).numpy()
Label the cells.
y = np.zeros(mix.shape[0]) y[:cd8.shape[1]] = 1 y = y.astype(bool)
Plot UMAP of the latent space.
embed = umap.UMAP(n_neighbors=5, metric='euclidean').fit_transform(z)
plt.clf() plt.gcf().set_size_inches(3, 3) plt.scatter(embed[y,0], embed[y,1], s=1, c='r', label='CD19', alpha=0.25) plt.scatter(embed[~y,0], embed[~y,1], s=1, c='b', label='CD8', alpha=0.25) plt.legend(frameon=False, handletextpad=0, markerscale=4) plt.xlabel('UMAP 1') _ = plt.ylabel('UMAP 2')
Discriminator sanity check
Make sure the discriminator can classify in count space.
cd8 = scipy.io.mmread('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/10xgenomics/cd8+_cytotoxic_t_cells/filtered_matrices_mex/hg19/matrix.mtx.gz').tocsr() cd19 = scipy.io.mmread('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/10xgenomics/cd19+_b_cells/filtered_matrices_mex/hg19/matrix.mtx.gz').tocsr()
Mix the cells and filter lowly expressed genes.
mix = ss.hstack((cd8, cd19)).tocsr() mix = mix[((mix > 0).mean(axis=1) >= 0.25).A.ravel()].T.astype(np.int) mix.shape
(20294, 404)
Label the cells.
y = np.zeros(mix.shape[0]) y[:cd8.shape[1]] = 1 y = y.astype(bool)
Split the cells.
train = np.zeros(mix.shape[0], dtype=np.bool) train[:int(cd8.shape[1] * .9)] = 1 train[cd8.shape[1]:cd8.shape[1] + int((mix.shape[0] - cd8.shape[1]) * .9)] = 1 val = ~train
train[y].sum(), train[~y].sum(), val[y].sum(), val[~y].sum()
(9188, 9076, 1021, 1009)
Train the discriminator.
data = torch.tensor(mix.A[train], dtype=torch.float) val_data = torch.tensor(mix.A[val], dtype=torch.float) labels = torch.tensor(y[train].astype(int), dtype=torch.long) val_labels = torch.tensor(y[val].astype(int), dtype=torch.long) f = scaa.modules.Discriminator(input_dim=mix.shape[1], num_classes=2) opt = torch.optim.Adam(f.parameters()) loss_fn = torch.nn.NLLLoss() for i in range(100): opt.zero_grad() loss = loss_fn(f.forward(data), labels) val_loss = loss_fn(f.forward(val_data), val_labels) print(f'[{i}] {loss} {val_loss}') loss.backward() opt.step()
loss.detach().numpy(), val_loss.detach().numpy()
(array(4.1733852e-05, dtype=float32), array(0.00011137, dtype=float32))
Evaluate the discriminator.
with torch.set_grad_enabled(False): ay = f.forward(val_data).numpy() py = ay / ay.sum() skme.average_precision_score(y[val], py[:,0]), skme.roc_auc_score(y[val], py[:,0])
(1.0, 1.0)
ZIPAAE sanity check
Simulate homogeneous low rank data and randomly split it into two groups (so differences are just sampling variation). The AAE should still be able to accurately model the data.
<<imports>> def evaluate(num_trials): result = [] for trial in range(num_trials): train, test = scaa.benchmark.train_test_split(x) y = (np.random.uniform(size=x.shape[0]) < 0.5).astype(np.int) result.append([trial, scaa.benchmark.generalization_score_oracle(train, test, eta=eta), scaa.benchmark.generalization_score_zipvae(train, test), scaa.benchmark.generalization_score_zipaae(train, test, y=y), ]) result = pd.DataFrame(result) result.columns = ['trial', 'Oracle', 'ZIPVAE', 'ZIPAAE'] return result res = evaluate(num_trials=20) res.to_csv('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/alignment/zipaae-generalization.txt.gz', compression='gzip', sep='\t')
sbatch --partition=gpu2 --gres=gpu:1 --time=60:00 --job-name=zipaae #!/bin/bash source activate singlecell python /project2/mstephens/aksarkar/projects/singlecell-ideas/code/zipaae-generalization.py
Submitted batch job 51428443
Read the results.
res = pd.read_table('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/alignment/zipaae-generalization.txt.gz', index_col=0)
Plot the results.
methods = ['Oracle', 'ZIPVAE', 'ZIPAAE'] plt.clf() plt.gcf().set_size_inches(3, 3) for j, method in enumerate(methods): y = res[method] f = st.gaussian_kde(y) py = f(y) x = j + .2 / py.max() * np.random.uniform(-py, py) plt.scatter(x, y, c='k', s=4) plt.xticks(range(len(methods)), methods) plt.xlim(-.5, len(methods) - .5) plt.xlabel('Method') _ = plt.ylabel('Validation set log likelihood')
ZIPAAE on known mixtures
Randomly split a knowm mixture of sorted cells into two groups, then fit the AAE to embed them.
cd8 = scipy.io.mmread('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/10xgenomics/cd8+_cytotoxic_t_cells/filtered_matrices_mex/hg19/matrix.mtx.gz').tocsr() cd19 = scipy.io.mmread('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/10xgenomics/cd19+_b_cells/filtered_matrices_mex/hg19/matrix.mtx.gz').tocsr()
Mix the cells and filter lowly expressed genes.
mix = ss.hstack((cd8, cd19)).tocsr() mix = mix[((mix > 0).mean(axis=1) >= 0.25).A.ravel()].T.astype(np.int) mix.shape
(20294, 404)
Label the cells.
y = np.zeros(mix.shape[0]) y[:cd8.shape[1]] = 1 y = y.astype(bool)
To make the problem harder, correlate the “batch” label with cell type.
batch = (np.random.uniform(size=mix.shape[0]) < np.where(y, .25, .75)).astype(int)
Embed the data in the ambient space.
embed = umap.UMAP(n_neighbors=5, metric='euclidean').fit_transform(mix)
plt.clf() plt.set_cmap('Dark2') fig, ax = plt.subplots(1, 2) fig.set_size_inches(6, 3) ax[0].scatter(embed[y,0], embed[y,1], s=1, c='C0', label='CD19', alpha=0.25) ax[0].scatter(embed[~y,0], embed[~y,1], s=1, c='C1', label='CD8', alpha=0.25) leg = ax[0].legend(frameon=False, handletextpad=0, markerscale=4) for h in leg.legendHandles: h.set_alpha(1) b = batch.astype(bool) ax[1].scatter(embed[b,0], embed[b,1], s=1, c='C2', label='Batch 1', alpha=0.25) ax[1].scatter(embed[~b,0], embed[~b,1], s=1, c='C3', label='Batch 2', alpha=0.25) leg = ax[1].legend(frameon=False, handletextpad=0, markerscale=4) for h in leg.legendHandles: h.set_alpha(1) for a in ax: a.set_xlabel('UMAP 1') a.set_ylabel('UMAP 2') plt.tight_layout()
Load the data.
training_data = torch.utils.data.DataLoader( scaa.dataset.SparseDataset(mix), batch_size=100, num_workers=4, pin_memory=True, shuffle=True, ) batch_data = torch.utils.data.DataLoader( torch.tensor(batch, dtype=torch.long), batch_size=100, num_workers=4, pin_memory=True, shuffle=True, ) eval_data = torch.utils.data.DataLoader( scaa.dataset.SparseDataset(mix), batch_size=100, num_workers=1, pin_memory=True, shuffle=False, )
Fit the model.
with torch.cuda.device(0): model = scaa.modules.ZIPAAE(input_dim=mix.shape[1], latent_dim=10, num_classes=2).fit(x=training_data, y=batch_data, max_epochs=2, verbose=True)
Get the embedding.
with torch.set_grad_enabled(False): q = model.vae.encoder.cpu() z = torch.cat([q.forward(batch)[0] for batch in eval_data]).numpy()
Plot UMAP of the latent space.
embed = umap.UMAP(n_neighbors=5, metric='euclidean').fit_transform(z)
plt.clf() plt.set_cmap('Dark2') fig, ax = plt.subplots(1, 2) fig.set_size_inches(6, 3) ax[0].scatter(embed[y,0], embed[y,1], s=1, c='C0', label='CD19', alpha=0.25) ax[0].scatter(embed[~y,0], embed[~y,1], s=1, c='C1', label='CD8', alpha=0.25) leg = ax[0].legend(frameon=False, handletextpad=0, markerscale=4) for h in leg.legendHandles: h.set_alpha(1) b = batch.astype(bool) ax[1].scatter(embed[b,0], embed[b,1], s=1, c='C2', label='Batch 1', alpha=0.25) ax[1].scatter(embed[~b,0], embed[~b,1], s=1, c='C3', label='Batch 2', alpha=0.25) leg = ax[1].legend(frameon=False, handletextpad=0, markerscale=4) for h in leg.legendHandles: h.set_alpha(1) for a in ax: a.set_xlabel('UMAP 1') a.set_ylabel('UMAP 2') plt.tight_layout()
Comparison to existing methods
Look at control versus stimulated CD4+ T cells from Kang et al 2018.
Download the data.
mkdir -p GSE96583 cd GSE96583 curl "https://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE96583&format=file" | tar xf - curl "https://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE96583&format=file&file=GSE96583_genes.txt.gz" -o GSE96583_genes.txt.gz
Read the data.
control = scipy.io.mmread('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/GSE96583/GSM2560248_2.1.mtx.gz') stim = scipy.io.mmread('/project2/mstephens/aksarkar/projects/singlecell-ideas/data/GSE96583/GSM2560249_2.2.mtx.gz')
First, ask whether batches can be classified. (Aside: this seems like a case where proper experimental design would eliminate the problem.)
mix = ss.hstack([control, stim]).tocsc() mix = mix[((mix > 0).mean(axis=1) >= 0.25).A.ravel()].T.astype(np.int) mix.shape
(29065, 482)
Label the cells.
y = np.zeros(mix.shape[0])
y[:control.shape[1]] = 1
m = sklm.LogisticRegressionCV(cv=4, fit_intercept=True, n_jobs=-1,).fit(mix, y) m.score(mix, y)
0.9884741097539996
Look at the embedding.
log_counts = ss.csr_matrix((np.log(mix.data), mix.indices, mix.indptr)) z = skd.TruncatedSVD(n_components=20).fit_transform(log_counts.T) embed = umap.UMAP(n_neighbors=10, metric='euclidean').fit_transform(z)
plt.clf() plt.gcf().set_size_inches(3, 3) b = y.astype(bool) plt.scatter(embed[b,0], embed[b,1], s=1, c='C0', label='Control', alpha=0.25) plt.scatter(embed[~b,0], embed[~b,1], s=1, c='C1', label='Stimulated', alpha=0.25) leg = plt.legend(frameon=False, handletextpad=0, markerscale=4) for h in leg.legendHandles: h.set_alpha(1) plt.xlabel('UMAP 1') plt.ylabel('UMAP 2')
Text(0,0.5,'UMAP 2')
Align the data using scVI.
import scvi.dataset import scvi.models import scvi.inference X = scvi.dataset.GeneExpressionDataset(*scvi.dataset.GeneExpressionDataset.get_attributes_from_matrix(mix, labels=y)) model = scvi.models.VAEC(X.nb_genes, n_batch=X.n_batches, n_labels=X.n_labels) infer = scvi.inference.UnsupervisedTrainer(model=model, gene_dataset=X) infer.train(n_epochs=10, lr=1e-2)
qz = infer.train_set.get_latent()
Align the data using AAE.
qz = scaa.align.align(mix, y, latent_dim=10, max_epochs=2, verbose=True)
Look at the disciminating in the embedding.
m = sklm.LogisticRegressionCV(cv=4, fit_intercept=True, n_jobs=-1,).fit(qz, y) m.score(qz, y)
0.9775331154309307
Look at the AAE embedding.
embed = umap.UMAP(n_neighbors=5, metric='euclidean').fit_transform(qz)
plt.clf() plt.gcf().set_size_inches(4, 3) plt.set_cmap('Dark2') b = y.astype(bool) plt.scatter(embed[b,0], embed[b,1], s=1, c='C0', label='Control', alpha=0.25) plt.scatter(embed[~b,0], embed[~b,1], s=1, c='C1', label='Stimulated', alpha=0.25) leg = plt.legend(frameon=False, handletextpad=0, markerscale=4, loc='center left', bbox_to_anchor=(1, .5)) for h in leg.legendHandles: h.set_alpha(1) plt.xlabel('UMAP 1') plt.ylabel('UMAP 2') plt.tight_layout()


Previous: Examples of Resolution of Apparent Conflict in Exceptional Situations, Up: Conforming Implementations
A conforming implementation shall produce a conformance statement as a consequence of using the implementation, or that statement shall be included in the accompanying documentation. If the implementation conforms in all respects with this standard, the conformance statement shall be
"<<Implementation>> conforms with the requirements of ANSI <<standard number>>"
If the implementation conforms with some but not all of the requirements of this standard, then the conformance statement shall be
"<<Implementation>> conforms with the requirements of ANSI <<standard number>> with the following exceptions: <<reference to or complete list of the requirements of the standard with which the implementation does not conform>>."


Checks for function and variable
declarations
to have explicit internal modifier. The validation works in ActionScript files.


An implementation of the surrogate approach to residuals and diagnostics for ordinal and general regression models; for details, see Liu and Zhang (2017) <doi:10.1080/01621459.2017.1292915>. These residuals can be used to construct standard residual plots for model diagnostics (e.g., residual-vs-fitted value plots, residual-vs-covariate plots, Q-Q plots, etc.). The package also provides an 'autoplot' function for producing standard diagnostic plots using 'ggplot2' graphics. The package currently supports cumulative link models from packages 'MASS', 'ordinal', 'rms', and 'VGAM'. Support for binary regression models using the standard 'glm' function is also available.
| Version: | 0.2.0 |
| Depends: | R (≥ 3.1) |
| Imports: | ggplot2 (≥ 2.2.1), goftest, gridExtra, stats |
| Suggests: | MASS, ordinal, rms, testthat, VGAM |
| Published: | 2017-09-19 |
| Author: | Brandon Greenwell [aut, cre], Andrew McCarthy [aut], Brad Boehmke [aut], Dungang Liu [ctb] |
| Maintainer: | Brandon Greenwell <greenwell.brandon at gmail.com> |
| BugReports: | https://github.com/AFIT-R/sure/issues |
| License: | GPL-2 | GPL-3 [expanded from: GPL (≥ 2)] |
| URL: | https://github.com/AFIT-R/sure |
| NeedsCompilation: | no |
| Materials: | README NEWS |
| CRAN checks: | sure results |
| Reference manual: | sure.pdf |
| Package source: | sure_0.2.0.tar.gz |
| Windows binaries: | r-devel: sure_0.2.0.zip, r-release: sure_0.2.0.zip, r-oldrel: sure_0.2.0.zip |
| OS X binaries: | r-release: sure_0.2.0.tgz, r-oldrel: sure_0.2.0.tgz |
| Old sources: | sure archive |
Please use the canonical form https://CRAN.R-project.org/package=sure to link to this page.


| Arrival: | {{ reservation.startDate | date:'MMMM d, yyyy' }} |
|---|---|
| Departure: | {{ reservation.endDate | date:'MMMM d, yyyy' }} |
| Number of beds: | {{reservation.room.numberOfBeds}} |
| Room number: | {{reservation.room.number}} |
| Type of room: | {{reservation.room.smoking?'Smoking': 'Non-smoking'}} {{reservation.room.seaView?'with sea-view':'without sea-view'}} |
| Price | Description | Date |
|---|---|---|
| {{purchase.amount | number}} Ft | {{purchase.description}} | {{purchase.createdAt | date:'MMMM d, yyyy'}} |


Before Change
"tests",
"test_config.yaml")
n_experiments = 2
results = {}
expected_results = {}
with CompletionService(experimentId,
config=config_path, cloud="gcspot") as cs:
for i in range(0, n_experiments):
key = cs.submitTask(
os.path.join(
mypath,
"completion_service_func.py"),
[i])
expected_results[key] = [i]
for i in range(0, n_experiments):
result = cs.getResults(blocking=True)
results[result[0]] = result[1]
self.assertEquals(results, expected_results)
@unittest.skipIf(
"GOOGLE_APPLICATION_CREDENTIALS" not in os.environ.keys(),
After Change
"tests",
"test_config.yaml")
self.test_two_experiments_with_cs_args(
config=config_path,
cloud="gcloud")
@unittest.skipIf(
"GOOGLE_APPLICATION_CREDENTIALS" not in os.environ.keys(),
"Need GOOGLE_APPLICATION_CREDENTIALS env variable to" +


Posted on 28/12/2016 at 2:00 PM by The Vibe.
Following the tradition of developers' blogging, this post has been named Hello World - to indicate that this post is where my blog started from. Through this post, I'd also like to explain about how this blog has been built. This blog is different from the other blogs in both - choice of UI framework and method of pages generation.
Semantic-UI, a new and growing front-end framework was chosen as the front-end framework, leaving behind other usual frameworks such as Bootstrap, MaterializeCSS and Zurb Foundation. After having chosen the front-end framework, the next task is of optimizing the front-end assets for faster rendering of the website - and the score on Pingdom tool was chosen as a reference. After a week of optimizing on CSS files, JS files and lazy-loading images, the website finally attained a performance grade of 92/100 - which is the highest possible score for free hosting on static GitHub Pages.
For continuing further on building blog functionality on a static website, some kind of static site generator like Jekyll had to be used. Rather, I re-invented the wheel by replicating the functionalities of Jekyll, with just a hand-made Ruby script. Here is a list of tasks that this Ruby script does -
blog
directory.data/
folder, for blog posts.posts/
, tags/
and page{i}
sub-directories.index.html
files in all sub-directories from ERB
templates, with proper back-linking to other entities.That way, whatever is shown on my blog is merely just an output of the powerful hand-made Ruby Script. Yet, there are scope of improvements left -
_config.yml
).< meta >
tags for SEO.
Incorporating the functionalities of the almighty Jekyll into a single Ruby script was pretty much fun and interesting. Even though it got challenging at times, I believe that overall, it was an awesome learning experience. For the attention of fellow developers, you can find the codes at the GitHub repository. Feel free to comment below - give your feedback about how Vibe writes.


xarray
Python library is great for analysing multi-dimensional arrays of data with labelled dimensions, which is a common situation in geosciences.
According to the docs, xarray has two core data structures. Both are fundamentally N-dimensional:
DataArray
is our implementation of a labeled, N-dimensional array. It is an N-D generalization of apandas.Series
.Dataset
is a multi-dimensional, in-memory array database. It serves a similar purpose in xarray to thepandas.DataFrame
.
Essentially, xarray adds dimensions names and coordinate indexes to numpy.ndarray
. This significantly simplifies such operations like indexing, subsetting, broadcasting and even plotting.
Let's have a look.
First, we import some necessary modules, including of course xarray
:
import numpy as np
import xarray as xr
import warnings
warnings.filterwarnings('ignore')
import matplotlib.pyplot as plt
%matplotlib inline
The following examples will be done for ERA-Interim reanalysis data of 2-m temperature and mean sea level pressure. As we will see, these are global daily data for 2014-2015.
ds = xr.open_dataset('../data/tsurf_slp.nc')
ds
Among other things, xarray prints out metadata in a much more readable format than, say, netCDF4.
Extracting variables¶
Since a Dataset
is a dict
-like object, variables can be accessed by keys.
ds['t2m']
But often it's even more convenient to access them as attributes of a Dataset
.
t2m = ds.t2m
Key properties of a DataArray¶
t2m.shape
t2m.ndim
etc...
Indexing¶
1. positional and by integer label, like numpy¶
t2m[456, :, 123]
2. positional and by coordinate label, like pandas¶
t2m.loc[dict(longitude=2.25)]
3. by dimension name and integer label¶
t2m[:2, :, 0]
t2m.isel(longitude=0, time=slice(None, 2))
4. by dimension name and coordinate label¶
import datetime
time_start = datetime.datetime.strptime('2014-02-03', '%Y-%m-%d')
time_end = datetime.datetime.strptime('2014-02-05', '%Y-%m-%d')
print('{}\n{}'.format(time_start, time_end))
t2m.sel(time=slice(time_start, time_end))
Nearest neighbour lookups¶
The following line would not work, because neither longitude coordinate array contains a value of 10
, nor latitude contains 20
.
t2m.sel(longitude=10, latitude=20) # Results in KeyError
However, we can use a built-in nearest-neighbour lookup method to find an element closest to the given coordinate values.
t2m.sel(longitude=10, latitude=15, method='nearest')
Saving data to netCDF¶
Let's extract a subset of the original data:
new_data = t2m.sel(longitude=slice(-5, 10), latitude=slice(55, 44))
new_data.shape
Saving data to a netCDF file cannot get any easier:
new_data.to_dataset('test.nc')
Converting to and from other objects¶
new_data.to_dataframe().head()
new_data.to_series().head()
Convert from pandas.DataFrame
¶
import pandas as pd
import string
df = pd.DataFrame(np.random.randn(365, 5),
index=pd.date_range(start='2014-1-1', periods=365),
columns=list(string.ascii_letters[:5]))
df.head()
df_ds = xr.Dataset.from_dataframe(df)
df_ds
GroupBy operations and resampling¶
xarray data structures allow us to perform resampling really easily:
ten_daily = t2m.resample('W', dim='time', how='mean')
Being pandas
's sibling, xarray supports groupby methods. For example, in the following line of code we do averaging of temperature by seasons (just 1 line of code!).
seas = t2m.groupby('time.season').mean('time')
print(seas.season)
Note how the time dimension was transformed into a 'season' dimension with the appropriate labels.
Plots - plots - plots¶
import cartopy.crs as ccrs
from calendar import month_name
Basics¶
You can use OO approach:
t2m.isel(time=10).plot.contourf()
Or create a figure and axis first, and then pass that as an argument to a plotting fucntions
fig, ax = plt.subplots(figsize=(10, 3),
subplot_kw=dict(projection=ccrs.PlateCarree()))
t2m.isel(time=10).plot.contourf(ax=ax)
ax.coastlines()
Now to something more interesting...¶
First, let's wrap the previous example into a small function:
def plot_field(da, ax=None, title=None):
if ax is None:
fig, ax = plt.subplots(figsize=(10, (da.shape[0] / da.shape[1]) * 10),
subplot_kw=dict(projection=ccrs.PlateCarree()))
da.plot.contourf(ax=ax)
ax.coastlines()
if title is not None:
ax.set_title(title)
Next, perform a monthly averaging on the original global data:
monthly_t2m = t2m.groupby('time.month').mean('time')
And finally, plot the result.
fig, axs = plt.subplots(nrows=4, ncols=3, figsize=(14, 10),
subplot_kw=dict(projection=ccrs.PlateCarree()))
fig.suptitle('Monthly averages of 2-m temperature')
axes = axs.flatten()
for month in range(1, 13):
ax = axes[month-1]
plot_field(monthly_t2m.sel(month=month),
ax=ax, title=month_name[month])
It's always good to close the file you are reading data from.
ds.close()
References¶
This notebook was inspired by:
HTML(html)
Comments
comments powered by Disqus


constructor
make a new instance of context menu.
new ContextMenu(menuInfo)
menuInfo
| object menuInfo { | |
| menuId: | html id of the menu |
| targetId: | id of the html element on which the context menu should be created |
| showAccessKey: | bool to show shortcut in context menu |
| beforeShowing: | function called before showing context menu. This function can update items before showing |
| items: | array of menuItems |
| } |
| object menuItem { | |
| key: | html id of the menu item |
| label: | Text of menu item |
| class: | Css class of menu item |
| fn: | function to call when the menu item is selected |
| accesskey: | shortcut key to select the menu item |
| showing: | function call to determine the css class of menu item WHEN showing menu |
| disabled: | bool to active or inactive the menu item |
| } |
right click here


#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the 'License'); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import re
from apiclient import errors
from airflow.contrib.hooks.gcp_mlengine_hook import MLEngineHook
from airflow.exceptions import AirflowException
from airflow.operators import BaseOperator
from airflow.utils.decorators import apply_defaults
from airflow.utils.log.logging_mixin import LoggingMixin
log = LoggingMixin().log
def _normalize_mlengine_job_id(job_id):
"""
Replaces invalid MLEngine job_id characters with '_'.
This also adds a leading 'z' in case job_id starts with an invalid
character.
Args:
job_id: A job_id str that may have invalid characters.
Returns:
A valid job_id representation.
"""
# Add a prefix when a job_id starts with a digit or a template
match = re.search(r'\d|\{{2}', job_id)
if match and match.start() is 0:
job = 'z_{}'.format(job_id)
else:
job = job_id
# Clean up 'bad' characters except templates
tracker = 0
cleansed_job_id = ''
for m in re.finditer(r'\{{2}.+?\}{2}', job):
cleansed_job_id += re.sub(r'[^0-9a-zA-Z]+', '_',
job[tracker:m.start()])
cleansed_job_id += job[m.start():m.end()]
tracker = m.end()
# Clean up last substring or the full string if no templates
cleansed_job_id += re.sub(r'[^0-9a-zA-Z]+', '_', job[tracker:])
return cleansed_job_id
[docs]class MLEngineBatchPredictionOperator(BaseOperator):
"""
Start a Google Cloud ML Engine prediction job.
NOTE: For model origin, users should consider exactly one from the
three options below:
1. Populate 'uri' field only, which should be a GCS location that
points to a tensorflow savedModel directory.
2. Populate 'model_name' field only, which refers to an existing
model, and the default version of the model will be used.
3. Populate both 'model_name' and 'version_name' fields, which
refers to a specific version of a specific model.
In options 2 and 3, both model and version name should contain the
minimal identifier. For instance, call
::
MLEngineBatchPredictionOperator(
...,
model_name='my_model',
version_name='my_version',
...)
if the desired model version is
"projects/my_project/models/my_model/versions/my_version".
See https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs
for further documentation on the parameters.
:param project_id: The Google Cloud project name where the
prediction job is submitted. (templated)
:type project_id: string
:param job_id: A unique id for the prediction job on Google Cloud
ML Engine. (templated)
:type job_id: string
:param data_format: The format of the input data.
It will default to 'DATA_FORMAT_UNSPECIFIED' if is not provided
or is not one of ["TEXT", "TF_RECORD", "TF_RECORD_GZIP"].
:type data_format: string
:param input_paths: A list of GCS paths of input data for batch
prediction. Accepting wildcard operator *, but only at the end. (templated)
:type input_paths: list of string
:param output_path: The GCS path where the prediction results are
written to. (templated)
:type output_path: string
:param region: The Google Compute Engine region to run the
prediction job in. (templated)
:type region: string
:param model_name: The Google Cloud ML Engine model to use for prediction.
If version_name is not provided, the default version of this
model will be used.
Should not be None if version_name is provided.
Should be None if uri is provided. (templated)
:type model_name: string
:param version_name: The Google Cloud ML Engine model version to use for
prediction.
Should be None if uri is provided. (templated)
:type version_name: string
:param uri: The GCS path of the saved model to use for prediction.
Should be None if model_name is provided.
It should be a GCS path pointing to a tensorflow SavedModel. (templated)
:type uri: string
:param max_worker_count: The maximum number of workers to be used
for parallel processing. Defaults to 10 if not specified.
:type max_worker_count: int
:param runtime_version: The Google Cloud ML Engine runtime version to use
for batch prediction.
:type runtime_version: string
:param gcp_conn_id: The connection ID used for connection to Google
Cloud Platform.
:type gcp_conn_id: string
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must
have doamin-wide delegation enabled.
:type delegate_to: string
Raises:
``ValueError``: if a unique model/version origin cannot be determined.
"""
template_fields = [
'_project_id',
'_job_id',
'_region',
'_input_paths',
'_output_path',
'_model_name',
'_version_name',
'_uri',
]
@apply_defaults
def __init__(self,
project_id,
job_id,
region,
data_format,
input_paths,
output_path,
model_name=None,
version_name=None,
uri=None,
max_worker_count=None,
runtime_version=None,
gcp_conn_id='google_cloud_default',
delegate_to=None,
*args,
**kwargs):
super(MLEngineBatchPredictionOperator, self).__init__(*args, **kwargs)
self._project_id = project_id
self._job_id = job_id
self._region = region
self._data_format = data_format
self._input_paths = input_paths
self._output_path = output_path
self._model_name = model_name
self._version_name = version_name
self._uri = uri
self._max_worker_count = max_worker_count
self._runtime_version = runtime_version
self._gcp_conn_id = gcp_conn_id
self._delegate_to = delegate_to
if not self._project_id:
raise AirflowException('Google Cloud project id is required.')
if not self._job_id:
raise AirflowException(
'An unique job id is required for Google MLEngine prediction '
'job.')
if self._uri:
if self._model_name or self._version_name:
raise AirflowException('Ambiguous model origin: Both uri and '
'model/version name are provided.')
if self._version_name and not self._model_name:
raise AirflowException(
'Missing model: Batch prediction expects '
'a model name when a version name is provided.')
if not (self._uri or self._model_name):
raise AirflowException(
'Missing model origin: Batch prediction expects a model, '
'a model & version combination, or a URI to a savedModel.')
[docs] def execute(self, context):
job_id = _normalize_mlengine_job_id(self._job_id)
prediction_request = {
'jobId': job_id,
'predictionInput': {
'dataFormat': self._data_format,
'inputPaths': self._input_paths,
'outputPath': self._output_path,
'region': self._region
}
}
if self._uri:
prediction_request['predictionInput']['uri'] = self._uri
elif self._model_name:
origin_name = 'projects/{}/models/{}'.format(
self._project_id, self._model_name)
if not self._version_name:
prediction_request['predictionInput'][
'modelName'] = origin_name
else:
prediction_request['predictionInput']['versionName'] = \
origin_name + '/versions/{}'.format(self._version_name)
if self._max_worker_count:
prediction_request['predictionInput'][
'maxWorkerCount'] = self._max_worker_count
if self._runtime_version:
prediction_request['predictionInput'][
'runtimeVersion'] = self._runtime_version
hook = MLEngineHook(self._gcp_conn_id, self._delegate_to)
# Helper method to check if the existing job's prediction input is the
# same as the request we get here.
def check_existing_job(existing_job):
return existing_job.get('predictionInput', None) == \
prediction_request['predictionInput']
try:
finished_prediction_job = hook.create_job(
self._project_id, prediction_request, check_existing_job)
except errors.HttpError:
raise
if finished_prediction_job['state'] != 'SUCCEEDED':
self.log.error('MLEngine batch prediction job failed: {}'.format(
str(finished_prediction_job)))
raise RuntimeError(finished_prediction_job['errorMessage'])
return finished_prediction_job['predictionOutput']
[docs]class MLEngineModelOperator(BaseOperator):
"""
Operator for managing a Google Cloud ML Engine model.
:param project_id: The Google Cloud project name to which MLEngine
model belongs. (templated)
:type project_id: string
:param model: A dictionary containing the information about the model.
If the `operation` is `create`, then the `model` parameter should
contain all the information about this model such as `name`.
If the `operation` is `get`, the `model` parameter
should contain the `name` of the model.
:type model: dict
:param operation: The operation to perform. Available operations are:
* ``create``: Creates a new model as provided by the `model` parameter.
* ``get``: Gets a particular model where the name is specified in `model`.
:param gcp_conn_id: The connection ID to use when fetching connection info.
:type gcp_conn_id: string
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
:type delegate_to: string
"""
template_fields = [
'_model',
]
@apply_defaults
def __init__(self,
project_id,
model,
operation='create',
gcp_conn_id='google_cloud_default',
delegate_to=None,
*args,
**kwargs):
super(MLEngineModelOperator, self).__init__(*args, **kwargs)
self._project_id = project_id
self._model = model
self._operation = operation
self._gcp_conn_id = gcp_conn_id
self._delegate_to = delegate_to
[docs] def execute(self, context):
hook = MLEngineHook(
gcp_conn_id=self._gcp_conn_id, delegate_to=self._delegate_to)
if self._operation == 'create':
return hook.create_model(self._project_id, self._model)
elif self._operation == 'get':
return hook.get_model(self._project_id, self._model['name'])
else:
raise ValueError('Unknown operation: {}'.format(self._operation))
[docs]class MLEngineVersionOperator(BaseOperator):
"""
Operator for managing a Google Cloud ML Engine version.
:param project_id: The Google Cloud project name to which MLEngine
model belongs.
:type project_id: string
:param model_name: The name of the Google Cloud ML Engine model that the version
belongs to. (templated)
:type model_name: string
:param version_name: A name to use for the version being operated upon.
If not None and the `version` argument is None or does not have a value for
the `name` key, then this will be populated in the payload for the
`name` key. (templated)
:type version_name: string
:param version: A dictionary containing the information about the version.
If the `operation` is `create`, `version` should contain all the
information about this version such as name, and deploymentUrl.
If the `operation` is `get` or `delete`, the `version` parameter
should contain the `name` of the version.
If it is None, the only `operation` possible would be `list`. (templated)
:type version: dict
:param operation: The operation to perform. Available operations are:
* ``create``: Creates a new version in the model specified by `model_name`,
in which case the `version` parameter should contain all the
information to create that version
(e.g. `name`, `deploymentUrl`).
* ``get``: Gets full information of a particular version in the model
specified by `model_name`.
The name of the version should be specified in the `version`
parameter.
* ``list``: Lists all available versions of the model specified
by `model_name`.
* ``delete``: Deletes the version specified in `version` parameter from the
model specified by `model_name`).
The name of the version should be specified in the `version`
parameter.
:type operation: string
:param gcp_conn_id: The connection ID to use when fetching connection info.
:type gcp_conn_id: string
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
:type delegate_to: string
"""
template_fields = [
'_model_name',
'_version_name',
'_version',
]
@apply_defaults
def __init__(self,
project_id,
model_name,
version_name=None,
version=None,
operation='create',
gcp_conn_id='google_cloud_default',
delegate_to=None,
*args,
**kwargs):
super(MLEngineVersionOperator, self).__init__(*args, **kwargs)
self._project_id = project_id
self._model_name = model_name
self._version_name = version_name
self._version = version or {}
self._operation = operation
self._gcp_conn_id = gcp_conn_id
self._delegate_to = delegate_to
[docs] def execute(self, context):
if 'name' not in self._version:
self._version['name'] = self._version_name
hook = MLEngineHook(
gcp_conn_id=self._gcp_conn_id, delegate_to=self._delegate_to)
if self._operation == 'create':
if not self._version:
raise ValueError("version attribute of {} could not "
"be empty".format(self.__class__.__name__))
return hook.create_version(self._project_id, self._model_name,
self._version)
elif self._operation == 'set_default':
return hook.set_default_version(self._project_id, self._model_name,
self._version['name'])
elif self._operation == 'list':
return hook.list_versions(self._project_id, self._model_name)
elif self._operation == 'delete':
return hook.delete_version(self._project_id, self._model_name,
self._version['name'])
else:
raise ValueError('Unknown operation: {}'.format(self._operation))
[docs]class MLEngineTrainingOperator(BaseOperator):
"""
Operator for launching a MLEngine training job.
:param project_id: The Google Cloud project name within which MLEngine
training job should run (templated).
:type project_id: string
:param job_id: A unique templated id for the submitted Google MLEngine
training job. (templated)
:type job_id: string
:param package_uris: A list of package locations for MLEngine training job,
which should include the main training program + any additional
dependencies. (templated)
:type package_uris: string
:param training_python_module: The Python module name to run within MLEngine
training job after installing 'package_uris' packages. (templated)
:type training_python_module: string
:param training_args: A list of templated command line arguments to pass to
the MLEngine training program. (templated)
:type training_args: string
:param region: The Google Compute Engine region to run the MLEngine training
job in (templated).
:type region: string
:param scale_tier: Resource tier for MLEngine training job. (templated)
:type scale_tier: string
:param runtime_version: The Google Cloud ML runtime version to use for
training. (templated)
:type runtime_version: string
:param python_version: The version of Python used in training. (templated)
:type python_version: string
:param job_dir: A Google Cloud Storage path in which to store training
outputs and other data needed for training. (templated)
:type job_dir: string
:param gcp_conn_id: The connection ID to use when fetching connection info.
:type gcp_conn_id: string
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
:type delegate_to: string
:param mode: Can be one of 'DRY_RUN'/'CLOUD'. In 'DRY_RUN' mode, no real
training job will be launched, but the MLEngine training job request
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
:type mode: string
"""
template_fields = [
'_project_id',
'_job_id',
'_package_uris',
'_training_python_module',
'_training_args',
'_region',
'_scale_tier',
'_runtime_version',
'_python_version',
'_job_dir'
]
@apply_defaults
def __init__(self,
project_id,
job_id,
package_uris,
training_python_module,
training_args,
region,
scale_tier=None,
runtime_version=None,
python_version=None,
job_dir=None,
gcp_conn_id='google_cloud_default',
delegate_to=None,
mode='PRODUCTION',
*args,
**kwargs):
super(MLEngineTrainingOperator, self).__init__(*args, **kwargs)
self._project_id = project_id
self._job_id = job_id
self._package_uris = package_uris
self._training_python_module = training_python_module
self._training_args = training_args
self._region = region
self._scale_tier = scale_tier
self._runtime_version = runtime_version
self._python_version = python_version
self._job_dir = job_dir
self._gcp_conn_id = gcp_conn_id
self._delegate_to = delegate_to
self._mode = mode
if not self._project_id:
raise AirflowException('Google Cloud project id is required.')
if not self._job_id:
raise AirflowException(
'An unique job id is required for Google MLEngine training '
'job.')
if not package_uris:
raise AirflowException(
'At least one python package is required for MLEngine '
'Training job.')
if not training_python_module:
raise AirflowException(
'Python module name to run after installing required '
'packages is required.')
if not self._region:
raise AirflowException('Google Compute Engine region is required.')
[docs] def execute(self, context):
job_id = _normalize_mlengine_job_id(self._job_id)
training_request = {
'jobId': job_id,
'trainingInput': {
'scaleTier': self._scale_tier,
'packageUris': self._package_uris,
'pythonModule': self._training_python_module,
'region': self._region,
'args': self._training_args,
}
}
if self._runtime_version:
training_request['trainingInput']['runtimeVersion'] = self._runtime_version
if self._python_version:
training_request['trainingInput']['pythonVersion'] = self._python_version
if self._job_dir:
training_request['trainingInput']['jobDir'] = self._job_dir
if self._mode == 'DRY_RUN':
self.log.info('In dry_run mode.')
self.log.info('MLEngine Training job request is: {}'.format(
training_request))
return
hook = MLEngineHook(
gcp_conn_id=self._gcp_conn_id, delegate_to=self._delegate_to)
# Helper method to check if the existing job's training input is the
# same as the request we get here.
def check_existing_job(existing_job):
return existing_job.get('trainingInput', None) == \
training_request['trainingInput']
try:
finished_training_job = hook.create_job(
self._project_id, training_request, check_existing_job)
except errors.HttpError:
raise
if finished_training_job['state'] != 'SUCCEEDED':
self.log.error('MLEngine training job failed: {}'.format(
str(finished_training_job)))
raise RuntimeError(finished_training_job['errorMessage'])