Cor
Cor (which means color in Portuguese) is a library for more easily
working with colors in Crystal. You can think of it as a more
powerful version of Colorize
. Cor allows you to very
easily convert RGB(A) values to hex and back again for
use in CSS and HTML. It also provides a set of
chainable methods for creting colorful strings
for the terminal using truecolor.
Installation
- Add the dependency to your
shard.yml
:
`yaml
dependencies:
cor:
github: watzon/cor
`
- Run
shards install
Usage
Creating a color
require "cor"
red = Cor.new(255, 0, 0)
# or you can do
red = Cor.new("FF0000")
# or you can also do
red = Cor.color(:red)
You can also include an alpha value
mid_red = Cor.new(255, 0, 0, 0.5)
Getting a hex string
blue = Cor.color(:blue)
puts blue.hex_string
# => 0000ff
#hex_string
also provides a couple of formatting options
puts blue.hex_string(prefix: true)
# => #0000ff
puts blue.hex_string(alpha: true)
# => 0000ffff
puts blue.hex_string(upcase: true)
# => 0000FF
Retting a RGB string
magenta = Cor.color(:magenta)
puts magenta.rgb_string
# => rgb(255, 0, 255)
puts magenta.rgb_string(alpha: true)
# => rgb(255, 0, 255, 1)
Math with colors
The Cor
class includes the basic math methods as well
which means if, for whatever reason, you want to add,
subtract, multiply, or divide colors, you can.
puts Cor.color(:magenta) - Cor.color(:blue)
# => #<Cor:0x7f5892e2df60 @alpha=255, @blue=255, @green=0, @red=0>
24 bit truecolor strings
Most modern terminals have support for True color
which allows you to add color to your terminal output. Crystal
already has support for color output via the Colorize
module in the standard library, but Cor takes things
a step further by allowing you to not only colorize
your output, but also bold, italicize, underline,
overline, blink, etc.
Cor also provides a String
patch that gives the String
class
chainable truecolor methods.
require "cor"
require "cor/string"
puts "This is awesome!".fore(:blue).back(:white)
puts "Bold me!".bold
puts "Italic me!".italic
puts "Strike me!".strike
puts "Blink me like it's 1999!".blink
puts "Faint me!".faint
puts "Underline me!".underline
puts "Overline me!".overline
Development
TODO Write development instructions here
Contributing
- Fork it (<https://github.com/watzon/cor/fork>)
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request
Contributors
- Chris Watson - creator and maintainer


{% include navbar.html%}
Access the MMT database as CSV files.
System
Choose...
One
Two
Three
Log
Choose...
One
Two
Three
Choose...
One
Two
Three
Choose...
One
Two
Three
Choose...
One
Two
Three
Submit
Start Date
End Date
Sample Interval
Seconds
Hours
Days
Submition Error
×
Modal body text goes here.
Modal title
×


simulator.register_file_utility module
import simpy
from copy import deepcopy
from simulator.register_file import RegReadReq, RegWriteReq, \
RegReadResp, RegWriteAck
class OperandReadReq:
def __init__(self, operand_id,
base_reg_addr, total_reg_size):
self.operand_id = operand_id
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
class OperandReadResp:
def __init__(self, operand_id,
base_reg_addr, total_reg_size, data):
self.operand_id = operand_id
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
self.data = data
class OperandWriteReq:
def __init__(self, base_reg_addr, total_reg_size,
simt_mask, data):
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
self.simt_mask = simt_mask
self.data = data
class OperandWriteResp:
def __init__(self, base_reg_addr, total_reg_size):
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
class RegFileOperandIOInterface:
"""Operand reading / writing interface for register file
"""
def __init__(self, env, log, config, clock_unit, reg_file, interface_type):
self.env = env
self.log = log
self.config = config
self.clock_unit = clock_unit
self.reg_file = reg_file
self.interface_type = interface_type
self.simt_len = self.config["num_threads_per_warp"]
self.alignment = self.reg_file.alignment
self.all_1s = int("1" * self.simt_len, 2)
# the following read queues also contains queues for
# read-modify-write's read request
self.read_req_queue = {}
self.read_resp_queue = {}
# write queues
self.write_req_queue = {}
if interface_type == "far-bank":
for i in range(self.config["num_subcore_reg_file_read_port"]):
self.read_req_queue[i] = simpy.Store(env, capacity=1)
self.read_resp_queue[i] = simpy.Store(env, capacity=1)
for i in range(self.config["num_subcore_reg_file_read_port"]):
self.env.process(self._handle_operand_read_req(i))
# for read-modify-write read ports only
self.rmw_base_read_port_id = \
self.config["base_regfile_read_port_id_fb_rmw"]
for i in range(self.config["num_subcore_reg_file_write_port"]):
self.write_req_queue[i] = simpy.Store(env, capacity=1)
# TODO: model write response queue more accurate
self.write_resp_queue = simpy.FilterStore(env, capacity=1)
for i in range(self.config["num_subcore_reg_file_write_port"]):
self.env.process(self._handle_operand_write_req(i))
elif interface_type == "near-bank":
for i in range(self.config["num_pg_reg_file_read_port"]):
self.read_req_queue[i] = simpy.Store(env, capacity=1)
self.read_resp_queue[i] = simpy.Store(env, capacity=1)
for i in range(self.config["num_pg_reg_file_read_port"]):
self.env.process(self._handle_operand_read_req(i))
# for read-modify-write read ports only
self.rmw_base_read_port_id = \
self.config["base_regfile_read_port_id_nb_rmw"]
for i in range(self.config["num_pg_reg_file_write_port"]):
self.write_req_queue[i] = simpy.Store(env, capacity=1)
# TODO: model write response queue more accurate
self.write_resp_queue = simpy.FilterStore(env, capacity=1)
for i in range(self.config["num_pg_reg_file_write_port"]):
self.env.process(self._handle_operand_write_req(i))
else:
raise NotImplementedError(
"Unknown register file interface type:{}"
.format(interface_type)
)
def _receive_reg_read_resp(self, regfile_read_port_id, operand_read_req,
aligned_start_addr, num_req):
cur_start_addr = aligned_start_addr
temp_buf = bytearray(num_req * self.alignment)
buf_start_addr = 0
for i in range(num_req):
resp = yield self.reg_file.read_resp_queue.get(
lambda x: x.reg_addr == cur_start_addr
)
assert isinstance(resp, RegReadResp), "The incorrect type" \
" from the response queue"
temp_buf[buf_start_addr: buf_start_addr + self.alignment] = \
deepcopy(resp.data)
cur_start_addr = cur_start_addr + self.alignment
buf_start_addr = buf_start_addr + self.alignment
# calculate correct index into the array
buf_start_addr = operand_read_req.base_reg_addr \
% self.alignment
buf_end_addr = buf_start_addr \
+ operand_read_req.total_reg_size
assert buf_end_addr <= num_req * self.alignment
# prepare a read response
operand_read_resp = OperandReadResp(
operand_id=operand_read_req.operand_id,
base_reg_addr=operand_read_req.base_reg_addr,
total_reg_size=operand_read_req.total_reg_size,
data=deepcopy(temp_buf[buf_start_addr: buf_end_addr])
)
yield self.read_resp_queue[regfile_read_port_id].put(operand_read_resp)
def _handle_operand_read_req(self, regfile_read_port_id):
while True:
operand_read_req = \
yield self.read_req_queue[regfile_read_port_id].get()
assert isinstance(operand_read_req, OperandReadReq)
ori_start_addr = operand_read_req.base_reg_addr
ori_end_addr = ori_start_addr + operand_read_req.total_reg_size
# calculate aligned address
aligned_start_addr = self.reg_file._align_down(ori_start_addr)
aligned_end_addr = self.reg_file._align_up(ori_end_addr)
num_req = (aligned_end_addr - aligned_start_addr) \
// self.alignment
cur_start_addr = aligned_start_addr
for i in range(num_req):
# compose and issue reg file req
bank_index = self.reg_file.calc_bank_index(cur_start_addr)
req = RegReadReq(reg_addr=cur_start_addr)
yield self.reg_file.read_req_queue[bank_index].put(req)
cur_start_addr = cur_start_addr + self.alignment
# spawn a process to handle returned read response
self.env.process(
self._receive_reg_read_resp(
regfile_read_port_id=regfile_read_port_id,
operand_read_req=operand_read_req,
aligned_start_addr=aligned_start_addr,
num_req=num_req
)
)
def _receive_reg_write_resp(self, operand_write_req, aligned_start_addr,
num_req):
cur_start_addr = aligned_start_addr
for i in range(num_req):
resp = yield self.reg_file.write_resp_queue.get(
lambda x: x.reg_addr == cur_start_addr
)
assert isinstance(resp, RegWriteAck), "The incorrect type" \
" from the response queue"
cur_start_addr = cur_start_addr + self.alignment
# prepare a write response
operand_write_resp = OperandWriteResp(
base_reg_addr=operand_write_req.base_reg_addr,
total_reg_size=operand_write_req.total_reg_size
)
yield self.write_resp_queue.put(operand_write_resp)
def _issue_aligned_uniform_write(self, operand_write_req,
aligned_start_addr, num_req,
aligned_data):
"""This function assumes: (1) write data and address are already
aligned to register file bank; (2) simt_mask are all 1s
"""
assert aligned_start_addr % self.alignment == 0
cur_start_addr = aligned_start_addr
buf_start_addr = 0
for i in range(num_req):
# compose and issue reg file req
bank_index = self.reg_file.calc_bank_index(cur_start_addr)
req = RegWriteReq(
reg_addr=cur_start_addr,
data=deepcopy(
aligned_data[buf_start_addr:
buf_start_addr + self.alignment]
)
)
yield self.reg_file.write_req_queue[bank_index].put(req)
cur_start_addr = cur_start_addr + self.alignment
buf_start_addr = buf_start_addr + self.alignment
# spawn a process to handle returned write response
self.env.process(
self._receive_reg_write_resp(
operand_write_req=operand_write_req,
aligned_start_addr=aligned_start_addr,
num_req=num_req
)
)
def _is_write_req_aligned(self, reg_addr, reg_size):
if reg_addr % self.alignment != 0:
return False
if reg_size % self.alignment != 0:
return False
return True
def _compose_preload_req(self, operand_write_req):
# calculate aligned access addresses
aligned_start_addr = self.reg_file._align_down(
operand_write_req.base_reg_addr
)
aligned_end_addr = self.reg_file._align_up(
operand_write_req.base_reg_addr
+ operand_write_req.total_reg_size
)
aligned_reg_size = aligned_end_addr\
- aligned_start_addr
# compose a read request
operand_read_req = OperandReadReq(
operand_id=None,
base_reg_addr=aligned_start_addr,
total_reg_size=aligned_reg_size
)
return operand_read_req
def _handle_operand_write_req(self, regfile_write_port_id):
regfile_rmw_read_port_id = self.rmw_base_read_port_id \
+ regfile_write_port_id
while True:
operand_write_req = \
yield self.write_req_queue[regfile_write_port_id].get()
assert isinstance(operand_write_req, OperandWriteReq)
simt_mask = operand_write_req.simt_mask
if simt_mask == self.all_1s:
# uniform write
if self._is_write_req_aligned(
reg_addr=operand_write_req.base_reg_addr,
reg_size=operand_write_req.total_reg_size
) is False:
# unaligned access
# read then modify: get original data first
operand_read_req = self._compose_preload_req(
operand_write_req=operand_write_req
)
# issue this to read interface
yield self.read_req_queue[regfile_rmw_read_port_id]\
.put(operand_read_req)
# get original data
operand_read_resp = yield self\
.read_resp_queue[regfile_rmw_read_port_id]\
.get()
# compose write data
write_data = deepcopy(operand_read_resp.data)
buf_start_addr = operand_write_req.base_reg_addr\
% self.alignment
buf_end_addr = buf_start_addr \
+ operand_write_req.total_reg_size
assert buf_end_addr <= len(write_data)
write_data[buf_start_addr: buf_end_addr] = \
deepcopy(operand_write_req.data)
# issue write request
num_req = len(write_data) // self.alignment
self.env.process(
self._issue_aligned_uniform_write(
operand_write_req=operand_write_req,
aligned_start_addr=operand_read_resp.base_reg_addr,
num_req=num_req,
aligned_data=write_data
)
)
else:
# aligned access
# issue write request
num_req = operand_write_req.total_reg_size\
// self.alignment
# compose write data
write_data = deepcopy(operand_write_req.data)
self.env.process(
self._issue_aligned_uniform_write(
operand_write_req=operand_write_req,
aligned_start_addr=operand_write_req.base_reg_addr,
num_req=num_req,
aligned_data=write_data
)
)
elif simt_mask > 0:
# divergent write
# read then modify: get original data first
operand_read_req = self._compose_preload_req(
operand_write_req=operand_write_req
)
# issue this to read interface
yield self.read_req_queue[regfile_rmw_read_port_id]\
.put(operand_read_req)
# get original data
operand_read_resp = \
yield self.read_resp_queue[regfile_rmw_read_port_id].get()
# compose write data
write_data = deepcopy(operand_read_resp.data)
assert operand_write_req.total_reg_size % self.simt_len == 0
data_width = operand_write_req.total_reg_size // self.simt_len
buf_start_addr = operand_write_req.base_reg_addr\
% self.alignment
for i in range(self.simt_len):
valid = (simt_mask >> i) & 1
base = data_width * i
buf_base = buf_start_addr + base
if valid:
write_data[buf_base: buf_base + data_width] = deepcopy(
operand_write_req.data[base: base + data_width]
)
# issue write request
num_req = len(write_data) // self.alignment
self.env.process(
self._issue_aligned_uniform_write(
operand_write_req=operand_write_req,
aligned_start_addr=operand_read_resp.base_reg_addr,
num_req=num_req,
aligned_data=write_data
)
)
else:
# empty request
assert False, "write simt mask is not allowed to be zero!"
Classes
class OperandReadReq
class OperandReadReq:
def __init__(self, operand_id,
base_reg_addr, total_reg_size):
self.operand_id = operand_id
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
Ancestors (in MRO)
- OperandReadReq
- builtins.object
Static methods
def __init__(
self, operand_id, base_reg_addr, total_reg_size)
Initialize self. See help(type(self)) for accurate signature.
def __init__(self, operand_id,
base_reg_addr, total_reg_size):
self.operand_id = operand_id
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
Instance variables
var base_reg_addr
var operand_id
var total_reg_size
class OperandReadResp
class OperandReadResp:
def __init__(self, operand_id,
base_reg_addr, total_reg_size, data):
self.operand_id = operand_id
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
self.data = data
Ancestors (in MRO)
- OperandReadResp
- builtins.object
Static methods
def __init__(
self, operand_id, base_reg_addr, total_reg_size, data)
Initialize self. See help(type(self)) for accurate signature.
def __init__(self, operand_id,
base_reg_addr, total_reg_size, data):
self.operand_id = operand_id
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
self.data = data
Instance variables
var base_reg_addr
var data
var operand_id
var total_reg_size
class OperandWriteReq
class OperandWriteReq:
def __init__(self, base_reg_addr, total_reg_size,
simt_mask, data):
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
self.simt_mask = simt_mask
self.data = data
Ancestors (in MRO)
- OperandWriteReq
- builtins.object
Static methods
def __init__(
self, base_reg_addr, total_reg_size, simt_mask, data)
Initialize self. See help(type(self)) for accurate signature.
def __init__(self, base_reg_addr, total_reg_size,
simt_mask, data):
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
self.simt_mask = simt_mask
self.data = data
Instance variables
var base_reg_addr
var data
var simt_mask
var total_reg_size
class OperandWriteResp
class OperandWriteResp:
def __init__(self, base_reg_addr, total_reg_size):
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
Ancestors (in MRO)
- OperandWriteResp
- builtins.object
Static methods
def __init__(
self, base_reg_addr, total_reg_size)
Initialize self. See help(type(self)) for accurate signature.
def __init__(self, base_reg_addr, total_reg_size):
self.base_reg_addr = base_reg_addr
self.total_reg_size = total_reg_size
Instance variables
var base_reg_addr
var total_reg_size
class RegFileOperandIOInterface
Operand reading / writing interface for register file
class RegFileOperandIOInterface:
"""Operand reading / writing interface for register file
"""
def __init__(self, env, log, config, clock_unit, reg_file, interface_type):
self.env = env
self.log = log
self.config = config
self.clock_unit = clock_unit
self.reg_file = reg_file
self.interface_type = interface_type
self.simt_len = self.config["num_threads_per_warp"]
self.alignment = self.reg_file.alignment
self.all_1s = int("1" * self.simt_len, 2)
# the following read queues also contains queues for
# read-modify-write's read request
self.read_req_queue = {}
self.read_resp_queue = {}
# write queues
self.write_req_queue = {}
if interface_type == "far-bank":
for i in range(self.config["num_subcore_reg_file_read_port"]):
self.read_req_queue[i] = simpy.Store(env, capacity=1)
self.read_resp_queue[i] = simpy.Store(env, capacity=1)
for i in range(self.config["num_subcore_reg_file_read_port"]):
self.env.process(self._handle_operand_read_req(i))
# for read-modify-write read ports only
self.rmw_base_read_port_id = \
self.config["base_regfile_read_port_id_fb_rmw"]
for i in range(self.config["num_subcore_reg_file_write_port"]):
self.write_req_queue[i] = simpy.Store(env, capacity=1)
# TODO: model write response queue more accurate
self.write_resp_queue = simpy.FilterStore(env, capacity=1)
for i in range(self.config["num_subcore_reg_file_write_port"]):
self.env.process(self._handle_operand_write_req(i))
elif interface_type == "near-bank":
for i in range(self.config["num_pg_reg_file_read_port"]):
self.read_req_queue[i] = simpy.Store(env, capacity=1)
self.read_resp_queue[i] = simpy.Store(env, capacity=1)
for i in range(self.config["num_pg_reg_file_read_port"]):
self.env.process(self._handle_operand_read_req(i))
# for read-modify-write read ports only
self.rmw_base_read_port_id = \
self.config["base_regfile_read_port_id_nb_rmw"]
for i in range(self.config["num_pg_reg_file_write_port"]):
self.write_req_queue[i] = simpy.Store(env, capacity=1)
# TODO: model write response queue more accurate
self.write_resp_queue = simpy.FilterStore(env, capacity=1)
for i in range(self.config["num_pg_reg_file_write_port"]):
self.env.process(self._handle_operand_write_req(i))
else:
raise NotImplementedError(
"Unknown register file interface type:{}"
.format(interface_type)
)
def _receive_reg_read_resp(self, regfile_read_port_id, operand_read_req,
aligned_start_addr, num_req):
cur_start_addr = aligned_start_addr
temp_buf = bytearray(num_req * self.alignment)
buf_start_addr = 0
for i in range(num_req):
resp = yield self.reg_file.read_resp_queue.get(
lambda x: x.reg_addr == cur_start_addr
)
assert isinstance(resp, RegReadResp), "The incorrect type" \
" from the response queue"
temp_buf[buf_start_addr: buf_start_addr + self.alignment] = \
deepcopy(resp.data)
cur_start_addr = cur_start_addr + self.alignment
buf_start_addr = buf_start_addr + self.alignment
# calculate correct index into the array
buf_start_addr = operand_read_req.base_reg_addr \
% self.alignment
buf_end_addr = buf_start_addr \
+ operand_read_req.total_reg_size
assert buf_end_addr <= num_req * self.alignment
# prepare a read response
operand_read_resp = OperandReadResp(
operand_id=operand_read_req.operand_id,
base_reg_addr=operand_read_req.base_reg_addr,
total_reg_size=operand_read_req.total_reg_size,
data=deepcopy(temp_buf[buf_start_addr: buf_end_addr])
)
yield self.read_resp_queue[regfile_read_port_id].put(operand_read_resp)
def _handle_operand_read_req(self, regfile_read_port_id):
while True:
operand_read_req = \
yield self.read_req_queue[regfile_read_port_id].get()
assert isinstance(operand_read_req, OperandReadReq)
ori_start_addr = operand_read_req.base_reg_addr
ori_end_addr = ori_start_addr + operand_read_req.total_reg_size
# calculate aligned address
aligned_start_addr = self.reg_file._align_down(ori_start_addr)
aligned_end_addr = self.reg_file._align_up(ori_end_addr)
num_req = (aligned_end_addr - aligned_start_addr) \
// self.alignment
cur_start_addr = aligned_start_addr
for i in range(num_req):
# compose and issue reg file req
bank_index = self.reg_file.calc_bank_index(cur_start_addr)
req = RegReadReq(reg_addr=cur_start_addr)
yield self.reg_file.read_req_queue[bank_index].put(req)
cur_start_addr = cur_start_addr + self.alignment
# spawn a process to handle returned read response
self.env.process(
self._receive_reg_read_resp(
regfile_read_port_id=regfile_read_port_id,
operand_read_req=operand_read_req,
aligned_start_addr=aligned_start_addr,
num_req=num_req
)
)
def _receive_reg_write_resp(self, operand_write_req, aligned_start_addr,
num_req):
cur_start_addr = aligned_start_addr
for i in range(num_req):
resp = yield self.reg_file.write_resp_queue.get(
lambda x: x.reg_addr == cur_start_addr
)
assert isinstance(resp, RegWriteAck), "The incorrect type" \
" from the response queue"
cur_start_addr = cur_start_addr + self.alignment
# prepare a write response
operand_write_resp = OperandWriteResp(
base_reg_addr=operand_write_req.base_reg_addr,
total_reg_size=operand_write_req.total_reg_size
)
yield self.write_resp_queue.put(operand_write_resp)
def _issue_aligned_uniform_write(self, operand_write_req,
aligned_start_addr, num_req,
aligned_data):
"""This function assumes: (1) write data and address are already
aligned to register file bank; (2) simt_mask are all 1s
"""
assert aligned_start_addr % self.alignment == 0
cur_start_addr = aligned_start_addr
buf_start_addr = 0
for i in range(num_req):
# compose and issue reg file req
bank_index = self.reg_file.calc_bank_index(cur_start_addr)
req = RegWriteReq(
reg_addr=cur_start_addr,
data=deepcopy(
aligned_data[buf_start_addr:
buf_start_addr + self.alignment]
)
)
yield self.reg_file.write_req_queue[bank_index].put(req)
cur_start_addr = cur_start_addr + self.alignment
buf_start_addr = buf_start_addr + self.alignment
# spawn a process to handle returned write response
self.env.process(
self._receive_reg_write_resp(
operand_write_req=operand_write_req,
aligned_start_addr=aligned_start_addr,
num_req=num_req
)
)
def _is_write_req_aligned(self, reg_addr, reg_size):
if reg_addr % self.alignment != 0:
return False
if reg_size % self.alignment != 0:
return False
return True
def _compose_preload_req(self, operand_write_req):
# calculate aligned access addresses
aligned_start_addr = self.reg_file._align_down(
operand_write_req.base_reg_addr
)
aligned_end_addr = self.reg_file._align_up(
operand_write_req.base_reg_addr
+ operand_write_req.total_reg_size
)
aligned_reg_size = aligned_end_addr\
- aligned_start_addr
# compose a read request
operand_read_req = OperandReadReq(
operand_id=None,
base_reg_addr=aligned_start_addr,
total_reg_size=aligned_reg_size
)
return operand_read_req
def _handle_operand_write_req(self, regfile_write_port_id):
regfile_rmw_read_port_id = self.rmw_base_read_port_id \
+ regfile_write_port_id
while True:
operand_write_req = \
yield self.write_req_queue[regfile_write_port_id].get()
assert isinstance(operand_write_req, OperandWriteReq)
simt_mask = operand_write_req.simt_mask
if simt_mask == self.all_1s:
# uniform write
if self._is_write_req_aligned(
reg_addr=operand_write_req.base_reg_addr,
reg_size=operand_write_req.total_reg_size
) is False:
# unaligned access
# read then modify: get original data first
operand_read_req = self._compose_preload_req(
operand_write_req=operand_write_req
)
# issue this to read interface
yield self.read_req_queue[regfile_rmw_read_port_id]\
.put(operand_read_req)
# get original data
operand_read_resp = yield self\
.read_resp_queue[regfile_rmw_read_port_id]\
.get()
# compose write data
write_data = deepcopy(operand_read_resp.data)
buf_start_addr = operand_write_req.base_reg_addr\
% self.alignment
buf_end_addr = buf_start_addr \
+ operand_write_req.total_reg_size
assert buf_end_addr <= len(write_data)
write_data[buf_start_addr: buf_end_addr] = \
deepcopy(operand_write_req.data)
# issue write request
num_req = len(write_data) // self.alignment
self.env.process(
self._issue_aligned_uniform_write(
operand_write_req=operand_write_req,
aligned_start_addr=operand_read_resp.base_reg_addr,
num_req=num_req,
aligned_data=write_data
)
)
else:
# aligned access
# issue write request
num_req = operand_write_req.total_reg_size\
// self.alignment
# compose write data
write_data = deepcopy(operand_write_req.data)
self.env.process(
self._issue_aligned_uniform_write(
operand_write_req=operand_write_req,
aligned_start_addr=operand_write_req.base_reg_addr,
num_req=num_req,
aligned_data=write_data
)
)
elif simt_mask > 0:
# divergent write
# read then modify: get original data first
operand_read_req = self._compose_preload_req(
operand_write_req=operand_write_req
)
# issue this to read interface
yield self.read_req_queue[regfile_rmw_read_port_id]\
.put(operand_read_req)
# get original data
operand_read_resp = \
yield self.read_resp_queue[regfile_rmw_read_port_id].get()
# compose write data
write_data = deepcopy(operand_read_resp.data)
assert operand_write_req.total_reg_size % self.simt_len == 0
data_width = operand_write_req.total_reg_size // self.simt_len
buf_start_addr = operand_write_req.base_reg_addr\
% self.alignment
for i in range(self.simt_len):
valid = (simt_mask >> i) & 1
base = data_width * i
buf_base = buf_start_addr + base
if valid:
write_data[buf_base: buf_base + data_width] = deepcopy(
operand_write_req.data[base: base + data_width]
)
# issue write request
num_req = len(write_data) // self.alignment
self.env.process(
self._issue_aligned_uniform_write(
operand_write_req=operand_write_req,
aligned_start_addr=operand_read_resp.base_reg_addr,
num_req=num_req,
aligned_data=write_data
)
)
else:
# empty request
assert False, "write simt mask is not allowed to be zero!"
Ancestors (in MRO)
- RegFileOperandIOInterface
- builtins.object
Static methods
def __init__(
self, env, log, config, clock_unit, reg_file, interface_type)
Initialize self. See help(type(self)) for accurate signature.
def __init__(self, env, log, config, clock_unit, reg_file, interface_type):
self.env = env
self.log = log
self.config = config
self.clock_unit = clock_unit
self.reg_file = reg_file
self.interface_type = interface_type
self.simt_len = self.config["num_threads_per_warp"]
self.alignment = self.reg_file.alignment
self.all_1s = int("1" * self.simt_len, 2)
# the following read queues also contains queues for
# read-modify-write's read request
self.read_req_queue = {}
self.read_resp_queue = {}
# write queues
self.write_req_queue = {}
if interface_type == "far-bank":
for i in range(self.config["num_subcore_reg_file_read_port"]):
self.read_req_queue[i] = simpy.Store(env, capacity=1)
self.read_resp_queue[i] = simpy.Store(env, capacity=1)
for i in range(self.config["num_subcore_reg_file_read_port"]):
self.env.process(self._handle_operand_read_req(i))
# for read-modify-write read ports only
self.rmw_base_read_port_id = \
self.config["base_regfile_read_port_id_fb_rmw"]
for i in range(self.config["num_subcore_reg_file_write_port"]):
self.write_req_queue[i] = simpy.Store(env, capacity=1)
# TODO: model write response queue more accurate
self.write_resp_queue = simpy.FilterStore(env, capacity=1)
for i in range(self.config["num_subcore_reg_file_write_port"]):
self.env.process(self._handle_operand_write_req(i))
elif interface_type == "near-bank":
for i in range(self.config["num_pg_reg_file_read_port"]):
self.read_req_queue[i] = simpy.Store(env, capacity=1)
self.read_resp_queue[i] = simpy.Store(env, capacity=1)
for i in range(self.config["num_pg_reg_file_read_port"]):
self.env.process(self._handle_operand_read_req(i))
# for read-modify-write read ports only
self.rmw_base_read_port_id = \
self.config["base_regfile_read_port_id_nb_rmw"]
for i in range(self.config["num_pg_reg_file_write_port"]):
self.write_req_queue[i] = simpy.Store(env, capacity=1)
# TODO: model write response queue more accurate
self.write_resp_queue = simpy.FilterStore(env, capacity=1)
for i in range(self.config["num_pg_reg_file_write_port"]):
self.env.process(self._handle_operand_write_req(i))
else:
raise NotImplementedError(
"Unknown register file interface type:{}"
.format(interface_type)
)
Instance variables
var alignment
var all_1s
var clock_unit
var config
var env
var interface_type
var log
var read_req_queue
var read_resp_queue
var reg_file
var simt_len
var write_req_queue


source
code documentation
sha1.h File Reference
Detailed Description
Header of US Secure Hash Algorithm 1 (SHA1).
- Author:
- Christian Scheurer <http://www.christianscheurer.ch>
This document is part of embedded IPsec
Copyright (c) 2003 Niklaus Schild and Christian Scheurer, HTI Biel/Bienne
All rights reserved.
This file contains code from the OpenSSL Project
portions Copyright (c) 1998-2003 OpenSSL (www.openssl.org)
Definition in file sha1.h.
#include "ipsec/types.h"
Go to the source code of this file.
|
Data Structures |
| struct | SHAstate_st |
Defines |
| #define | SHA_LONG __u32 |
| #define | SHA_LBLOCK 16 |
| #define | SHA_DIGEST_LENGTH 20 |
Typedefs |
| typedef SHAstate_st | SHA_CTX |
Functions |
| void | SHA1_Init (SHA_CTX *c) |
| void | SHA1_Update (SHA_CTX *c, const void *data, unsigned long len) |
| void | SHA1_Final (unsigned char *md, SHA_CTX *c) |
| unsigned char * | SHA1 (const unsigned char *d, unsigned long n, unsigned char *md) |
| void | SHA1_Transform (SHA_CTX *c, const unsigned char *data) |
| void | hmac_sha1 (unsigned char *, int, unsigned char *, int, unsigned char *) |
Define Documentation
| #define SHA_DIGEST_LENGTH 20
|
|
Typedef Documentation
Function Documentation
| void hmac_sha1 |
( |
unsigned char * |
, |
|
|
int |
, |
|
|
unsigned char * |
, |
|
|
int |
, |
|
|
unsigned char * |
|
|
) |
|
|
| unsigned char* SHA1 |
( |
const unsigned char * |
d, |
|
|
unsigned long |
n, |
|
|
unsigned char * |
md |
|
) |
|
|
| void SHA1_Final |
( |
unsigned char * |
md, |
|
|
SHA_CTX * |
c |
|
) |
|
|
| void SHA1_Transform |
( |
SHA_CTX * |
c, |
|
|
const unsigned char * |
data |
|
) |
|
|
| void SHA1_Update |
( |
SHA_CTX * |
c, |
|
|
const void * |
data, |
|
|
unsigned long |
len |
|
) |
|
|
|


[t]Do you want to become our customer?[/t]
{if $postback.customer_enroll_error}
{if $postback.customer_enroll_error == 1}
[t]Missing fields![/t]
{/if}
{if $postback.customer_enroll_error == 2}
[t]Please choose another email address![/t]
{/if}
{if $postback.customer_enroll_error == 3 || $postback.customer_enroll_error == 4 || $postback.customer_enroll_error == 5}
[t]Some error occured during data processing![/t]
{/if}
{if $postback.customer_enroll_error_invalid_captcha}
[t]Please enter correct security code[/t]
{/if}
{/if}
[t]Please enter your contact details below.[/t]


Interac Online is a transaction method available to Canadian merchants only. It allows customers to authenticate direct debits without sharing their debit card details with the merchant. An Interac payment flow is similar to the 3D Secure payment flow. It has three steps:
In addition to this guide feel free to check out our Payment APIs Demo implementation on GitHub.
https://api.na.bambora.com/v1/payments
https://api.na.bambora.com/v1/payments/{id}/continue
Send a request to our Payments API to initiate the Interac process. The response from this request will provide you with the redirect HTML code that you render to the customer in order to redirect them to the banking portal.
```shell POST /v1/payments
curl https://api.na.bambora.com/v1/payments
-H “Authorization: Passcode MzAwMjAwNTc4OjRCYU…”
-H “Content-Type: application/json”
-d ‘{
“order_number”:”MyOrderId-01234”,
“amount”:100.00,
“payment_method”:”interac”
}’
```
javascript
// Response object (JSON)
{
merchant_data (string, max length 32): ,
contents (string): ,
links (JSON):
}
Before returning the response to your users HTML client, you will need to save the merchant_data
string in the users session. This value can be used as the {id}
value when creating your ‘continue’ endpoint URL for the final request in step 3.
The response will have HTML in the contents
. This should be embedded in the user’s browser client and this needs to be displayed to the customer to redirect them to the Interac login page. Here the customer will log onto their bank account and approve the payment. An approved or declined payment will forward the customer back to the Funded or Non-funded URLs (respectively) on your website.
```html
<FORM action="https://iOnlinegateway.asp" method=POST id=frmIOnline name=frmIOnline>```
If the transaction is cancelled or declined at any point, the bank forwards a response to the merchant’s Non-funded URL. Otherwise, the bank response is forwarded to the merchant’s Funded URL. The Funded and Non-funded URLs are values the merchant must provide to us before account activation. These values are stored internally by us. Contact our customer support team to set these URLs.
When you receive a redirect back from the bank site on your Funded URL you must gather the idebit_
variables they pass to use in the ‘interac_response’ objects properties shown for the final request in step 3.
javascript
InteracResponse {
funded (string, max length 20): ,
idebit_track2 (string, max length 256): ,
idebit_isslang (string, max length 2): ,
idebit_version (number, max length 1): ,
idebit_issconf (string, max length 32): ,
idebit_issname (string, max length 32): ,
idebit_amount (number, max length 9): ,
idebit_invoice (string, max length 20):
}
The final step is to submit a request to our continue endpoint to complete the transaction. The {id}
in the URL should be the merchant_data
from the first response (Step 2).
Note: The idebit_amount
is in cents.
```shell POST /v1/payments/{id}/continue Content-Type: application/json
curl https://api.na.bambora.com/v1/payments/{id}/continue -H “Authorization: Passcode MzAwMjAwNTc4OjRCYU…” -H “Content-Type: application/json” -d ‘{ “payment_method”: “interac”, “interac_response”: { “idebit_track2”: “<string, max length 256)” , “idebit_isslang”: “<string, max length 2)” , “idebit_version”: “<number, max length 1)” , “idebit_issconf”: “<string, max length 32)” , “idebit_issname”: “<string, max length 32)” , “idebit_amount”: “<number, max length 9)” , “idebit_invoice”: “<string, max length 20)” } }’ ```
javascript
PaymentResponse {
payment_method (string): ,
id (number, max length 9): ,
approved (number, max length 1): ,
message_id (number, max length 3): ,
message (string, max length 32): ,
auth_code (string, max length 32): ,
created (date): ,
order_number (string, max length 30): ,
type (string, max length 16): ,
interac_online (json):
{
idebit_issconf (string, max length 32): ,
idebit_issname (string, max length 32):
} ,
links (json):
}


isva: Independent Surrogate Variable Analysis
Independent Surrogate Variable Analysis is an algorithm
for feature selection in the presence of potential confounding
factors (see Teschendorff AE et al 2011, <doi:10.1093/bioinformatics/btr171>).
| Version: |
1.9 |
| Depends: |
qvalue, fastICA, JADE |
| Published: |
2017-01-14 |
| Author: |
Andrew E Teschendorff |
| Maintainer: |
Andrew Teschendorff <a.teschendorff at ucl.ac.uk> |
| License: |
GPL-2 |
| NeedsCompilation: |
no |
| CRAN checks: |
isva results |
Downloads:
Reverse dependencies:
Linking:
Please use the canonical form
https://CRAN.R-project.org/package=isva
to link to this page.


 We want as many people as possible to be able to use this website. For example, that means you should be able to:
We’ve also made the website text as simple as possible to understand.
AbilityNet has advice on making your device easier to use if you have a disability.
We know some parts of this website are not fully accessible:
If you need information on this website in a different format like accessible PDF, large print, easy read, audio recording or braille: contact us
We’re always looking to improve the accessibility of this website. If you find any problems not listed on this page or think we’re not meeting accessibility requirements, contact us.
The Equality and Human Rights Commission (EHRC) is responsible for enforcing the Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018 (the ‘accessibility regulations’). If you’re not happy with how we respond to your complaint, contact the Equality Advisory and Support Service (EASS).
The Department for Transport is committed to making this service accessible, in accordance with the Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018.
This website is partially compliant with the Web Content Accessibility Guidelines version 2.1 AA standard, due to the non-compliances listed below.
The content listed below is non-accessible for the following reasons.
Aria was used on all pages of the site that did not have the correct roles. Having the incorrect markup may lead to issues for users of assistive technologies such as screen reading software. This fails WCAG 2.1 success criterion 4.1.1 (Parsing). We plan to fix this by April 2020.
The fieldset tags have elements present within them that are not valid. This means that screen reader users may not receive the correct information. This fails WCAG 2.1 success criterion 1.3.1 (Info and Relationships) and 3.3.2 (Labels or Instructions). We plan to fix this by April 2020.
There were list elements found that have direct children that are not allowed inside <li> elements. This fails WCAG 2.1 success criterion 1.3.1 (Info and Relationships). We plan to fix this by April 2020.
The calendars cannot be actioned via keyboard commands. These can only be actioned by mouse users. Some calendars did not receive focus on tab which caused confusion for our analysts. This fails WCAG 2.1 success criterion 2.1.1 (Keyboard). We plan to fix this by April 2020.
There were illogical heading structures found on pages. Having a logical and hierarchical heading structure will ensure that screen reader users can navigate the page by importance of content. This fails WCAG 2.1 success criterion 2.4.10 (Section Headings). We plan to fix this by April 2020.
We do not plan on making the map screens fully usable by users with screen readers or unable to use a mouse. Instead those users should use our non-map flow to submit information via forms.
Some PDFs and Word documents are essential to providing our services. For example, uploaded traffic management plan PDFs and forms uploaded as Word documents.
This service was last tested for accessibility in Sep 2019. The test was carried out by accessibility specialists DAC (Digital Accessibility Centre) against level AA of the Web Content Accessibility Guidelines (WCAG) 2.1.
We plan to identify and fix issues according to the timescales shown for each area above.
This statement was prepared on 1st Oct 2019. It was last updated on 1st Oct 2019.


Despite our recent global shift toward digital communication, there are still reasons we might come across scanned documents in our every day life. Scanned documents don’t inherently come with searchable or copy-able text embedded within. More often, they’re basically just images. It can be useful to extract the text from these documents though, whether for record keeping or simply organization. Manually transcribing documents is tedious, especially if there are more than about a handful.
This is where Optical Character Recognition (OCR) is useful. We can start with a document, then process that document to not only extract the text on the page, but overlay the text on the document for readability and convenience later on. Most documents we face this kind of problem with are PDFs, a near ubiquitous format by now, hence our focus on tackling OCR with PDFs.
In this short article, I’ll run us through an introduction to OCR for PDF documents in Python, with a focus on using Open Source Software (OSS).
First, a Portable Document Format (PDF) is a file which presents information, including text, images, forms, interactive content, and more. PDFs contain 7-bit ASCII characters, so opening one in a text editor will show you mostly a garble of characters you’ve never seen before. This is all in the Carousel Object Structure, which is almost like a predecessor to XML and JSON. However, what comes off like a mess is actually a detailed layout of the document, including layers with all the text within the document, complex data structures, and embedded images.
OCR has a history dating back to the early 1900s, whose progress has picked up pace during the 60s and 70s and has made an occasional jump forward in the last couple decades. Because it is a well known problem, many initiatives have made progress in accuracy, and one in particular is widely acclaimed. This is Tesseract OCR.
Tesseract is an open source OCR engine with more than 100 recognized languages, and a number of useful output types (another image, text, PDF, etc). It is moderately configurable, but has a large following and maintainer community. Most importantly though, in general it works well.
While there are APIs that layer on top of Tesseract, the engine itself is largely written in C++ and used through a command line interface. The engine accepts as input an image, and if it recognizes text within the image, it will attempt to classify the letters and words in the image and then output the transcription.
For example, in the above figure, we see how after cropping an image of a bus to retain mostly just the text, we can run Tesseract which outputs the text in the image.
This is the essence of Tesseract. Its simplicity, efficiency, and accuracy make it an ideal solution for targeted (i.e. minimal content surrounding the text) and fairly readable (i.e. success not guaranteed on Captchas) images.
Tesseract works great for images, but as discussed above, most documents come as PDFs, and PDFs are a sophisticated data structure. Tesseract does allow the output of PDFs, but one cannot input a PDF to Tesseract. While we could hack our way around PDFs to apply Tesseract on them, there exists an open source solution which I’ve recently worked with and found useful.
Namely, OCRmyPDF is a specialized command line tool and Python package which is built on a Tesseract OCR engine. OCRmyPDF does accept PDFs as input, and can not only output the text as a companion (sidecar) text file, but also overlays the text directly on top of the underlying images in the PDF. OCRmyPDF essentially pulls out the bitmap images from the PDF, performs a series of pre-processing steps (e.g. denoising, deskewing, etc.), then performs OCR on those images.
Suppose we have a PDF which looks like the following:
Yes, I chose the 2018 tax returns of ProPublica. They’re always investigating people, I think it’s only fair we investigate them too. Besides saving only the most interesting pages in the full 65 page IRS report, I didn’t alter the document at all. You’ll notice you can’t select any of the text in the document though!
OCRmyPDF to the rescue. If we save this document, and in the working folder run the following Bash command:
ocrmypdf propublica-tax.pdf propublica-tax-out.pdf --sidecar propublica-tax.txt
the package will process the original PDF (the first argument), then output the OCR’d PDF to the output file (second argument), and will also output a text file with the extracted text (file provided by the --sidecar
flag). Here is the OCR’d file:
While I encourage you to highlight and select text within the document, for those of you on mobile, here is a sample of what’s in the text file:
A For the 2019 calendar year, or tax year beginning 01-01-2018 , and ending 12-31-2018
B Check if applicable
OO Address change
O Name change
Return of Organization Exempt From Income Tax
Under section 501(c), 527, or 4947(a)(1) of the Internal Revenue Code (except private foundations)
® Do not enter social security numbers on this form as it may be made public
» Go to www.irs.gov/Form990 for instructions and the latest information.
OMB No 1545-0047
2018
Open to Public
Inspection
C Name of organization
PRO PUBLICA INC
D Employer identification number
14-2007220
Clearly, it has flaws. Namely, it’s fairly poor at disambiguating table and cell structures, and special characters are difficult to interpret. It gets some structure correct, but there’s only so much formatting we can represent with plain text files. Overall though, we could do a lot of processing with this text data. Across many tax returns, I’m sure we’d notice plenty of patterns and extracting the most useful information would not be that difficult.
OCRmyPDF may work very well in this way should you have only a handful of PDFs to process. If you have dozens, hundreds, or millions of PDFs, this process would take too long. One way to make this more efficient is to use multiprocessing, or concurrency.
We can do this with a little Python, although I’ve recently looked into learning some Go since I hear that concurrency is natively supported with their syntax and functions very well (see this cool concurrency visualization using Go).
Using the multiprocessing
native library in Python, roughly, we can spin up multiple processes (spread across however many CPUs you have) that each control an OCR job in parallel. If you have \(N\) processes, and they’ll all able to run in parallel, the time to run the total task will be divided by \(N\). If you have 100 PDFs, and each takes 20 seconds to OCR, this would take 30 minutes in serial—-in parallel on 4 processes, this would take (surprise), 8.3 minutes.
To make a small script to OCR many local documents, it would: first, load the names of the files to process. Second, distribute those files to parallel jobs. Third, execute each job. All of this can be executed in essentially less than 10 lines of code:
import os
from multiprocessing import Pool
import ocrmypdf
PROCESSES = 4
files = [f.name for f in os.scandir('pdfs/')]
chunk_size = int(len(files) / PROCESSES)
file_chunks = [files[i:i + chunk_size] for i in range(0, len(files), chunk_size)]
def ocr_files(file_list):
for input_file in file_list:
ocrmypdf.ocr("pdfs/" + input_file, "pdfs/out_" + input_file)
with Pool(processes=PROCESSES) as pool:
pool.map(ocr_files, file_chunks)
To break this down a little:
os.cpu_count()
).pdfs/
directory). Take this list of files and group it into evenly sized chunks.There are numerous improvements we could make, but this is a start. We’re not catching exceptions, we’re not allowing processes to recover if they face an error and shut down, we’re not logging, we’re not properly creating file paths (i.e. using os.path.join
), etc.
However, this would OCR a decent sized batch of PDFs for the lay person. Hopefully this tutorial is useful to those looking to dip their feet into OCR with Python without spending excessive amounts of time or money on cloud compute or proprietary software. My final comment and question for the reader: what is the best way to say, in verb form, that one will “OCR” a document (to Optical Character Recognize sounds awful)?
If you have any questions or would like (me) to clarify anything, please let me know!
Update (2022-02-21): I originally used the multiprocessing
library to create processes for parallel tasks. If I were rewriting this article I might instead take advantage of the pqdm
package, which is more user friendly and very helpfully includes a progress bar.


Ghosted
Muted List:
Reset List
No one muted… yet
Profile URL
Transparency
50%
75%
90%
100% (invisible)
Extension enabled (Requires page reload)


MOVEFLIX
MOVIEFLIX ORIGINALS
TRENDING NOW
FAMILY FEATURES
TV COMEDIES
NEW RELEASES
POPULAR ON MOVIEFLIX


Developer Student Clubs are University based community groups for students interested in Google Developer technologies.Students from all undergraduate or graduate programs with intwrest in growing as a Developer are welcome.By joining DSC, students grow their knowledge in a peer-to-peer learning environment and build solutions for local businesses and their communities.


This is our chat - conversation
Here you can listen the emotions
Press this button if you want to go back →
See page before
Send


About
Inspirations
Social
Social
I don't use a lot of social media but here are my social:
Discord: Penguin#1271
Github


| NetworkType | Description |
|---|---|
BackgroundFetch.NETWORK_TYPE_NONE |
This job doesn't care about network constraints, either any or none. |
BackgroundFetch.NETWORK_TYPE_ANY |
This job requires network connectivity. |
BackgroundFetch.NETWORK_TYPE_CELLULAR |
This job requires network connectivity that is a cellular network. |
BackgroundFetch.NETWORK_TYPE_UNMETERED |
This job requires network connectivity that is unmetered. |
BackgroundFetch.NETWORK_TYPE_NOT_ROAMING |
This job requires network connectivity that is not roaming. |
Generated using TypeDoc
BackgroundFetch.STATUS_RESTRICTED
BackgroundFetch.STATUS_DENIED
BackgroundFetch.STATUS_AVAILABLE


Upcoming local concert for Townsville Orchestra "Barrier Reef Orchestra"
Beethoven, Bohemian Classics and evening serenade.
AUSTRALIAN CONVERTO AND VOCAL COMPETITION
Bohemian Classics- Sunday March 12 5pm.
Tickets available at Ticketsop.
47 27 97 97.
Adult: $38
Concession: $34
Student: $15
NQE Member: $32


Brightness
Hue
Saturation
More/Less Controls
Blur
Grayscale
Invert
Opacity
Sepia
Reset Filters
Copy CSS


--- layout: page title: About header-img: "img/about-bg.png" ---
I am a polymer chemist with a Bachelor of Science in chemistry, currently making the leap into data science through the Flatiron School immersive online data science program.


Created by Wellcome’s Data Labs team and the Data for Science and Health Priority Area
This is a tool for researchers to examine the demographics of commonly used datasets and how they compare to the UK population.It was created by the Wellcome Trust with the intention of comparing how the UK population is represented in these datasets, and highlighting where there are disparities.
The datasets we are highlighting are:
How we chose the datasets
The datasets represented by these labels are some of the most commonly used and cited datasets in the UK today. The data displayed here was collated using a combination of metadata available in published papers and online platforms (such as Closer Discovery and the datasets own webpages). No raw data was accessed for the purpose of this project.
Known limitations:
We know that the groupings of sub-populations used in the datasets, e.g. ethnicity groupings, are subjective and potentially inaccurate at times. Future datasets and studies need more granular categories and allow people to ‘self-describe’ their background.
Please don’t hesitate to contact us with any questions, feedback or suggestions at<EMAIL_ADDRESS>UK Biobank
The UK Biobank is a prospective cohort study that recruited 500,000 adults aged between 40-69 years in the UK in 2006-2010, aiming to improve the prevention, diagnosis and treatment of a wide range of illnesses.
ALSPAC
The Avon Longitudinal Study of Children and Parents (ALSPAC) is a prospective cohort study which recruited 14,541 pregnant women living in the South West of England during 1990-1992, aiming to understand how genetic and environmental factors influence health and development in parents and children by collecting information on demographics, lifestyle behaviours, physical and mental health
CPRD
The Clinical Practice Research Datalink (CPRD, formerly the General Practice Research Database) is a primary care research database which collates de-identified patient data from general practices across the UK, covering 50 million patients, including 16 million currently registered patients.
National Child Development Study
The 1958 National Child Development Study (NCDS) is a prospective cohort study of 17,415 people born in England, Scotland and Wales in a single week of 1958. It has been used to understand topics such as the effects of socioeconomic circumstances, and child adversities on health, and social mobility.
Whitehall Study II
The Whitehall II Study (also know as the Stress and Health Study) is a prospective cohort of 10,308 participants aged 35-55, of whom 3,413 were women and 6,895 men, was recruited from the British Civil Service in 1985.
HES
Hospital Episode Statistics (HES) is a database containing details of all admissions, A and E attendances and outpatient appointments at NHS hospitals in England. It contains over 200 million records with a wide range of information about an individual patient admitted to an NHS hospital such as diagnoses, operations, demographics, and administrative information. It is often used by linking to other datasets such as the UK Biobank.
Dataset and UK Population Ratio
This shows how each demographic group is represented in comaprison to the makeup of the UK. This is calculated by taking the percent of a group in a data set and dividing it by the percent of that group in the UK population. For example: If women make up 10% of a dataset, and women comprise 50% of the population at large, that means this dataset has 20% of the number of women required to be truly representative in this metric (this doesnt include missing data)


--- layout: page title: About header-img: "img/about-bg.png" ---
Hi my name is Bram! I was born in Jakarta, Indonesia and moved to suburbs of Atlanta when he was 7 years old. I now lives in Decatur, GA with his wife, son, and cat name Hodor. I have passion for generating value using data, focusing on intuition and practical benefits. I enjoy story telling with data through data manipulation, machine learning, and visualizations.


I have tried a wide varieties of opiates. From Heroin, to pills, to raw opium.
I have learned that I prefer raw opium as opposed to any other synthetic or extracted substances. Unfortunately, high quality black tar opium is difficult to come by. Often, I have found that much circulating opium, is incense, or at best, a poppy straw extract. Occasionally, you are lucky enough to find a high quality gram of smokable O, but these opportunities are few and far between. This is not even to mention the extremely inflated cost, 20 - 30 dollars a gram for quality O out here in the great lake states. I have tried growing, but poppies are difficult to grow indoors (I don't have a house) and growing is risky.
The best way, in my opinion, to enjoy a decent opium high, is through dried poppy pods. Granted you can't smoke pods. You could do a water, ether, or alcohol extraction on the pods and smoke the extract. I have done this before and found it can occasionally be successful. Using ether or alcohol and the 'hen and chicken' variety poppy, I find I can do an extraction that is equal in potency to most decent O. But I find it is preferable and much more euphoric to grind up the pods (I use 3-5 pods). The affect is on par, for me, with a large dose of codene, or a pair of vicoden or darvocet. I wouldn't consider it as strong as percocet as one psychonaut mentioned in another report, unless 7 or 8 pods are used. This, however, as I will mention later, is a bit too much for most people.
Smoking a little marijuana will increase the effect signifcantly, and minimizes the urge to vomit. As a consequence of the amount of weed I smoke, I haven't thrown up on any opiate since I did a large amount of Morphine. But back to the point.
My strongest experience was the first time I got Poppy pods ground, not whole. I wasn't exactly sure how many grams to put in, I normally go by a certain number of pods. This method can be inaccurate, but I deal with same variety of poppy and are familiar with the shape, size, and potency of the strain. I put in what appeared to by 3-5 pods for each person, and tried to take into account that the pods were ground better than I could ever grind them.
The tea was brewed, I added modest portion of cinnamon and honey. I find this makes the flavor of the tea most bearable.
The tea was portioned out to each person with about one cup of tea leftover. With me was my grrlfriend and another friend, also a female.
We each drank our tea. While we were waiting the effects to kick in we smoked a half gram of raw Opium and we smoked 5 or 6 bowls of pot. Within 45minutes we could feel the effects coming on, I felt I could finish off the remaining tea and I did.
After another bowl or two of marijuana, the grrls were feeling extremely O'ped up, as was I. However, they began to complain of nausea. I was trying my best to help them to their respectable bathrooms to vomit, without making myself too sick. It was difficult for any of us to move at all and the grrls were vomiting all over my apartment by this point. Realizing there was nothing I could do now, I let them vomit. I realized I had messed up and should have wieghed out the pods.
I was dissappointed in myself, normally I am more cautious when doing the dosage for someone else. I felt a bit guilty for making them sick. Somehow, I managed to avoid puking. Once they were no longer ill, we dragged ourselves to my bedroom. We spent hours lying down, feeling extremely pleasant, somewhere between sleep and conciousness. Finally we passed out. Since that expierence I have learned to be more careful.
Reading the other opium expierence reports, I was dissappointed to hear so many negative reports regarding opium tea. I find Poppy tea and opium to be extremely pleasant, but regretably addictive substances.
I highly recommend opium, but respect the pod. Don't be a slave.
Experience Reports are the writings and opinions of the individual authors who submit them.
Some of the activities described are dangerous and/or illegal and none are recommended by Erowid.


--- layout: post title: Meta Programming In... disqus_id: /2009/11/11/meta-programming-in/ categories: - ruby-rails - heuristics ---
Last nights 757.rb meeting was a great success. We did a talk titled Introduction To Ruby & Rails for all the new comers that might have been interested in learning more about both - from the ground up. Since Ruby was my first language and my toolbelt only extends to JavaScript and Objective-C, I'm always interested in how other languages do the things that I love so much in Ruby. One of the biggest pluses for Ruby, to me, is the support for meta programming. In a typical rails model, you might see something like this.
{% highlight ruby %} class User < ActiveRecord::Base has_many :articles named_scope :active, :conditions => {:active => true} validates_uniqueness_of :email validates_presence_of :email, :username, :twitter_handle end {% endhighlight %}For the untrained, those are class level methods writing code for an model that persists objects to a database. It looks like you are typing a spec or some declaration. In fact this is code writing other code. For newcomers to the language this is voodoo magic, but if you write programs and interfaces for others, meta programming is what it is all about. In fact, I use this technique all the time instead of writing the same code over and over again.
This concept is built into the core of Ruby. In fact here is an example from my talk on the class method built into Ruby called attr_accessor
which wraps a common paradigm of generating getter/setter methods for an instance variable. No magic here, the two class definitions below are equal and the interface of the object is the same. One is you do not have to write code over an over again in an vanilla class you care to have getter/setter methods for.
So here is where I want to learn. I know how to write Ruby well enough that I can show one way below of how to write attr_accessor
if it did not exist in core Ruby. In my example below I made the method named attribute_accessor
to avoid the name conflict. Can anyone show me how this would be done in PHP, Java, Anything? What is meta programming like in those languages? How about posting a code example on pastie.org and school me. I really am interested in knowing. BTW, my example only shows how the attribute_accessor
takes on argument. In actuality in Ruby, the built in attr_accessor
method takes multiple args and generates methods for each. For simplicity I narrowed down my examples and code to only take one argument so we can all stay focused.
Update: A friend was asking me about the namespaces you see in the module. It is technically possible to just open up the Class object and add my example additions. Typically in Ruby you are encouraged to create your own namespaced module and inject class/instance modules into the included class via the module's hook method self.included
. The pattern is like so http://pastie.org/694019. So for instance, I could have done this http://pastie.org/694038 without taking advantages of the modules hook method.


Java 8
20 Dec 2014This is a basic introduction to the Java 8 new features.
- Lambda
- Functional Interfaces
- Method references
- Streams
- Default methods
- Nashorn
- New Date and Time API
- Comparisons to Java 7
Lambda
- Maintainability and Readability vs Less code writing. However, Java has an excessive verbosity
- Less boilerplate code
- "Kind of syntatic sugar for annonymous classes"
- "Lambda support is one of the most visible enhancements of java 8".
- Cleaner syntax to event listeners
This:
Can be written like this:
And even in one line:
With the Runnable interface:
Functional Interfaces
- You still need to type your references (Java 8 will not infer it).
- Markup interfaces for 'functional style' code blocks.
- An interface with exactly one abstract method.
java.util.function
contains some of them
Can be assigned to anything that fullfill their contract. Example:
Sample java.util.function
functional interfaces:
Function<T,R>
- takes an object of type T and returns R.Predicate<T>
- returns a boolean value based on input of type T.Consumer<T>
- performs an action with given object of type T.
Method references
- Not abuse of lambda expressions.
- Can refer to a method in its context.
Example:
Streams
java.util.stream
package- Supports functional style operations on streams of values
- Something like an iterator. Can be traversed once, and can also be infinite.
- Sequential or Parallel. Can be switched using
stream.sequential()
andstream.parallel()
- Some operations available:
map
,filter
,flatMap
,sorted
,forEach
,count
, and more.
To obtain a stream from a collection:
Example of functional operations on a stream:
Default methods
- Default methods = Defender Methods = Virtual Extension methods
- Backwards compatibility
- Collections API needed to support a new feature to
- Can be added to any interface
- Any class that implements the interface, but doesn't override the method will get the default implementation.
Multiple defaults:
Nashorn
- Successor of Rhino
- JavaScript integrated to the JVM
- Proposal: Nashorn JEP
New Date and Time API
- Historically bad in Java
- Often requires external libraries (e.g. JodaTime)
Comparisons to Java 7
Creating an ActionListener
Printing out a list of Strings
Sorting a list of Strings
Sorting complex objects:
References:
- https://leanpub.com/whatsnewinjava8/read
- https://www.techempower.com/blog/2013/03/26/everything-about-java-8/


The simple design of the WAV3BOT utilizes 3D printing and waste to create a device that is easily accessible to all.
Access all of the code on the GitHub and develop new variants of the WAV3BOT for your business or personal needs.
The system is inexpensive and easy to construct, resulting in a quick build with the potential for several user variants.
“The WAV3BOT allows me to use my motions to control its movements. Over time, the movements become fluid and natural. I myself am a developer and am seeing several possible purposes for WAV3BOT technology. The Open Source nature of this project will allow for my creative liberty when developing products based on WAV3BOT accelerometer input.”
“Me using the WAV3BOT was like bringing something to life through my motions. It was like I transfered everything I wanted it to do without ever speaking a word or giving a direct command. I plan to take this technology to develop a paint bot that allows me to create works of art controlled by my motions. Human nature and emotions cannot be digitally recreated, but through this we can get pretty close.”
“It's a very easy open source project to get involved with. There are very few componenets and set up requirements, but the variations that can be made from that point of basic completion are essentially limitless. The true development here is the WAV3BOT Gauntlet which is a new and very cheap input device that allows for complete immersion and seamless control.”
Have an idea for how to use WAV3BOT technology? Contact us and we will help you bring your idea to fruition. The possibilities are truly endless and we are very excited for what you have to show us.
The WAV3BOT utilizes Arduino. This open source and easily accessible electronics system and open source code will allow users to easily create WAV3BOT and develop their own variants. We will provide assistance to anyone who has an idea and is attempting to navigate the interface.
Through photo and video attachments and developments, users will be able to record their robot's pattern movements and travels.
If you are attempting to develop WAV3BOT for business purposes, feel free to contact us regarding your needs. We are always available to help.
Vignesh is a student at J.L.Mann High School and he likes to research and make cool things.
After graduating from MIT with degrees in life sciences and electrical engineering, Jonathan was a clinical engineer with the Veterans Administration from 1975-2000, with a multitude of projects including prosthetic devices and interfacing, ICU, cath lab, and operating room design, and other projects.
Diane develops K-12 STEM curriculum and works with Cambridge schools to implement week-long STEM experiences. She is an MIT alumna and worked in product design as a mechanical engineer before going into education full time. She strongly believes that engineering is a great way to inspire kids to be problem solvers, be creative, and be empowered in their own lives.


Retainer Sent Date
Please enter date
WL Numbers
Each WL number should be on a separate line
Please enter WL Numbers (each number should be on a separate line)
Update Retainer Sent


We are a group of software developers from India.Most of us have previously worked in software companies .This project is our joint effort to ease the file conversion process
We love to hang around ,make friends and travel a lot .If you want to join us , send request to our facebook profile Krazylearner and leave a message there or use contact form on this webpage
We love to help you in your projects.Please send us brief details using the contact form and we will get back to you as soon as possible
We are continuously working to add more features in this online converter.Currently it supports only image to pdf conversion but we are seriously looking to add more formats in near future
Any suggestion including the functionality ,user interface or performance is most welcomed.Use this FEEDBACK link to give us feedback or suggestion.
Privacy Policy Copyright @ online-pdfconverter.net


Don’t limit yourself to the materials provided by Levine when preparing. There are also other good resources such as those prepared by Silver on traditional RL. We’ll try to post links and synchronize where possible.
| Date | Description | Deadlines |
|---|---|---|
| Preflight Week of 17, 24 Jan |
Introduction and Course Overview
[Video (@UCB),
Slides (@UCB)]
Supervised Learning and Imitation [Video (@UCB), Slides (@UCB)] , & TensorFlow and Neural Nets Review Session (notebook) [Video (@UCB), Slides (@UCB)] |
|
| Week 3 31 Jan |
Reinforcement Learning Introduction
[Video (@UCB),
Slides (@UCB)]
,
Policy Gradients Introduction [Video (@UCB), Slides (@UCB)] , & [ » Scribe Notes ] [ » Recording @ YouTube ] |
|
| Week 4 7 Feb |
Actor-Critic Introduction
[Video (@UCB),
Slides (@UCB)]
Value Functions and Q-Learning [Video (@UCB), Slides (@UCB)] , & [ » Scribe Notes ] [ » Recording @ YouTube ] |
|
| Week 5 14 Feb |
Advanced Q-Learning Algorithms
[Video (@UCB),
Slides (@UCB)]
[ » Scribe Notes ] [ » Recording @ YouTube ] |
|
| Week 6 21 Feb |
Advanced Policy Gradients
[Video (@UCB),
Slides (@UCB)]
[ » Scribe Notes ] [ » Recording @ YouTube ] |
Preliminary project titles and team members due on Slack's #projects
|
| Recess Week 28 Feb |
Optimal Control and Planning
[Video (@UCB),
Slides (@UCB)]
[ » Scribe Notes ] [ » Recording @ YouTube ] |
|
| Week 7 7 Mar |
Model-Based Reinforcement Learning
[Video (@UCB),
Slides (@UCB)]
Advanced Model Learning and Images [Video (@UCB), Slides (@UCB)] [ » Scribe Notes ] [ » Recording @ YouTube ] |
Preliminary abstracts due to #projects
|
| Week 8 14 Mar |
Learning Policies by Imitating Other Policies
[Video (@UCB),
Slides (@UCB)]
[ » Scribe Notes ] [ » Recording @ YouTube ] |
|
| Week 9 21 Mar |
Probability and Variational Inference Primer
[Video (@UCB),
Slides (@UCB)]
Connection between Inference and Control [Video (@UCB), Slides (@UCB)] [ » Scribe Notes ] [ » Recording @ YouTube ] |
|
| Week 10 28 Mar |
Inverse Reinforcement Learning
[Video (@UCB),
Slides (@UCB)]
[ » Recording @ YouTube ] |
|
| Week 11 4 Apr |
(warning: we skip a few lectures to get here) Exploration: Part 1 [Video (@UCB), Slides (@UCB)] , & Exploration: Part 2 [Video (@UCB), Slides (@UCB)] [ » Recording @ YouTube ] |
|
| Week 12 11 Apr |
Parallelism and RL System Design
[Video (@UCB),
Slides (@UCB)]
[ » Slides (local, .pdf) ] [ » Recording @ Youtube ] |
|
| Week 13 18 Apr |
Recent Advances in Game RL: OpenAI 5's DotA 2
[ » Slides (local, .pdf) ]
[ » Recording @ Youtube ] |
Participation on evening of 14th STePS |


|
|
bias correction |
|
|
wavelength |
|
|
median ratio |
|
|
absorption coefficient of detrital particles and coloured dissolved organic matter (or gelbstoff) combined |
|
|
absorption coefficient of phytoplankton |
|
|
total absorption coefficient |
|
|
back-scattering coefficient for particles |
|
|
vertical attenuation coefficient for downwelling irradiance |
|
|
normalised remote-sensing reflectance |
| AOP |
Apparent Optical Property |
| CCI |
Climate Change Initiative |
| CF |
Climate and Forecast |
| ECV |
Essential Climate Variable |
| ESA |
European Space Agency |
| GAC |
Global Area Coverage |
| GCOS |
Global Climate Observing System |
| HDFS |
Hadoop Distributed File System |
| HPLC |
High-Performance Liquid Chromatography |
| IOCCG |
International Ocean Colour Coordinating Group |
| IOP |
Inherent Optical Property |
| LAC |
Local Area Coverage |
| L2Gen |
NASA’s Level 2 Generator |
| MERIS |
MEdium spectral Resolution Imaging Spectrometer |
| MERMAID |
MERis MAtchup In Situ Database |
| MODIS-Aqua |
Moderate-resolution Imaging Spectroradiometer-Aqua |
| NASA |
National Aeronautics and Space Administration |
| NCEP |
National Center for Environmental Prediction |
| NetCDF |
Network Common Data Form |
| NOMAD |
NASA bio-Optical Marine Algorithm Dataset |
| OC-CCI |
Ocean-Colour Climate Change Initiative |
| OBPG |
Ocean Biology Processing Group (of NASA) |
| PAR |
Photosynthetically Available Radiation |
| POLYMER |
POLYnomial based algorithm applied to MERIS |
| QAA |
Quasi-Analytical Algorithm (QAA [
19])
|
| SVC |
System Vicarious Calibration |
| SWIR |
Short-wave infrared |
| VIIRS |
Visible and Infrared Imaging Radiometer Suite |


website
.
omegaPsi
();
here is a list of commands that are available in my bot, Omega Psi.
page
commands


First Name |
|
Last Name |
|
House Number |
|
Street |
|
city |
|
Province |
|
Zipcode |
|
Phone number |
|
Cell phone number |
|
Birthday |
|
Citizenship number |
|
Zodiac Sign |
|
Zodiac Year |
|
Birth Day Of Week |
|


DIFFERENTIAL EVOLUTION
Preamble
I have put a lot of effort into this contribution to Matlab Central. As I
used the code successfully for myself for quite some time, I am sure that it can
be rather valuable for the one or the other. If you find any errors or bugs,
if you have problems in using the function or if you find the documentation
insufficiently detailed:
Please contact me and give me the chance to help you before giving a bad
rating on Matlab Central!
Contact details at the bottom of this page.
Thanks
I have spent many hours to develop this package. If you would like to let me know that you appreciate my work, you can do so by leaving a donation.
Introduction
This contribution provides functions for finding an optimum parameter set
using the
Evolutionary Algorithm of Differential Evolution. Simply
speaking: If you have some complicated function of which you are unable to
compute a derivative, and you want to find the parameter set minimizing the
output of the function, using this package is one possible way to go.
The core of the optimization is the Differential Evolution algorithm. For an
introduction to the algorithm, see the
basics section on the
Differential Evolution homepage of Rainer Storn. You will also find a demo
applet and code for different programming languages there. For in-depth documentation and
publications, check out the
Literature Section. Even several
books about Differential Evolution are available.
This package provides much more than the code available on the referenced
homepage. Here is a list of some features:
- Optimization can run in parallel on multiple cores/computers.
- Extensive and configurable progress information during optimization.
- Intermediate results are stored for later review of optimization progress.
- Progress information can be sent by E-mail.
- Optimization toolbox is not needed.
- Quick start with demo functions.
- Intermediate results are displayed after the optimization.
- Different end conditions can be chosen (maximum time, value to reach etc.).
- Each parameter value can be constrained to an interval.
- Each parameter value can be quantized (for example for parameters of integer nature).
- Code can easily be extended to use the evolutionary algorithm of your choice.
I have developed this package for an own project. A single evaluation of my
objective function took between 30 and 60 seconds and the parameter
space was galactically large. If your objective function needs only milliseconds to
evaluate and your optimization is expected to finish in seconds or minutes,
you can still use this package. However, much of its power (parallel processing,
progress notifications) will not be of much use.
In addition to this documentation, I have summarized E-mail conversations in
a list of Frequently Asked Questions.
Quick start
To get into the usage of the package quickly, check out the demo functions
demo1.m,
demo2.m and
demo3.m. Modify those files
to start your first optimization. Essentially you only have to define
which parameters to optimize and provide a handle to your objective function.
You can learn about everything else later.
Your objective function can be called in different ways:
- With a scalar or column vector as the only input argument.
- With a structure containing the current parameters as only input argument.
- With one or more user-defined fixed arguments first, then a parameter vector/structure as last argument.
If your objective function has an argument list that does not comply with
one of these possibilities, you have to write a small wrapper function which
brings the arguments into the correct order and calls your objective function.
The definition of the parameters to optimize is expected as a cell array. The
first column of the cell array has to contain the parameter names. In the second
column, you have to provide the parameter ranges. For a scalar parameter, the
range is expected as a two-element row vector. For vector-valued parameters, you
have to give the ranges of the elements as a two-column matrix with ranges in
rows. The third column contains the parameter quantization steps as a
scalar or a column vector (set to zero for no quantization). The fourth column
(optional) can contain the initial values of the parameters. If your objective
function shall be called with a column parameter vector instead of a structure
as input, define only one single parameter with an empty string as parameter name.
See the help text ("help differentialevolution") and the demos for
more details and examples.
Parallel processing on multiple cores
This package allows to work in parallel on multiple cores in order to increase
the speed of the optimization. One process acts as the master and all other
processes act as slaves. The only requirements for parallel processing are:
- All involved processes have to have read- and write-access to a common directory,
for example on a network share.
- All involved processes have to have access to identical code versions of the
objective function.
For parallel processing, the main function differentialevolution.m has to be started in
one Matlab process. In one or more other processes, the function differentialevolutionslave.m
has to be started. The master process will save files including the parameter
sets to evaluate into the common directory.
The slave processes load the parameter files, evaluate the objective function and
save the obtainted results into other files. After each iteration, the master process
collects the evaluated results and feeds the slave processes with data files
again. If there are no results found, the master process will evaluate the
parameter sets himself. This way, the slaves can never cause the master to
get stuck.
The
Multicore package on Matlab Central actually emanated from this function
and works quite similar.
The usage in detail
The details of how to use the package are contained as help lines in the file
differentialevolution.m. I paste the same lines here:
%DIFFERENTIALEVOLUTION Start Differential Evolution optimization.
% BESTMEM = DIFFERENTIALEVOLUTION(DEPARAMS, ...) starts a Differential
% Evolution (DE) optimization to minimize the cost returned by a given
% function. For a quick start, check out and modify the functions DEMO1
% and DEMO2.
%
% Output arguments:
%
% bestmem - Best population member.
% bestval - Lowest evaluated cost.
% bestFctParams - Structure like input objFctParams containing the
% best parameter set.
% nrOfIterations - Number of iterations done.
%
%
% Input arguments:
%
% DEParams - Struct with parameters for DE.
% paramDefCell - Cell specifying the parameters to optimize.
% objFctHandle - Handle to the objective function.
% objFctSettings - Additional settings to be passed (a cell array will
% be expanded using {:}). If no additional settings
% are needed, set objFctSettings to an empty cell: {}
% objFctParams - Struct with initial parameters.
% emailParams - Struct with fields serveraddress, fromaddress,
% toaddress, and, if needed, username and password.
% Parameters are used for sending E-mail
% notifications.
% optimInfo - Info about current optimization task. Fields 'title'
% and 'subtitle' are displayed and included in saved
% files if existing. No influence on optimization.
%
% The structure DEParams needs to contain the following fields:
%
% VTR "Value To Reach" (set to empty matrix for no VTR).
% NP Number of population members (e.g. 10*D).
% F DE-stepsize F from interval [0, 2]. A good initial guess
% is the interval [0.5, 1], e.g. 0.8.
% CR Crossover probability constant from interval [0, 1]. If
% the parameters are correlated, high values of CR work
% better. The reverse is true for no correlation.
% strategy 1 --> DE/best/1/exp (def.) 6 --> DE/best/1/bin
% 2 --> DE/rand/1/exp 7 --> DE/rand/1/bin
% 3 --> DE/rand-to-best/1/exp 8 --> DE/rand-to-best/1/bin
% 4 --> DE/best/2/exp 9 --> DE/best/2/bin
% 5 --> DE/rand/2/exp else DE/rand/2/bin
% Experiments suggest that /bin likes to have a slightly
% larger CR than /exp
% maxiter Maximum number of iterations.
% maxtime Maximum time (in seconds) before finishing optimization.
% Set to empty or Inf for no time limit.
% maxclock Time (as returned by function clock.m) when to
% finish optimization. Set to empty for no end time.
% minvalstddev Population is reinitialized if the standard deviation of
% the cost values in the population is lower than
% minvalstddev.
% minparamstddev Population is reinitialized if the maximum parameter
% standard deviation (normalized to the parameter range)
% is lower than minparamstddev.
% nofevaliter Population is reinitialized if there was no function
% evaluation during the last nofevaliter iterations.
% nochangeiter Population is reinitialized if there was no change in
% the population during the last nochangeiter iterations.
% refreshiter Info is displayed and current state is saved every
% refreshiter iterations.
% refreshtime State is saved after refreshtime seconds.
% refreshtime2 Additional progress information is displayed every
% refreshtime2 seconds (usually refreshtime2 >>
% refreshtime).
% refreshtime3 Progress information is sent by E-mail every
% refreshtime3 seconds (usually refreshtime3 >>
% refreshtime2).
% useInitParams If one, the given parameters in struct objFctParams
% OR those in the fourth column of paramDefCell are used
% as one of the initial population members. If two,
% additionally the first twenty percent of the population
% members are computed as the given initial parameter
% vector plus small random noise.
% saveHistory Save intermediate results.
% displayResults Draw graphs for visualization of the optimization
% result.
% feedSlaveProc Use slave process for parallel computation.
% maxMasterEvals Maximum number of function evaluations done by the
% master process. Warning: Use this option with caution!
% If maxMasterEvals is set to a number less than the
% number of population members and one of the slave
% processes is interrupted, the optimization will be
% stuck!
% slaveFileDir Base directory for saving slave files.
% minimizeValue If true, the evaluation value is minimized, otherwise
% maximized.
% validChkHandle Handle to a function which takes the same arguments as
% the objective function and decides if a given parameter
% set is valid (subject to a constraint) or not. The
% function should return 1 if the constraint is fulfilled
% and 0 if not. Set to empty string if no constraint is
% needed.
% playSound Play a short sound when a new best member was found.
%
% If DEParams is empty or fields are missing, default parameters are used
% (see function GETDEFAULTPARAMS).
%
% The cell array paramDefCell has to contain the names of the parameters
% to optimize, its ranges, their quantizations and the initial values.
% Each parameter may be a real-valued scalar or column vector.
%
% Example 1 (only scalar parameters):
%
% paramDefCell = {
% 'useSmoothing', [0 1], 1, 0
% 'nrOfCoefficients', [5 20], 1, 10
% 'threshold', [0.01 1], 0.001, 0.5 }
%
% The first cell in each row contains the name of the parameter, the
% second a two-element row vector specifying the allowed range, the third
% the quantization and the fourth the initial values (the fourth column
% of the cell array may be omitted). Provide a non-empty value either in
% objFctParams or in the fourth column of paramDefCell as initial value.
% If both are present, a warning message is issued and the value in
% paramDefCell is used. If objFctParams is empty and no initial
% parameters are given in paramDefCell, the centers of the parameter
% ranges are used as initial parameters.
%
% Using parameter quantization allows for the use of binary variables
% like 'useSmoothing' above as well for parameters that are of integer
% nature, like a number of coefficients. If the quantization of a
% parameter is set to zero, the parameter is not quantized. Using a
% quantization grid for continuous parameters can save computational
% effort. If saveHistory is true, all evaluated parameter vectors are
% saved with the corresponding cost value and the same parameter value
% will never be evaluated twice. With quantization, it is more likely
% that a generated parameter vector was already evaluated and saved
% before.
%
% Example 2 (vector parameter):
%
% paramDefCell = {'weightings', [0 1; 0 2], [0.01; 0.02], [0.5; 0.5]};
%
% Here, the parameter weightings is defined as a two-element column
% vector. The ranges are set to [0, 1] for the first element and [0, 2]
% for the second. The quantizations are 0.01 and 0.02 and the initial
% values are both 0.5.
%
% The objective function (given as function handle objFctHandle) is
% started as
%
% value = objFctHandle(objFctSettings, objFctParams) or
% value = objFctHandle(objFctSettings{:}, objFctParams).
%
% The second case is used if objFctSettings is a cell array, thus
% allowing for an arbitrary number of additional input arguments. The
% provided structure objFctParams may contain further fixed parameters
% and/or the current parameter values. The fields with the names of the
% parameters given in paramDefCell are overwritten by the values of the
% current parameters. If the objective function handle is empty, the
% distance to a randomly chosen optimal parameter vector is used as cost
% value (for testing purposes).
%
% Example 3 (vector parameter):
%
% paramDefCell = {'', [0 1; 0 2], [0.01; 0.02], [0.5; 0.5]};
%
% In this special case (one single parameter with empty name), the
% objective function is called as
%
% value = objFctHandle(objFctSettings, paramVec) or
% value = objFctHandle(objFctSettings{:}, paramVec)
%
% with the current parameters in column vector paramVec.
%
% When displaying an info string, the current optimization state
% including all tested members etc. is saved in the file
% XXX_result_YYY_ZZ.mat, where XXX is the name of the objective function,
% YYY is the name of the current host and ZZ is a number between 1 and 50
% (to avoid overwriting old results).
%
% A 'time over'-file is saved at the start of the optimization. The
% optimization is stopped if this file is deleted. Using this mechanism
% to stop the simulation avoids to break Matlab during saving a file,
% which can make a file unaccessible for the rest of the session and
% leads to repeating warning messages. The name of the file to delete is
% XXX_timeover_YYY.mat, where XXX is the name of the objective function
% and YYY is the hostname. Result- and 'time over'-files are saved in
% directory 'data' if existing, otherwise in the current directory.
%
% The optimization can be performed in parallel by more than one
% processor/computer. Function DIFFERENTIALEVOLUTION has to be started in
% one Matlab session, function DIFFERENTIALEVOLUTIONSLAVE on one or
% more other Matlab sessions. Function DIFFERENTIALEVOLUTION acts as
% master and saves information about which function to evaluate and which
% parameters to use into files in a commonly accessible directory. The
% Distributed Computing toolbox is not used. If input parameter
% slaveFileDir is empty, the directory differentialevolution is used (or
% created) below the temporary directory returned by function TEMPDIR2
% (something like C:\Documents and Settings\\Local Settings\
% Temp\@\MATLAB).
%
% Function DIFFERENTIALEVOLUTION was developed for objective functions
% that need relatively long for one function evaluation (several seconds
% or more). When used with objective functions that evaluate very fast,
% memory problems could occur. When saveHistory is true, every evaluated
% parameter vector is kept in memory. Further, the overhead for checking
% if a parameter vector was already evaluated might be larger than a
% function evaluation itself.
%
% Start this function without input arguments or with only the first
% input argument DEParams to run a demo optimization of Rosenbrock's
% saddle. No files are saved during the demo.
%
% This function is based on the differential evolution (DE) algorithm of
% Rainer Storn (http://www.icsi.berkeley.edu/~storn/code.html). The core
% evolutionary algorithm was taken from function devec3.m.
%
% differentialevolution.html File Exchange Donate via PayPal
%
% Markus Buehren
% Last modified 05.07.2011
%
% See also DIFFERENTIALEVOLUTIONSLAVE, DISPLAYOPTIMIZATIONHISTORY,
% GETDEFAULTPARAMS, DEMO1, DEMO2, TEMPDIR2.
Please check the help lines in file differentialevolution.m to be sure to read the
latest version.
Functions contained in this package
In the following, the most important functions contained in this package are
listed. Every file contains its own help comments which you can access by
typing "help functionname" on the Matlab command line.
differentialevolution.m
The main function to call after preparing the input arguments.
getdefaultparams.m
When starting to work with this package, you probably do not want to handle
with every existing parameter for function differentialevolution.m. You can
get a default parameter set by calling getdefaultparams.m.
differentialevolutionslave.m
When working in parallel on multiple cores/computers, this
function has to be started in every Matlab process that shall act as slave.
computenewpopulation.m
The core Differential Evolution algorithm resides in this functions. If you
like to use your own favorite evolutionary algorithm, you can put the code
into this function.
demo1.m, demo2.m, demo3.m
Demo functions you can modify for a quick start.
foxholes.m, rosenbrocksaddle.m
These two functions are used for the demos. They implement two functions that
are often cited in the context of optimization algorithms ("Shekel's
Foxholes" and "Rosenbrock's Saddle").
Parameter quantization
The original Differential Evolution algorithm only knows unbounded, continuous
parameters. In order to include parameters of integer nature into the optimization,
I have extended the algorithm in this way. Internally, all parameters are
continuous. Only before passing them to the objective function, the parameter
values are quantized.
Hard parameter bounds
In my own project, and I guess in most other optimization problems as well,
I always had an idea about the possible range for each optimal parameter. A hard
parameter range has to be given here for every parameter. Using -Inf or Inf as
lower or upper bound is not possible. A side effect of the hard bounds
is that parameter sets including boundary values can be evaluated with higher
probability under certain circumstances.
Breaking using Ctrl-C
When breaking the optimization using Ctrl-C, it might happen that you catch
Matlab just when writing to a file. When you start the optimization again and
the same file needs to be accessed again, the file is locked until you quit
and restart Matlab (at least on Windows).
To avoid the need for using Ctrl-C, a 'time-over'-file is saved in the current
directory. After each function evaluation, the master process checks if the file
is still existing. If it was deleted, the optimization is finished cheerfully.
So instead of pressing Ctrl-C, delete that file.
Shrinking slave support in parallel processing
When processing in parallel, the master process feeds the slaves at the start
of each iteration and collects the results at the end. Further, as all parameter
sets that have been evaluated before are saved, not all parameter vectors in
the current population might need to be evaluated. If the population has already
converged very well, it can happen that there are less new parameter vectors
in the population than processes are working on the optimization problem. In
this case, some or all slaves are sleeping, as there is no more work to do.
Parameter vectors in rows and columns
I always save parameter vectors as column vectors. However, the core
Differential Evolution algorithm taken from the
Differential
Evolution homepage expects parameter vectors to be stored in rows.
Unfortunately, the code now contains a mixture of representations as rows
and columns. You will never get involved with this issue until you edit the
code. Parameter vectors are passed to your objective functions as column vectors.
If your objective function needs to get parameter vectors passed as row vectors,
you will have to write a small wrapper function that transposed the vectors
and calls the objective function in the right way.
Links
Differential Evolution
homepage.
Wikipedia about
Evolutionary Algorithms.
Latest Version of this package on Matlab Central.
The
Multicore package on Matlab Central.
Blat homepage (E-mails from the command line on Windows).
Contact
Dr.-Ing. Markus Buehren
Stuttgart, Germany
http://www.markusbuehren.de
Version
Last modified 20.09.2011
Latest version on Matlab Central.


Build your 360/AR catalog within weeks without the risk of working with one specific studio.
We take our pride in our quality.
In house 3D experts and artists leading the production process including our unique QA approach.
We produced thousands of 3D models and worked with companies featured below
We categorize the models into 3 main complexity tiers of pricing.
We provide the 3d model at any required format including texture maps. The customer will be able to track the progress and quality in realtime via our web interface.
Embed Polyview interactive web viewer in your app/website.


Setup
Software configuration
Database setup
By default, ENM Topic Curation Toolkit Backend expects PostgreSQL to have a user postgres
with the password cFHg*Liw45(
, and a database called nyuotl_db
. If different settings are required, you can change the DATABASES
value in nyu/production_settings.py
, which is just a python dictionary of expected database settings:
DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', 'NAME': 'nyuotl_db', 'USER': 'postgres', 'PASSWORD': 'cFHg*Liw45(', 'HOST': 'localhost', } }
Warning
As you can see from the settings here, the database password is currently stored in cleartext in the repo. Make sure to change the database password in production.
Secret Keys
ENM Topic Curation Toolkit Backend requires a file nyu/secretkeys.json
with the following structure:
{ "SECRET_KEY": "some_long_random_string_of_letters_numbers_and_symbols" }
The secret key is primarily used for hashing passwords, and for security reasons is therefore not kept in the repo. If you transfer a database with user information (such as from a full .sql dump of a database), you must also transfer the secretkeys.json
file to the new installation, or the password hashes will no longer work and user passwords will all have to be manually reset.
Log Files
ENM Topic Curation Toolkit Backend will not run if the expected log file is not present. To create the necessary log files, run
sudo mkdir /var/log/django sudo touch /var/log/django/nyu.log sudo chmod 666 /var/log/django/nyu.log
If you would like to change the location or name of the log file, you can update the LOGGING dictionary in nyu/production_settings.py
Importing Data
There are two ways of getting batch data into the ENM Topic Curation Toolkit Backend:
Database Import
Assuming you have a .sql file which is a full database dump of a currently existing database, import the data means simply running
sudo psql nyuotl_db < DUMP_FILE_NAME.sql
If you have changed the name of the database in the settings file, make sure to use the new database name instead of nyuotl_db
.
Warning
This import process requires an empty database. Running this command on a database with already existing data may result in duplicate data or data corruption.
Batch Ingest
Ingesting a set of EPUBs from scratch requires a little more setup and configuration. If you are starting with a fresh database, you'll first need to create the requisite tables with the following command (run from the folder in which the software was installed):
python manage.py migrate
If you are using the existing set of EPUBs, you'll need to load the IndexPattern data with the following command:
python manage.py loaddata indexpatterns.json
If you are not using the existing set of EPUBs, please see the adding a new EPUB section for instructions on how to configure the system to accept a new EPUB.
Lastly, ensure that the EPUB files to be ingested are in the proper place. That depends on two settings in the nyu/production_settings.py
folder:
MEDIA_ROOT = '/www/nyu.infoloom.nyc/media/' EPUB_SOURCES_FOLDER = os.path.join(MEDIA_ROOT, 'epubs')
The MEDIA_ROOT
folder specifies where all user provided files should be placed on the server, and the EPUB_SOURCES_FOLDER
is a subfolder inside that where all EPUBS to be processed should be placed. Change the folders to whatever location on the server you would like to store the EPUBS, make sure that the application has write-access to those files (for unzipping the EPUBs), and then run the following command:
python manage.py runscript full_batch


The questions that a submitter must answer in their submission
| Name | Description |
|---|---|
| peer_assignment_id | Assignment that this submission schema part lives in |
| peer_assignment_submission_schema_part_id | This part's id within the assignment |
| peer_assignment_submission_schema_part_type | The type ("fileUpload", "plainText", "richText", "url") |
| peer_assignment_submission_schema_part_order | The order within the assignment (lowest comes first) |
| peer_assignment_submission_schema_part_prompt | The prompt (a question for the submitter to answer) |
CREATE TABLE peer_assignment_submission_schema_parts ( peer_assignment_id VARCHAR(50) ,peer_assignment_submission_schema_part_id VARCHAR(50) ,peer_assignment_submission_schema_part_type VARCHAR(50) ,peer_assignment_submission_schema_part_order INT4 ,peer_assignment_submission_schema_part_prompt VARCHAR(65535) ,PRIMARY KEY (peer_assignment_id, peer_assignment_submission_schema_part_id) ,FOREIGN KEY (peer_assignment_id) REFERENCES peer_assignments(peer_assignment_id) );


Hide your navigation options or wathever you want in a nice left panel
Download View project on GithubThis component renders a left side menu like the one we can see in a lot of mobile applications.
In this page we can see the behaviour of the layout clicking in the button at the top left corner close to the navigation options.

 ... Menu content ...

 ... Content ...  ... Place the toggle button wherever you want ... ... More content ...

Stashy.Flyout(selector, options)
| Parameter | Type | Default | Description |
|---|---|---|---|
| selector | string | .st-flyout | CSS selector for your flyout |
| options | object | See options | You can pass the options to the flyout instance |
| Option | Type | Default | Description |
|---|---|---|---|
| slideType | string | push | You can choose between "push" or "reveal" in order to show the panel |
| closeOnClickOutside | bool | true | Close Flyout panel on click outside the toggle button (.st-flyout-toggle) |
| enableTouch | bool | false | If touch is enabled and the device is touch capable, then you can swipe to reveal the left menu. The touch behaviour is controlled by Hammer.js so you need to add a reference to jquery.hammer.js. Get it in the project site or in the dist folder of the Stashy Github project |
| Name | Description |
|---|---|
| layout() | Starts flyout layout |
| open() | Reveal or push menu panel |
| close() | Close menu panel |
var flyout = new Stashy.Flyout().layout();
less
For using just this component, create a .less file similar to Stashy.less, include in order the following imports and compile:
Stashy.Variables.less Stashy.Mixins.less Stashy.Flyout.lessJS
jQuery.js jquery.hammer.js (if touch enabled) Stashy.Flyout.js