참고
Click here to download the full example code
Reinforcement Learning (PPO) with TorchRL Tutorial¶
Author: Vincent Moens
This tutorial demonstrates how to use PyTorch and torchrl
to train a parametric policy
network to solve the Inverted Pendulum task from the OpenAI-Gym/Farama-Gymnasium
control library.
Key learnings:
How to create an environment in TorchRL, transform its outputs, and collect data from this environment;
How to make your classes talk to each other using
tensordict.TensorDict
;The basics of building your training loop with TorchRL:
How to compute the advantage signal for policy gradient methods;
How to create a stochastic policy using a probabilistic neural network;
How to create a dynamic replay buffer and sample from it without repetition.
We will cover six crucial components of TorchRL:
If you are running this in Google Colab, make sure you install the following dependencies:
!pip3 install torchrl
!pip3 install gym[mujoco]
!pip3 install tqdm
Proximal Policy Optimization (PPO) is a policy-gradient algorithm where a batch of data is being collected and directly consumed to train the policy to maximise the expected return given some proximality constraints. You can think of it as a sophisticated version of REINFORCE, the foundational policy-optimization algorithm. For more information, see the Proximal Policy Optimization Algorithms paper.
PPO is usually regarded as a fast and efficient method for online, on-policy reinforcement algorithm. TorchRL provides a loss-module that does all the work for you, so that you can rely on this implementation and focus on solving your problem rather than re-inventing the wheel every time you want to train a policy.
For completeness, here is a brief overview of what the loss computes, even though
this is taken care of by our ClipPPOLoss
module—the algorithm works as follows:
1. we will sample a batch of data by playing the
policy in the environment for a given number of steps.
2. Then, we will perform a given number of optimization steps with random sub-samples of this batch using
a clipped version of the REINFORCE loss.
3. The clipping will put a pessimistic bound on our loss: lower return estimates will
be favored compared to higher ones.
The precise formula of the loss is:
There are two components in that loss: in the first part of the minimum operator, we simply compute an importance-weighted version of the REINFORCE loss (for example, a REINFORCE loss that we have corrected for the fact that the current policy configuration lags the one that was used for the data collection). The second part of that minimum operator is a similar loss where we have clipped the ratios when they exceeded or were below a given pair of thresholds.
This loss ensures that whether the advantage is positive or negative, policy updates that would produce significant shifts from the previous configuration are being discouraged.
This tutorial is structured as follows:
First, we will define a set of hyperparameters we will be using for training.
Next, we will focus on creating our environment, or simulator, using TorchRL’s wrappers and transforms.
Next, we will design the policy network and the value model, which is indispensable to the loss function. These modules will be used to configure our loss module.
Next, we will create the replay buffer and data loader.
Finally, we will run our training loop and analyze the results.
Throughout this tutorial, we’ll be using the tensordict
library.
tensordict.TensorDict
is the lingua franca of TorchRL: it helps us abstract
what a module reads and writes and care less about the specific data
description and more about the algorithm itself.
from collections import defaultdict
import matplotlib.pyplot as plt
import torch
from tensordict.nn import TensorDictModule
from tensordict.nn.distributions import NormalParamExtractor
from torch import nn
from torchrl.collectors import SyncDataCollector
from torchrl.data.replay_buffers import ReplayBuffer
from torchrl.data.replay_buffers.samplers import SamplerWithoutReplacement
from torchrl.data.replay_buffers.storages import LazyTensorStorage
from torchrl.envs import (
Compose,
DoubleToFloat,
ObservationNorm,
StepCounter,
TransformedEnv,
)
from torchrl.envs.libs.gym import GymEnv
from torchrl.envs.utils import check_env_specs, set_exploration_mode
from torchrl.modules import ProbabilisticActor, TanhNormal, ValueOperator
from torchrl.objectives import ClipPPOLoss
from torchrl.objectives.value import GAE
from tqdm import tqdm
Define Hyperparameters¶
We set the hyperparameters for our algorithm. Depending on the resources
available, one may choose to execute the policy on GPU or on another
device.
The frame_skip
will control how for how many frames is a single
action being executed. The rest of the arguments that count frames
must be corrected for this value (since one environment step will
actually return frame_skip
frames).
device = "cpu" if not torch.has_cuda else "cuda:0"
num_cells = 256 # number of cells in each layer
lr = 3e-4
max_grad_norm = 1.0
Data collection parameters¶
When collecting data, we will be able to choose how big each batch will be
by defining a frames_per_batch
parameter. We will also define how many
frames (such as the number of interactions with the simulator) we will allow ourselves to
use. In general, the goal of an RL algorithm is to learn to solve the task
as fast as it can in terms of environment interactions: the lower the total_frames
the better.
We also define a frame_skip
: in some contexts, repeating the same action
multiple times over the course of a trajectory may be beneficial as it makes
the behavior more consistent and less erratic. However, 《skipping》
too many frames will hamper training by reducing the reactivity of the actor
to observation changes.
When using frame_skip
it is good practice to
correct the other frame counts by the number of frames we are grouping
together. If we configure a total count of X frames for training but
use a frame_skip
of Y, we will be actually collecting XY
frames in total
which exceeds our predefined budget.
frame_skip = 1
frames_per_batch = 1000 // frame_skip
# For a complete training, bring the number of frames up to 1M
total_frames = 50_000 // frame_skip
PPO parameters¶
At each data collection (or batch collection) we will run the optimization
over a certain number of epochs, each time consuming the entire data we just
acquired in a nested training loop. Here, the sub_batch_size
is different from the
frames_per_batch
here above: recall that we are working with a 《batch of data》
coming from our collector, which size is defined by frames_per_batch
, and that
we will further split in smaller sub-batches during the inner training loop.
The size of these sub-batches is controlled by sub_batch_size
.
sub_batch_size = 64 # cardinality of the sub-samples gathered from the current data in the inner loop
num_epochs = 10 # optimization steps per batch of data collected
clip_epsilon = (
0.2 # clip value for PPO loss: see the equation in the intro for more context.
)
gamma = 0.99
lmbda = 0.95
entropy_eps = 1e-4
Define an environment¶
In RL, an environment is usually the way we refer to a simulator or a control system. Various libraries provide simulation environments for reinforcement learning, including Gymnasium (previously OpenAI Gym), DeepMind Control Suite, and many others. As a general library, TorchRL’s goal is to provide an interchangeable interface to a large panel of RL simulators, allowing you to easily swap one environment with another. For example, creating a wrapped gym environment can be achieved with few characters:
base_env = GymEnv("InvertedDoublePendulum-v4", device=device, frame_skip=frame_skip)
There are a few things to notice in this code: first, we created
the environment by calling the GymEnv
wrapper. If extra keyword arguments
are passed, they will be transmitted to the gym.make
method, hence covering
the most common environment construction commands.
Alternatively, one could also directly create a gym environment using gym.make(env_name, **kwargs)
and wrap it in a GymWrapper class.
Also the device
argument: for gym, this only controls the device where
input action and observed states will be stored, but the execution will always
be done on CPU. The reason for this is simply that gym does not support on-device
execution, unless specified otherwise. For other libraries, we have control over
the execution device and, as much as we can, we try to stay consistent in terms of
storing and execution backends.
Transforms¶
We will append some transforms to our environments to prepare the data for
the policy. In Gym, this is usually achieved via wrappers. TorchRL takes a different
approach, more similar to other pytorch domain libraries, through the use of transforms.
To add transforms to an environment, one should simply wrap it in a TransformedEnv
instance, and append the sequence of transforms to it. The transformed environment will inherit
the device and meta-data of the wrapped environment, and transform these depending on the sequence
of transforms it contains.
Normalization¶
The first to encode is a normalization transform. As a rule of thumbs, it is preferable to have data that loosely match a unit Gaussian distribution: to obtain this, we will run a certain number of random steps in the environment and compute the summary statistics of these observations.
We’ll append two other transforms: the DoubleToFloat
transform will
convert double entries to single-precision numbers, ready to be read by the
policy. The StepCounter
transform will be used to count the steps before
the environment is terminated. We will use this measure as a supplementary measure
of performance.
As we will see later, many of the TorchRL’s classes rely on tensordict.TensorDict
to communicate. You could think of it as a python dictionary with some extra
tensor features. In practice, this means that many modules we will be working
with need to be told what key to read (in_keys
) and what key to write
(out_keys
) in the tensordict
they will receive. Usually, if out_keys
is omitted, it is assumed that the in_keys
entries will be updated
in-place. For our transforms, the only entry we are interested in is referred
to as "observation"
and our transform layers will be told to modify this
entry and this entry only:
env = TransformedEnv(
base_env,
Compose(
# normalize observations
ObservationNorm(in_keys=["observation"]),
DoubleToFloat(in_keys=["observation"]),
StepCounter(),
),
)
As you may have noticed, we have created a normalization layer but we did not
set its normalization parameters. To do this, ObservationNorm
can
automatically gather the summary statistics of our environment:
env.transform[0].init_stats(num_iter=1000, reduce_dim=0, cat_dim=0)
The ObservationNorm
transform has now been populated with a
location and a scale that will be used to normalize the data.
Let us do a little sanity check for the shape of our summary stats:
print("normalization constant shape:", env.transform[0].loc.shape)
normalization constant shape: torch.Size([11])
An environment is not only defined by its simulator and transforms, but also
by a series of metadata that describe what can be expected during its
execution.
For efficiency purposes, TorchRL is quite stringent when it comes to
environment specs, but you can easily check that your environment specs are
adequate.
In our example, the GymWrapper
and GymEnv
that inherits
from it already take care of setting the proper specs for your environment so
you should not have to care about this.
Nevertheless, let’s see a concrete example using our transformed
environment by looking at its specs.
There are three specs to look at: observation_spec
which defines what
is to be expected when executing an action in the environment,
reward_spec
which indicates the reward domain and finally the
input_spec
(which contains the action_spec
) and which represents
everything an environment requires to execute a single step.
print("observation_spec:", env.observation_spec)
print("reward_spec:", env.reward_spec)
print("input_spec:", env.input_spec)
print("action_spec (as defined by input_spec):", env.action_spec)
observation_spec: CompositeSpec(
observation: UnboundedContinuousTensorSpec(
shape=torch.Size([11]), space=None, device=cuda:0, dtype=torch.float32, domain=continuous),
step_count: UnboundedDiscreteTensorSpec(
shape=torch.Size([]), space=ContinuousBox(minimum=Tensor(shape=torch.Size([]), device=cuda:0, dtype=torch.int64, contiguous=True), maximum=Tensor(shape=torch.Size([]), device=cuda:0, dtype=torch.int64, contiguous=True)), device=cuda:0, dtype=torch.int64, domain=continuous), device=cuda:0, shape=torch.Size([]))
reward_spec: UnboundedContinuousTensorSpec(
shape=torch.Size([1]), space=ContinuousBox(minimum=Tensor(shape=torch.Size([]), device=cpu, dtype=torch.float32, contiguous=True), maximum=Tensor(shape=torch.Size([]), device=cpu, dtype=torch.float32, contiguous=True)), device=cuda:0, dtype=torch.float32, domain=continuous)
input_spec: CompositeSpec(
action: BoundedTensorSpec(
shape=torch.Size([1]), space=ContinuousBox(minimum=Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, contiguous=True), maximum=Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, contiguous=True)), device=cuda:0, dtype=torch.float32, domain=continuous),
step_count: UnboundedDiscreteTensorSpec(
shape=torch.Size([]), space=ContinuousBox(minimum=Tensor(shape=torch.Size([]), device=cuda:0, dtype=torch.int64, contiguous=True), maximum=Tensor(shape=torch.Size([]), device=cuda:0, dtype=torch.int64, contiguous=True)), device=cuda:0, dtype=torch.int64, domain=continuous), device=cuda:0, shape=torch.Size([]))
action_spec (as defined by input_spec): BoundedTensorSpec(
shape=torch.Size([1]), space=ContinuousBox(minimum=Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, contiguous=True), maximum=Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, contiguous=True)), device=cuda:0, dtype=torch.float32, domain=continuous)
the check_env_specs()
function runs a small rollout and compares its output against the environment
specs. If no error is raised, we can be confident that the specs are properly defined:
check_env_specs(env)
check_env_specs succeeded!
For fun, let’s see what a simple random rollout looks like. You can call env.rollout(n_steps) and get an overview of what the environment inputs and outputs look like. Actions will automatically be drawn from the action spec domain, so you don’t need to care about designing a random sampler.
Typically, at each step, an RL environment receives an
action as input, and outputs an observation, a reward and a done state. The
observation may be composite, meaning that it could be composed of more than one
tensor. This is not a problem for TorchRL, since the whole set of observations
is automatically packed in the output tensordict.TensorDict
. After executing a rollout
(for example, a sequence of environment steps and random action generations) over a given
number of steps, we will retrieve a tensordict.TensorDict
instance with a shape
that matches this trajectory length:
rollout = env.rollout(3)
print("rollout of three steps:", rollout)
print("Shape of the rollout TensorDict:", rollout.batch_size)
rollout of three steps: TensorDict(
fields={
action: Tensor(shape=torch.Size([3, 1]), device=cuda:0, dtype=torch.float32, is_shared=True),
done: Tensor(shape=torch.Size([3, 1]), device=cuda:0, dtype=torch.bool, is_shared=True),
next: TensorDict(
fields={
done: Tensor(shape=torch.Size([3, 1]), device=cuda:0, dtype=torch.bool, is_shared=True),
observation: Tensor(shape=torch.Size([3, 11]), device=cuda:0, dtype=torch.float32, is_shared=True),
reward: Tensor(shape=torch.Size([3, 1]), device=cuda:0, dtype=torch.float32, is_shared=True),
step_count: Tensor(shape=torch.Size([3]), device=cuda:0, dtype=torch.int64, is_shared=True)},
batch_size=torch.Size([3]),
device=cuda:0,
is_shared=True),
observation: Tensor(shape=torch.Size([3, 11]), device=cuda:0, dtype=torch.float32, is_shared=True),
reward: Tensor(shape=torch.Size([3, 1]), device=cuda:0, dtype=torch.float32, is_shared=True),
step_count: Tensor(shape=torch.Size([3]), device=cuda:0, dtype=torch.int64, is_shared=True)},
batch_size=torch.Size([3]),
device=cuda:0,
is_shared=True)
Shape of the rollout TensorDict: torch.Size([3])
Our rollout data has a shape of torch.Size([3])`, which matches the number of steps
we ran it for. The ``"next"
entry points to the data coming after the current step.
In most cases, the "next""
data at time t matches the data at t+1
, but this
may not be the case if we are using some specific transformations (for example, multi-step).
Policy¶
PPO utilizes a stochastic policy to handle exploration. This means that our neural network will have to output the parameters of a distribution, rather than a single value corresponding to the action taken.
As the data is continuous, we use a Tanh-Normal distribution to respect the action space boundaries. TorchRL provides such distribution, and the only thing we need to care about is to build a neural network that outputs the right number of parameters for the policy to work with (a location, or mean, and a scale):
The only extra-difficulty that is brought up here is to split our output in two equal parts and map the second to a strictly positive space.
We design the policy in three steps:
Define a neural network
D_obs
->2 * D_action
. Indeed, ourloc
(mu) andscale
(sigma) both have dimensionD_action
;Append a
NormalParamExtractor
to extract a location and a scale (for example, splits the input in two equal parts
and applies a positive transformation to the scale parameter);
Create a probabilistic
TensorDictModule
that can create this distribution and sample from it.
actor_net = nn.Sequential(
nn.LazyLinear(num_cells, device=device),
nn.Tanh(),
nn.LazyLinear(num_cells, device=device),
nn.Tanh(),
nn.LazyLinear(num_cells, device=device),
nn.Tanh(),
nn.LazyLinear(2 * env.action_spec.shape[-1], device=device),
NormalParamExtractor(),
)
To enable the policy to 《talk》 with the environment through the tensordict
data carrier, we wrap the nn.Module
in a TensorDictModule
. This
class will simply ready the in_keys
it is provided with and write the
outputs in-place at the registered out_keys
.
policy_module = TensorDictModule(
actor_net, in_keys=["observation"], out_keys=["loc", "scale"]
)
We now need to build a distribution out of the location and scale of our
normal distribution. To do so, we instruct the ProbabilisticActor
class to build a TanhNormal
out of the location and scale
parameters. We also provide the minimum and maximum values of this
distribution, which we gather from the environment specs.
The name of the in_keys
(and hence the name of the out_keys
from
the TensorDictModule
above) cannot be set to any value one may
like, as the TanhNormal
distribution constructor will expect the
loc
and scale
keyword arguments. That being said,
ProbabilisticActor
also accepts Dict[str, str]
typed in_keys
where the key-value pair indicates what in_key
string should be used for
every keyword argument that is to be used.
policy_module = ProbabilisticActor(
module=policy_module,
spec=env.action_spec,
in_keys=["loc", "scale"],
distribution_class=TanhNormal,
distribution_kwargs={
"min": env.action_spec.space.minimum,
"max": env.action_spec.space.maximum,
},
return_log_prob=True,
# we'll need the log-prob for the numerator of the importance weights
)
Value network¶
The value network is a crucial component of the PPO algorithm, even though it won’t be used at inference time. This module will read the observations and return an estimation of the discounted return for the following trajectory. This allows us to amortize learning by relying on the some utility estimation that is learned on-the-fly during training. Our value network share the same structure as the policy, but for simplicity we assign it its own set of parameters.
value_net = nn.Sequential(
nn.LazyLinear(num_cells, device=device),
nn.Tanh(),
nn.LazyLinear(num_cells, device=device),
nn.Tanh(),
nn.LazyLinear(num_cells, device=device),
nn.Tanh(),
nn.LazyLinear(1, device=device),
)
value_module = ValueOperator(
module=value_net,
in_keys=["observation"],
)
let’s try our policy and value modules. As we said earlier, the usage of
TensorDictModule
makes it possible to directly read the output
of the environment to run these modules, as they know what information to read
and where to write it:
print("Running policy:", policy_module(env.reset()))
print("Running value:", value_module(env.reset()))
Running policy: TensorDict(
fields={
action: Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, is_shared=True),
done: Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.bool, is_shared=True),
loc: Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, is_shared=True),
observation: Tensor(shape=torch.Size([11]), device=cuda:0, dtype=torch.float32, is_shared=True),
reward: Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, is_shared=True),
sample_log_prob: Tensor(shape=torch.Size([]), device=cuda:0, dtype=torch.float32, is_shared=True),
scale: Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, is_shared=True),
step_count: Tensor(shape=torch.Size([]), device=cuda:0, dtype=torch.int64, is_shared=True)},
batch_size=torch.Size([]),
device=cuda:0,
is_shared=True)
Running value: TensorDict(
fields={
done: Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.bool, is_shared=True),
observation: Tensor(shape=torch.Size([11]), device=cuda:0, dtype=torch.float32, is_shared=True),
reward: Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, is_shared=True),
state_value: Tensor(shape=torch.Size([1]), device=cuda:0, dtype=torch.float32, is_shared=True),
step_count: Tensor(shape=torch.Size([]), device=cuda:0, dtype=torch.int64, is_shared=True)},
batch_size=torch.Size([]),
device=cuda:0,
is_shared=True)
Data collector¶
TorchRL provides a set of DataCollector
classes. Briefly, these
classes execute three operations: reset an environment, compute an action
given the latest observation, execute a step in the environment, and repeat
the last two steps until the environment reaches a stop signal (or "done"
state).
They allow you to control how many frames to collect at each iteration
(through the frames_per_batch
parameter),
when to reset the environment (through the max_frames_per_traj
argument),
on which device
the policy should be executed, etc. They are also
designed to work efficiently with batched and multiprocessed environments.
The simplest data collector is the SyncDataCollector
: it is an
iterator that you can use to get batches of data of a given length, and
that will stop once a total number of frames (total_frames
) have been
collected.
Other data collectors (MultiSyncDataCollector
and
MultiaSyncDataCollector
) will execute the same operations in synchronous
and asynchronous manner over a set of multiprocessed workers.
As for the policy and environment before, the data collector will return
tensordict.TensorDict
instances with a total number of elements that will
match frames_per_batch
. Using tensordict.TensorDict
to pass data to the
training loop allows you to write data loading pipelines
that are 100% oblivious to the actual specificities of the rollout content.
collector = SyncDataCollector(
env,
policy_module,
frames_per_batch=frames_per_batch,
total_frames=total_frames,
split_trajs=False,
device=device,
)
Replay buffer¶
Replay buffers are a common building piece of off-policy RL algorithms. In on-policy contexts, a replay buffer is refilled every time a batch of data is collected, and its data is repeatedly consumed for a certain number of epochs.
TorchRL’s replay buffers are built using a common container
ReplayBuffer
which takes as argument the components of the buffer:
a storage, a writer, a sampler and possibly some transforms. Only the
storage (which indicates the replay buffer capacity) is mandatory. We
also specify a sampler without repetition to avoid sampling multiple times
the same item in one epoch.
Using a replay buffer for PPO is not mandatory and we could simply
sample the sub-batches from the collected batch, but using these classes
make it easy for us to build the inner training loop in a reproducible way.
replay_buffer = ReplayBuffer(
storage=LazyTensorStorage(frames_per_batch),
sampler=SamplerWithoutReplacement(),
)
Loss function¶
The PPO loss can be directly imported from TorchRL for convenience using the
ClipPPOLoss
class. This is the easiest way of utilizing PPO:
it hides away the mathematical operations of PPO and the control flow that
goes with it.
PPO requires some 《advantage estimation》 to be computed. In short, an advantage
is a value that reflects an expectancy over the return value while dealing with
the bias / variance tradeoff.
To compute the advantage, one just needs to (1) build the advantage module, which
utilizes our value operator, and (2) pass each batch of data through it before each
epoch.
The GAE module will update the input tensordict
with new "advantage"
and
"value_target"
entries.
The "value_target"
is a gradient-free tensor that represents the empirical
value that the value network should represent with the input observation.
Both of these will be used by ClipPPOLoss
to
return the policy and value losses.
advantage_module = GAE(
gamma=gamma, lmbda=lmbda, value_network=value_module, average_gae=True
)
loss_module = ClipPPOLoss(
actor=policy_module,
critic=value_module,
advantage_key="advantage",
clip_epsilon=clip_epsilon,
entropy_bonus=bool(entropy_eps),
entropy_coef=entropy_eps,
# these keys match by default but we set this for completeness
value_target_key=advantage_module.value_target_key,
critic_coef=1.0,
gamma=0.99,
loss_critic_type="smooth_l1",
)
optim = torch.optim.Adam(loss_module.parameters(), lr)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
optim, total_frames // frames_per_batch, 0.0
)
Training loop¶
We now have all the pieces needed to code our training loop. The steps include:
Collect data
Compute advantage
Loop over the collected to compute loss values
Back propagate
Optimize
Repeat
Repeat
Repeat
logs = defaultdict(list)
pbar = tqdm(total=total_frames * frame_skip)
eval_str = ""
# We iterate over the collector until it reaches the total number of frames it was
# designed to collect:
for i, tensordict_data in enumerate(collector):
# we now have a batch of data to work with. Let's learn something from it.
for _ in range(num_epochs):
# We'll need an "advantage" signal to make PPO work.
# We re-compute it at each epoch as its value depends on the value
# network which is updated in the inner loop.
advantage_module(tensordict_data)
data_view = tensordict_data.reshape(-1)
replay_buffer.extend(data_view.cpu())
for _ in range(frames_per_batch // sub_batch_size):
subdata = replay_buffer.sample(sub_batch_size)
loss_vals = loss_module(subdata.to(device))
loss_value = (
loss_vals["loss_objective"]
+ loss_vals["loss_critic"]
+ loss_vals["loss_entropy"]
)
# Optimization: backward, grad clipping and optimization step
loss_value.backward()
# this is not strictly mandatory but it's good practice to keep
# your gradient norm bounded
torch.nn.utils.clip_grad_norm_(loss_module.parameters(), max_grad_norm)
optim.step()
optim.zero_grad()
logs["reward"].append(tensordict_data["next", "reward"].mean().item())
pbar.update(tensordict_data.numel() * frame_skip)
cum_reward_str = (
f"average reward={logs['reward'][-1]: 4.4f} (init={logs['reward'][0]: 4.4f})"
)
logs["step_count"].append(tensordict_data["step_count"].max().item())
stepcount_str = f"step count (max): {logs['step_count'][-1]}"
logs["lr"].append(optim.param_groups[0]["lr"])
lr_str = f"lr policy: {logs['lr'][-1]: 4.4f}"
if i % 10 == 0:
# We evaluate the policy once every 10 batches of data.
# Evaluation is rather simple: execute the policy without exploration
# (take the expected value of the action distribution) for a given
# number of steps (1000, which is our ``env`` horizon).
# The ``rollout`` method of the ``env`` can take a policy as argument:
# it will then execute this policy at each step.
with set_exploration_mode("mean"), torch.no_grad():
# execute a rollout with the trained policy
eval_rollout = env.rollout(1000, policy_module)
logs["eval reward"].append(eval_rollout["next", "reward"].mean().item())
logs["eval reward (sum)"].append(
eval_rollout["next", "reward"].sum().item()
)
logs["eval step_count"].append(eval_rollout["step_count"].max().item())
eval_str = (
f"eval cumulative reward: {logs['eval reward (sum)'][-1]: 4.4f} "
f"(init: {logs['eval reward (sum)'][0]: 4.4f}), "
f"eval step-count: {logs['eval step_count'][-1]}"
)
del eval_rollout
pbar.set_description(", ".join([eval_str, cum_reward_str, stepcount_str, lr_str]))
# We're also using a learning rate scheduler. Like the gradient clipping,
# this is a nice-to-have but nothing necessary for PPO to work.
scheduler.step()
0%| | 0/50000 [00:00<?, ?it/s]
2%|2 | 1000/50000 [00:06<05:31, 147.74it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.0858 (init= 9.0858), step count (max): 12, lr policy: 0.0003: 2%|2 | 1000/50000 [00:06<05:31, 147.74it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.0858 (init= 9.0858), step count (max): 12, lr policy: 0.0003: 4%|4 | 2000/50000 [00:13<05:21, 149.41it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.1470 (init= 9.0858), step count (max): 17, lr policy: 0.0003: 4%|4 | 2000/50000 [00:13<05:21, 149.41it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.1470 (init= 9.0858), step count (max): 17, lr policy: 0.0003: 6%|6 | 3000/50000 [00:19<05:10, 151.15it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.1712 (init= 9.0858), step count (max): 23, lr policy: 0.0003: 6%|6 | 3000/50000 [00:19<05:10, 151.15it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.1712 (init= 9.0858), step count (max): 23, lr policy: 0.0003: 8%|8 | 4000/50000 [00:26<05:06, 150.20it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.1892 (init= 9.0858), step count (max): 27, lr policy: 0.0003: 8%|8 | 4000/50000 [00:26<05:06, 150.20it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.1892 (init= 9.0858), step count (max): 27, lr policy: 0.0003: 10%|# | 5000/50000 [00:33<04:57, 151.49it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2174 (init= 9.0858), step count (max): 31, lr policy: 0.0003: 10%|# | 5000/50000 [00:33<04:57, 151.49it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2174 (init= 9.0858), step count (max): 31, lr policy: 0.0003: 12%|#2 | 6000/50000 [00:39<04:51, 151.18it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2201 (init= 9.0858), step count (max): 29, lr policy: 0.0003: 12%|#2 | 6000/50000 [00:39<04:51, 151.18it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2201 (init= 9.0858), step count (max): 29, lr policy: 0.0003: 14%|#4 | 7000/50000 [00:46<04:43, 151.78it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2346 (init= 9.0858), step count (max): 34, lr policy: 0.0003: 14%|#4 | 7000/50000 [00:46<04:43, 151.78it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2346 (init= 9.0858), step count (max): 34, lr policy: 0.0003: 16%|#6 | 8000/50000 [00:52<04:34, 153.25it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2351 (init= 9.0858), step count (max): 53, lr policy: 0.0003: 16%|#6 | 8000/50000 [00:52<04:34, 153.25it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2351 (init= 9.0858), step count (max): 53, lr policy: 0.0003: 18%|#8 | 9000/50000 [00:59<04:25, 154.23it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2392 (init= 9.0858), step count (max): 42, lr policy: 0.0003: 18%|#8 | 9000/50000 [00:59<04:25, 154.23it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2392 (init= 9.0858), step count (max): 42, lr policy: 0.0003: 20%|## | 10000/50000 [01:05<04:18, 154.67it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2424 (init= 9.0858), step count (max): 66, lr policy: 0.0003: 20%|## | 10000/50000 [01:05<04:18, 154.67it/s]
eval cumulative reward: 111.1220 (init: 111.1220), eval step-count: 11, average reward= 9.2424 (init= 9.0858), step count (max): 66, lr policy: 0.0003: 22%|##2 | 11000/50000 [01:11<04:10, 155.45it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2643 (init= 9.0858), step count (max): 55, lr policy: 0.0003: 22%|##2 | 11000/50000 [01:12<04:10, 155.45it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2643 (init= 9.0858), step count (max): 55, lr policy: 0.0003: 24%|##4 | 12000/50000 [01:18<04:07, 153.67it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2645 (init= 9.0858), step count (max): 55, lr policy: 0.0003: 24%|##4 | 12000/50000 [01:18<04:07, 153.67it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2645 (init= 9.0858), step count (max): 55, lr policy: 0.0003: 26%|##6 | 13000/50000 [01:25<04:01, 153.37it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2679 (init= 9.0858), step count (max): 57, lr policy: 0.0003: 26%|##6 | 13000/50000 [01:25<04:01, 153.37it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2679 (init= 9.0858), step count (max): 57, lr policy: 0.0003: 28%|##8 | 14000/50000 [01:31<03:52, 154.66it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2747 (init= 9.0858), step count (max): 70, lr policy: 0.0003: 28%|##8 | 14000/50000 [01:31<03:52, 154.66it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2747 (init= 9.0858), step count (max): 70, lr policy: 0.0003: 30%|### | 15000/50000 [01:38<03:48, 153.40it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2788 (init= 9.0858), step count (max): 81, lr policy: 0.0002: 30%|### | 15000/50000 [01:38<03:48, 153.40it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2788 (init= 9.0858), step count (max): 81, lr policy: 0.0002: 32%|###2 | 16000/50000 [01:44<03:41, 153.48it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2868 (init= 9.0858), step count (max): 73, lr policy: 0.0002: 32%|###2 | 16000/50000 [01:44<03:41, 153.48it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2868 (init= 9.0858), step count (max): 73, lr policy: 0.0002: 34%|###4 | 17000/50000 [01:50<03:33, 154.81it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2838 (init= 9.0858), step count (max): 62, lr policy: 0.0002: 34%|###4 | 17000/50000 [01:50<03:33, 154.81it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2838 (init= 9.0858), step count (max): 62, lr policy: 0.0002: 36%|###6 | 18000/50000 [01:57<03:26, 155.30it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2867 (init= 9.0858), step count (max): 68, lr policy: 0.0002: 36%|###6 | 18000/50000 [01:57<03:26, 155.30it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2867 (init= 9.0858), step count (max): 68, lr policy: 0.0002: 38%|###8 | 19000/50000 [02:03<03:18, 155.80it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2785 (init= 9.0858), step count (max): 60, lr policy: 0.0002: 38%|###8 | 19000/50000 [02:03<03:18, 155.80it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2785 (init= 9.0858), step count (max): 60, lr policy: 0.0002: 40%|#### | 20000/50000 [02:10<03:12, 156.02it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2860 (init= 9.0858), step count (max): 85, lr policy: 0.0002: 40%|#### | 20000/50000 [02:10<03:12, 156.02it/s]
eval cumulative reward: 354.0260 (init: 111.1220), eval step-count: 37, average reward= 9.2860 (init= 9.0858), step count (max): 85, lr policy: 0.0002: 42%|####2 | 21000/50000 [02:16<03:05, 156.35it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2891 (init= 9.0858), step count (max): 63, lr policy: 0.0002: 42%|####2 | 21000/50000 [02:16<03:05, 156.35it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2891 (init= 9.0858), step count (max): 63, lr policy: 0.0002: 44%|####4 | 22000/50000 [02:23<03:00, 154.99it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2968 (init= 9.0858), step count (max): 92, lr policy: 0.0002: 44%|####4 | 22000/50000 [02:23<03:00, 154.99it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2968 (init= 9.0858), step count (max): 92, lr policy: 0.0002: 46%|####6 | 23000/50000 [02:29<02:53, 155.52it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.3114 (init= 9.0858), step count (max): 75, lr policy: 0.0002: 46%|####6 | 23000/50000 [02:29<02:53, 155.52it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.3114 (init= 9.0858), step count (max): 75, lr policy: 0.0002: 48%|####8 | 24000/50000 [02:35<02:46, 155.72it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2850 (init= 9.0858), step count (max): 106, lr policy: 0.0002: 48%|####8 | 24000/50000 [02:35<02:46, 155.72it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2850 (init= 9.0858), step count (max): 106, lr policy: 0.0002: 50%|##### | 25000/50000 [02:42<02:40, 155.58it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2743 (init= 9.0858), step count (max): 54, lr policy: 0.0002: 50%|##### | 25000/50000 [02:42<02:40, 155.58it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2743 (init= 9.0858), step count (max): 54, lr policy: 0.0002: 52%|#####2 | 26000/50000 [02:48<02:34, 155.15it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2955 (init= 9.0858), step count (max): 79, lr policy: 0.0001: 52%|#####2 | 26000/50000 [02:48<02:34, 155.15it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2955 (init= 9.0858), step count (max): 79, lr policy: 0.0001: 54%|#####4 | 27000/50000 [02:55<02:27, 155.66it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2973 (init= 9.0858), step count (max): 77, lr policy: 0.0001: 54%|#####4 | 27000/50000 [02:55<02:27, 155.66it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2973 (init= 9.0858), step count (max): 77, lr policy: 0.0001: 56%|#####6 | 28000/50000 [03:01<02:20, 156.05it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2914 (init= 9.0858), step count (max): 67, lr policy: 0.0001: 56%|#####6 | 28000/50000 [03:01<02:20, 156.05it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2914 (init= 9.0858), step count (max): 67, lr policy: 0.0001: 58%|#####8 | 29000/50000 [03:08<02:15, 155.03it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2907 (init= 9.0858), step count (max): 60, lr policy: 0.0001: 58%|#####8 | 29000/50000 [03:08<02:15, 155.03it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2907 (init= 9.0858), step count (max): 60, lr policy: 0.0001: 60%|###### | 30000/50000 [03:14<02:08, 155.50it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2787 (init= 9.0858), step count (max): 76, lr policy: 0.0001: 60%|###### | 30000/50000 [03:14<02:08, 155.50it/s]
eval cumulative reward: 437.5383 (init: 111.1220), eval step-count: 46, average reward= 9.2787 (init= 9.0858), step count (max): 76, lr policy: 0.0001: 62%|######2 | 31000/50000 [03:21<02:03, 153.86it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3022 (init= 9.0858), step count (max): 135, lr policy: 0.0001: 62%|######2 | 31000/50000 [03:21<02:03, 153.86it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3022 (init= 9.0858), step count (max): 135, lr policy: 0.0001: 64%|######4 | 32000/50000 [03:27<01:58, 151.36it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3077 (init= 9.0858), step count (max): 141, lr policy: 0.0001: 64%|######4 | 32000/50000 [03:27<01:58, 151.36it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3077 (init= 9.0858), step count (max): 141, lr policy: 0.0001: 66%|######6 | 33000/50000 [03:34<01:52, 150.50it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.2934 (init= 9.0858), step count (max): 83, lr policy: 0.0001: 66%|######6 | 33000/50000 [03:34<01:52, 150.50it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.2934 (init= 9.0858), step count (max): 83, lr policy: 0.0001: 68%|######8 | 34000/50000 [03:41<01:44, 152.52it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3033 (init= 9.0858), step count (max): 124, lr policy: 0.0001: 68%|######8 | 34000/50000 [03:41<01:44, 152.52it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3033 (init= 9.0858), step count (max): 124, lr policy: 0.0001: 70%|####### | 35000/50000 [03:47<01:37, 153.10it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.2947 (init= 9.0858), step count (max): 115, lr policy: 0.0001: 70%|####### | 35000/50000 [03:47<01:37, 153.10it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.2947 (init= 9.0858), step count (max): 115, lr policy: 0.0001: 72%|#######2 | 36000/50000 [03:53<01:31, 153.80it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3124 (init= 9.0858), step count (max): 125, lr policy: 0.0001: 72%|#######2 | 36000/50000 [03:53<01:31, 153.80it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3124 (init= 9.0858), step count (max): 125, lr policy: 0.0001: 74%|#######4 | 37000/50000 [04:00<01:24, 153.31it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3116 (init= 9.0858), step count (max): 135, lr policy: 0.0001: 74%|#######4 | 37000/50000 [04:00<01:24, 153.31it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3116 (init= 9.0858), step count (max): 135, lr policy: 0.0001: 76%|#######6 | 38000/50000 [04:06<01:17, 154.78it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3100 (init= 9.0858), step count (max): 141, lr policy: 0.0000: 76%|#######6 | 38000/50000 [04:06<01:17, 154.78it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3100 (init= 9.0858), step count (max): 141, lr policy: 0.0000: 78%|#######8 | 39000/50000 [04:13<01:10, 155.75it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3141 (init= 9.0858), step count (max): 166, lr policy: 0.0000: 78%|#######8 | 39000/50000 [04:13<01:10, 155.75it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3141 (init= 9.0858), step count (max): 166, lr policy: 0.0000: 80%|######## | 40000/50000 [04:19<01:03, 156.52it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3133 (init= 9.0858), step count (max): 150, lr policy: 0.0000: 80%|######## | 40000/50000 [04:19<01:03, 156.52it/s]
eval cumulative reward: 1083.0696 (init: 111.1220), eval step-count: 115, average reward= 9.3133 (init= 9.0858), step count (max): 150, lr policy: 0.0000: 82%|########2 | 41000/50000 [04:25<00:57, 157.23it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3172 (init= 9.0858), step count (max): 158, lr policy: 0.0000: 82%|########2 | 41000/50000 [04:26<00:57, 157.23it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3172 (init= 9.0858), step count (max): 158, lr policy: 0.0000: 84%|########4 | 42000/50000 [04:33<00:53, 149.76it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3292 (init= 9.0858), step count (max): 230, lr policy: 0.0000: 84%|########4 | 42000/50000 [04:33<00:53, 149.76it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3292 (init= 9.0858), step count (max): 230, lr policy: 0.0000: 86%|########6 | 43000/50000 [04:39<00:46, 150.83it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3447 (init= 9.0858), step count (max): 494, lr policy: 0.0000: 86%|########6 | 43000/50000 [04:39<00:46, 150.83it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3447 (init= 9.0858), step count (max): 494, lr policy: 0.0000: 88%|########8 | 44000/50000 [04:46<00:39, 152.16it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3517 (init= 9.0858), step count (max): 811, lr policy: 0.0000: 88%|########8 | 44000/50000 [04:46<00:39, 152.16it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3517 (init= 9.0858), step count (max): 811, lr policy: 0.0000: 90%|######### | 45000/50000 [04:52<00:32, 153.67it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3459 (init= 9.0858), step count (max): 455, lr policy: 0.0000: 90%|######### | 45000/50000 [04:52<00:32, 153.67it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3459 (init= 9.0858), step count (max): 455, lr policy: 0.0000: 92%|#########2| 46000/50000 [04:58<00:25, 154.43it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3563 (init= 9.0858), step count (max): 999, lr policy: 0.0000: 92%|#########2| 46000/50000 [04:58<00:25, 154.43it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3563 (init= 9.0858), step count (max): 999, lr policy: 0.0000: 94%|#########3| 47000/50000 [05:05<00:19, 154.85it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3478 (init= 9.0858), step count (max): 709, lr policy: 0.0000: 94%|#########3| 47000/50000 [05:05<00:19, 154.85it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3478 (init= 9.0858), step count (max): 709, lr policy: 0.0000: 96%|#########6| 48000/50000 [05:11<00:12, 155.49it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3389 (init= 9.0858), step count (max): 421, lr policy: 0.0000: 96%|#########6| 48000/50000 [05:11<00:12, 155.49it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3389 (init= 9.0858), step count (max): 421, lr policy: 0.0000: 98%|#########8| 49000/50000 [05:18<00:06, 154.04it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3391 (init= 9.0858), step count (max): 451, lr policy: 0.0000: 98%|#########8| 49000/50000 [05:18<00:06, 154.04it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3391 (init= 9.0858), step count (max): 451, lr policy: 0.0000: 100%|##########| 50000/50000 [05:24<00:00, 154.83it/s]
eval cumulative reward: 2523.4971 (init: 111.1220), eval step-count: 269, average reward= 9.3486 (init= 9.0858), step count (max): 732, lr policy: 0.0000: 100%|##########| 50000/50000 [05:24<00:00, 154.83it/s]
Results¶
Before the 1M step cap is reached, the algorithm should have reached a max step count of 1000 steps, which is the maximum number of steps before the trajectory is truncated.
plt.figure(figsize=(10, 10))
plt.subplot(2, 2, 1)
plt.plot(logs["reward"])
plt.title("training rewards (average)")
plt.subplot(2, 2, 2)
plt.plot(logs["step_count"])
plt.title("Max step count (training)")
plt.subplot(2, 2, 3)
plt.plot(logs["eval reward (sum)"])
plt.title("Return (test)")
plt.subplot(2, 2, 4)
plt.plot(logs["eval step_count"])
plt.title("Max step count (test)")
plt.show()
Conclusion and next steps¶
In this tutorial, we have learned:
How to create and customize an environment with
torchrl
;How to write a model and a loss function;
How to set up a typical training loop.
If you want to experiment with this tutorial a bit more, you can apply the following modifications:
From an efficiency perspective, we could run several simulations in parallel to speed up data collection. Check
torchrl.envs.ParallelEnv
for further information.From a logging perspective, one could add a
torchrl.record.VideoRecorder
transform to the environment after asking for rendering to get a visual rendering of the inverted pendulum in action. Checktorchrl.record
to know more.
Total running time of the script: ( 5 minutes 28.793 seconds)


TATA PRASHIKSHAN
Shavak Nanavati Technical Institute (SNTI), the erstwhile "Jamshedpur Technical Institute" (JTI) was established in 1921, with the inspiration of the Founder "Let the Indians learn to do things by themselves". Today SNTI, an integral part of the Human Resource Division of Tata Steel, has rendered commendable service in nation building through the development of technical manpower, not only for Tata Steel but, for many other Tata organisations, private and public sector Steel Plants and manufacturing industries.
"Career at Tata Steel gives you the opportunity to be part of an organisation, which has always been ahead of the times: a combination of blistering pace of growth and unflinching values. A career with us is more than just a job; it is an opportunity to join a company, which has woven itself into the social fabric around it."
Products and Services
Further, SNTI can also design and deliver customized training programme for SMEs at SNTI or at the client's premises. SNTI has identified the following programmes exclusively for entrepreneurs located in Adityapur
SNTI conducts training programmes under the following categories -
i. Pre-Employment training : Training for the cadres at various levels like Management Trainees, Junior Engineer Trainees, Trade Apprentice Trainees and other new joinees.
ii. Post employment training (Day) : Training for those who are already working in Tata Steel and are nominated by respective departments for knowledge and skill development in their respective work areas. Such training programmes are planned annually, based on requirement of the respective departments. The employees are withdrawn from the departments for this purpose. 'Thrust Area' programmes are also conducted with a focus on specific topics or requirement as indicated by the departments. Some growth related competency development programmes are also organized under this category.
iii. Post employment training (Evening) : imparted to the supervisory/ associate category of employees basically for their competency development and qualification enhancement. Such programmes are conducted off-duty hours on original and repeat basis at the employees' own time.
iv. External Training : as a measure Corporate Sustainability Services (CSR), SNTI also takes up many programmes like -
* Vocational training - for Engg. Students, through guided project work,
* Industrial/ Plant visits - for Technical Institutes and other industries,
* Customised programmes for other organizations.
Operations and Facilities
Operations at SNTI are segregated into the following five groups/ sections -
1. Process operation: responsible for training in Iron making, Steel making and Rolling technology primarily in the Metallurgical areas.
2. Electrical, Electronics and Instrumentation (EEI) : responsible for training in the IEM maintenance related areas.
3. Mechanical: responsible for training in the mechanical maintenance areas.
4. IT : IT Application (ITA) responsible for training in IT and Automation related training programmes.
5. Cadre Training, Recruitment and Administrative (CTRA) : responsible for the recruitment activities, coordinating training for the cadres and entire administrative jobs of SNTI covering all the locations.
Class rooms, laboratories and other training facilities are spread over 4 locations -
- N-Road campus: This campus includes the main office, the auditorium, the conference room and the class rooms of different capacities and styles, all air-conditioned. The campus also has a world class library facility.
- Training Shop: This campus, located inside the Works, constitutes class rooms and the laboratories in various disciplines viz. Mechanical assembly, Fluid power, Automobile, Welding, Digital and analog drives, Process control and Instrumentation, Electrical, Electronics and Communication, Refractories and Information technology. All the laboratories are equipped with facilities to deliver demonstration and hands-on training for development of entire range of skills required to run the steel plant including trade testing facilities. The 'Advance Training Facility' building, with all the state-of-the-art training facilities, came up in 1989 in collaboration with the then 'Hoogovens Technical Services' in Netherlands for upgrading the existing technical training systems here.
- Crane Training Centre (CTC) : This is also situated inside the Works at a separated location and is specifically dedicated for training on crane operations and rigging.
- Employee Training Centre (ETC) : This centre is situated outside but very near to the plant, 1.5 KM away from the N-Road campus. Presently it is used for training of the Contractors' labours and some special requirement like for the expansion programmes.


|
b2
|
#include <b2Contact.h>
Public Member Functions | |
| b2Manifold * | GetManifold () |
| const b2Manifold * | GetManifold () const |
| void | GetWorldManifold (b2WorldManifold *worldManifold) const |
| Get the world manifold. | |
| bool | IsTouching () const |
| Is this contact touching? | |
| void | SetEnabled (bool flag) |
| bool | IsEnabled () const |
| Has this contact been disabled? | |
| b2Contact * | GetNext () |
| Get the next contact in the world's contact list. | |
| const b2Contact * | GetNext () const |
| b2Fixture * | GetFixtureA () |
| Get fixture A in this contact. | |
| const b2Fixture * | GetFixtureA () const |
| int32 | GetChildIndexA () const |
| Get the child primitive index for fixture A. | |
| b2Fixture * | GetFixtureB () |
| Get fixture B in this contact. | |
| const b2Fixture * | GetFixtureB () const |
| int32 | GetChildIndexB () const |
| Get the child primitive index for fixture B. | |
| void | SetFriction (float32 friction) |
| float32 | GetFriction () const |
| Get the friction. | |
| void | ResetFriction () |
| Reset the friction mixture to the default value. | |
| void | SetRestitution (float32 restitution) |
| float32 | GetRestitution () const |
| Get the restitution. | |
| void | ResetRestitution () |
| Reset the restitution to the default value. | |
| void | SetTangentSpeed (float32 speed) |
| Set the desired tangent speed for a conveyor belt behavior. In meters per second. | |
| float32 | GetTangentSpeed () const |
| Get the desired tangent speed. In meters per second. | |
| virtual void | Evaluate (b2Manifold *manifold, const b2Transform &xfA, const b2Transform &xfB)=0 |
| Evaluate this contact with your own manifold and transforms. | |
Protected Types | |
| enum | { e_islandFlag = 0x0001, e_touchingFlag = 0x0002, e_enabledFlag = 0x0004, e_filterFlag = 0x0008, e_bulletHitFlag = 0x0010, e_toiFlag = 0x0020 } |
Protected Member Functions | |
| void | FlagForFiltering () |
| Flag this contact for filtering. Filtering will occur the next time step. | |
| b2Contact (b2Fixture *fixtureA, int32 indexA, b2Fixture *fixtureB, int32 indexB) | |
| void | Update (b2ContactListener *listener) |
Static Protected Member Functions | |
| static void | AddType (b2ContactCreateFcn *createFcn, b2ContactDestroyFcn *destroyFcn, b2Shape::Type typeA, b2Shape::Type typeB) |
| static void | InitializeRegisters () |
| static b2Contact * | Create (b2Fixture *fixtureA, int32 indexA, b2Fixture *fixtureB, int32 indexB, b2BlockAllocator *allocator) |
| static void | Destroy (b2Contact *contact, b2Shape::Type typeA, b2Shape::Type typeB, b2BlockAllocator *allocator) |
| static void | Destroy (b2Contact *contact, b2BlockAllocator *allocator) |
Protected Attributes | |
| uint32 | m_flags |
| b2Contact * | m_prev |
| b2Contact * | m_next |
| b2ContactEdge | m_nodeA |
| b2ContactEdge | m_nodeB |
| b2Fixture * | m_fixtureA |
| b2Fixture * | m_fixtureB |
| int32 | m_indexA |
| int32 | m_indexB |
| b2Manifold | m_manifold |
| int32 | m_toiCount |
| float32 | m_toi |
| float32 | m_friction |
| float32 | m_restitution |
| float32 | m_tangentSpeed |
Static Protected Attributes | |
| static b2ContactRegister | s_registers [b2Shape::e_typeCount][b2Shape::e_typeCount] |
| static bool | s_initialized = false |
Friends | |
| class | b2ContactManager |
| class | b2World |
| class | b2ContactSolver |
| class | b2Body |
| class | b2Fixture |
The class manages contact between two shapes. A contact exists for each overlapping AABB in the broad-phase (except if filtered). Therefore a contact object may exist that has no contact points.
|
inline |
Get the contact manifold. Do not modify the manifold unless you understand the internals of Box2D.
|
inline |
Enable/disable this contact. This can be used inside the pre-solve contact listener. The contact is only disabled for the current time step (or sub-step in continuous collisions).
|
inline |
Override the default friction mixture. You can call this in b2ContactListener::PreSolve. This value persists until set or reset.
|
inline |
Override the default restitution mixture. You can call this in b2ContactListener::PreSolve. The value persists until you set or reset.


|
Add ntlm v1/v2 challenge respose (netntlm, netntlmv2) support to hashcat plus
|
|
01-30-2013, 05:45 AM
(This post was last modified: 01-30-2013 11:59 PM by vima.)
Post: #1
|
|||
|
|||
|
Add ntlm v1/v2 challenge respose (netntlm, netntlmv2) support to hashcat plus
Hi
We have been using oclhashcat-plus for several months now to do GPU cracking of passwords obtained during pentests, and it works great! During this period however, theres one hash type that we continually find ourselves capturing that hashcat doesnt support - NTLM challenge response hashes. We often grab these using something like the SMB capture module from Metasploit, and have always had to resort to using John the Ripper to crack them, as there is no GPU based cracker that we are aware of that will do the job. Attempting to crack these hashes using CPU when you have an 8 GPU system sitting idle is the definition of pain. Consequently, Id like to request that support be added for NTLM challenge response version 1 and 2 (known in john as netntlm and netntlmv2) in oclHashcat-plus. References for these protocols are here: http://en.wikipedia.org/wiki/NTLM http://davenport.sourceforge.net/ntlm.html I'll provide a quick summary of the generation of NTLMv1, since that the bit Im most interested in at the moment. Im happy to also do this for NTLMv2 if it will help and youre interested in implementing it. Hash creation requires a password ('hashcat' in the example below) and a variable 8 byte challenge (0x1122334455667788 is used in the example below) which is provided by the server to the client during the auth process. NTLMv1 usually generates two hashes, one based on LM hashes, and the other based on NTLM ones, although if LM hashes are disabled (e.g. via domain policy) then you wont get anything useful for the LM portion. I'll cover the generation of the NTLM version of the hash below (this is what is implemented as netntlm in john). The generation of the LM hash in NTLMv1 (implemented in john as netlm) is exactlty the same, but uses the LM hash in place of the NTLM one. Step 1 The client converts the password (hashcat) to a NTLM hash by Unicoding the password and running it though MD4: Code: hashcat -> b4b9b02e6f09a9bd760f388b67351e2b Step 2 5 null bytes are appended to the hash to take it to 21 bytes: Code: b4b9b02e6f09a9bd760f388b67351e2b0000000000 Step 3 The value is then split in 3 x 56 bit parts: Code: b4b9b02e6f09a9 Step 4 Each 56 bit part is odd parity adjusted to result in 3 x 64 bit parts. (Convert to binary, break into 8 x 7 bit chunks, append an odd parity bit to each chunk to result in 8 x 8 bit chunks. Then concatenate the chunks into 64 bit parts.) : Code: b4b9b02e6f09a9 -> b55d6d04e6792652 Step 5 Each of these values is then used as a key to DES encrypt the 8 byte challenge (for this example a value of<PHONE_NUMBER>667788 is used) resulting in 3 ciphertext blocks: Code: b55d6d04e6792652 KEY-> DES(1122334455667788) -> 51a539e6ee061f64 Step 6 The ciphertext blocks are then concatenated together to provide the final result: Code: 51a539e6ee061f647cd5d48ce6c686653737c5e1de26ac4c Let me know if any clarification is needed |
|||
|
01-30-2013, 07:20 AM
Post: #2
|
|||
|
|||
|
RE: Add ntlm v1/v2 challenge respose (netntlm, netntlmv2) support to hashcat plus
+1 for this one. Would be great to impletement.
|
|||
|
01-30-2013, 11:59 AM
Post: #3
|
|||
|
|||
|
RE: Add ntlm v1/v2 challenge respose (netntlm, netntlmv2) support to hashcat plus
Excellent description. Please add this as a trac ticket, I will accept it.
|
|||
|
01-30-2013, 11:33 PM
Post: #4
|
|||
|
|||
|
RE: Add ntlm v1/v2 challenge respose (netntlm, netntlmv2) support to hashcat plus
Cool, thanks.
Added as ticket 51. https://hashcat.net/trac/ticket/51 I'll work on a description for NTLMv2 as well. |
|||
|
01-31-2013, 06:54 PM
(This post was last modified: 01-31-2013 06:54 PM by atom.)
Post: #5
|
|||
|
|||
|
RE: Add ntlm v1/v2 challenge respose (netntlm, netntlmv2) support to hashcat plus
Perfect! If you are done with the description please add it as Ticket.
Oh and btw, i've already finished v1 on CPU. Porting to GPU will be next. |
|||
|
02-01-2013, 05:05 AM
(This post was last modified: 02-01-2013 03:16 PM by vima.)
Post: #6
|
|||
|
|||
|
RE: Add ntlm v1/v2 challenge respose (netntlm, netntlmv2) support to hashcat plus
Wow, that's an awesomely quick response atom.
Here goes with the description of NTLMv2, added as ticket 56. https://hashcat.net/trac/ticket/56 The NTLMv2 algorithm requires a username, authentication target (i.e. domain name) which is case sensitive but usually provided in all caps, a variable 8 byte server challenge and a variable sized blob of data which includes a variable client challenge and some other data like a timestamp. This last field is referred to in the Metasploit SMB capture output as NT_CLIENT_CHALLENGE, and in other sources its referred to as a 'blob', which is what I will call it. In the example I will use the following values: Password: hashcat Username: user Domain: DOMAIN Challenge: 0x1122334455667788 Blob: 0x0101000000000000e3a17e6c2600ce0191f59c598f8c6f4d00000000020000000000000000000000 The generation process is as follows: Step 1 A regular NTLM hash is produced from the password (hashcat) by unicoding it and passing through MD4. Code: hashcat-> b4b9b02e6f09a9bd760f388b67351e2b Step 2 The username (user) is converted to uppercase and concatenated with the domain name (DOMAIN). This string is converted to unicode and hashed using HMAC_MD5 with the NTLM hash generated earlier used as the key. Code: USERDOMAIN -> HMAC_MD5 KEY(b4b9b02e6f09a9bd760f388b67351e2b) -> fb659fcf061b8a92be83024db10434b1 Step 3 The server challenge (0x1122334455667788) is then concatenated with the blob (0x0101000000000000e3a17e6c2600ce0191f59c598f8c6f4d00000000020000000000000000000000) and this is fed into HMAC_MD5 with the hash just generated (fb659fcf061b8a92be83024db10434b1) used as the key. Code: 0x11223344556677880101000000000000e3a17e6c2600ce0191f59c598f8c6f4d00000000020000000000000000000000 Step 4 The final result. Code: d2d10fab083e422123c693587315814a Let me know if any clarification is needed. |
|||
|
« Next Oldest | Next Newest »
|


Review the following case study to answer this question. This case study is the same across all questions
Case Study - e-Commerce solution (online storefront) for ABC Ltd.
You are required to produce a design for an e-commerce solution (online storefront) for the retail company ABC Ltd.
ABC will use the solution to promote and sell the thousands of products listed in ABC's product catalog. Though ABC expects to add other products in the near future the catalog currently includes books, DVDs, music CDs, apparel, consumer electronics, beauty products, kitchen items, jewelry, watches, garden items, and toys. Potential customers must be able to visit the online storefront to:
- Search or browse ABC'S product catalog
- View product details (including description, price, customer ratings and reviews, etc.)
- Manage their shopping cart (add products to cart, remove products, etc.)
In addition, registered customers must be able to login, manage their user account, check out/place orders, and submit reviews of items previously purchased. To register a customer user must complete and submit an online registration form, providing ABC with their email address, password, and one or more of each of the following, phone number, shipping address, billing address, and payment details.ABC's customer service, order fulfillment, and other employee users must also be able to use the system to support business operations.
There are no hints for this question


ABOUT GOD
google
Naira Bet
Rotten Tomatoes
Good is wonderful
Good
looked
at moses and said you are my boy
what a boy:
moses is a good boy
he obeyed
God's
word
About us
Movies
Contact
Email jon
Yes he is a good boy


I've done some work on putting my itunes library on the web I intend to document some of it here.
It started out with a perl script to generate HTML from the library XML. Thhe HTML is limited to just playlists.
I'll be working on putting it into a MySQL database so I can generate more interesting things from it.
I just found this http://www.djml.com/software.djml it claims to take my iTunes and put them into MySQL! awesome.


We exchange networks!
- leverage cutting-edge bandwidth
- matrix mission-critical platforms
- synthesize world-class deliverables
- extend value-added synergies
- revolutionize rich channels
- scale end-to-end eyeballs
- extend enterprise solutions
- reintermediate one-to-one web services
- extend leading-edge action-items
- engage frictionless schemas
- implement transparent synergies
- visualize dot-com technologies
- embrace magnetic mindshare


The following are all local operations.
| MPI_COMM_SIZE(comm, size) | |
| IN comm | communicator (handle) |
| OUT size | number of processes in the group of comm (integer) |
int MPI_Comm_size(MPI_Comm comm, int *size)
MPI_COMM_SIZE(COMM, SIZE, IERROR)
This function is equivalent to accessing the communicator's group with
MPI_COMM_GROUP (see above), computing the size using
MPI_GROUP_SIZE,
and then freeing the temporary group via MPI_GROUP_FREE. However,
this function is so commonly used, that this shortcut was introduced.
( End of rationale.)
This function indicates the number of processes involved in a communicator.
For MPI_COMM_WORLD, it indicates the total number of processes
available (for this version of MPI, there is no standard way to change
the number of processes once initialization has taken place).
This call is often used with the next call to determine the amount of
concurrency available for a specific library or program. The following
call, MPI_COMM_RANK indicates the rank of the process
that calls it in the range from 0... size-1, where size
is the return value of MPI_COMM_SIZE. ( End of advice to users.)
int MPI_Comm_rank(MPI_Comm comm, int *rank)
MPI_COMM_RANK(COMM, RANK, IERROR)
This function is equivalent to accessing the communicator's group with
MPI_COMM_GROUP (see above), computing the rank using
MPI_GROUP_RANK,
and then freeing the temporary group via MPI_GROUP_FREE. However,
this function is so commonly used, that this shortcut was introduced.
( End of rationale.)
This function gives the rank of the process in the particular communicator's
group. It is useful, as noted above, in conjunction with
MPI_COMM_SIZE.
Many programs will be written with the master-slave model, where one process
(such as the rank-zero process) will play a supervisory role, and the other
processes will serve as compute nodes. In this framework, the two preceding
calls are useful for determining the roles of the various processes of a
communicator.
( End of advice to users.)
int MPI_Comm_compare(MPI_Comm comm1,MPI_Comm comm2, int *result)
MPI_COMM_COMPARE(COMM1, COMM2, RESULT, IERROR)
INTEGER COMM, SIZE, IERROR
{ int MPI::Comm::Get_size() const (binding deprecated, see Section Deprecated since MPI-2.2
) }
Rationale.
Advice to users.
MPI_COMM_RANK(comm, rank) IN comm communicator (handle) OUT rank rank of the calling process in group of
comm (integer)
INTEGER COMM, RANK, IERROR
{ int MPI::Comm::Get_rank() const (binding deprecated, see Section Deprecated since MPI-2.2
) }
Rationale.
Advice to users.
MPI_COMM_COMPARE(comm1, comm2, result) IN comm1 first communicator (handle) IN comm2 second communicator (handle) OUT result result (integer)
INTEGER COMM1, COMM2, RESULT, IERROR
{ static int MPI::Comm::Compare(const MPI::Comm& comm1, const MPI::Comm& comm2) (binding deprecated, see Section Deprecated since MPI-2.2
) }
MPI_IDENT results if and only if
comm1 and comm2 are handles for the same object
(identical groups and same contexts).
MPI_CONGRUENT results if the underlying groups are identical
in constituents and rank order; these communicators differ only by context.
MPI_SIMILAR results if the group members of both
communicators are the same but the rank order
differs. MPI_UNEQUAL results otherwise.
Up: Communicator Management
Next: Communicator Constructors
Previous: Communicator Management
Return to MPI-2.2 Standard Index
Return to MPI Forum Home Page
(Unofficial) MPI-2.2 of September 4, 2009
HTML Generated on September 10, 2009


AddFidoViewController
class AddFidoViewController : UIViewController, UINavigationBarDelegate
The purpose of the AddFidoViewController
ViewController is to manage the Fido View hierarchy.
-
Undocumented
Declaration
Swift
@IBOutlet weak var loadingIndicator: UIActivityIndicatorView!
-
Undocumented
Declaration
Swift
@IBOutlet weak var scanButton: UIButton!
-
Undocumented
Declaration
Swift
@IBOutlet weak var scanButtonItem: UIBarButtonItem!
-
Undocumented
Declaration
Swift
@IBOutlet weak var registerButton: UIButton!
-
Undocumented
Declaration
Swift
@IBOutlet weak var authenticateButton: UIButton!
-
Undocumented
Declaration
Swift
@IBOutlet weak var activityIndicator: UIActivityIndicatorView!
-
Undocumented
Declaration
Swift
@IBOutlet weak var stopButton: UIButton!
-
Undocumented
Declaration
Swift
@IBOutlet weak var stopButtonItem: UIBarButtonItem!
-
Undocumented
Declaration
Swift
@IBOutlet weak var navBar: UINavigationBar!
-
Undocumented
Declaration
Swift
@IBOutlet weak var toolBar: UIToolbar!
-
Undocumented
Declaration
Swift
override func viewDidLoad()
-
Undocumented
Declaration
Swift
override func viewDidAppear(_ animated: Bool)
-
Undocumented
Declaration
Swift
override func viewWillAppear(_ animated: Bool)
-
Undocumented
Declaration
Swift
func position(for bar: UIBarPositioning) -> UIBarPosition
-
Scan bluetooth devices.
Declaration
Swift
@IBAction func scanForDevice()
-
Stop the bluetooth scan or connection session with the MultiPass FIDO.
Declaration
Swift
@IBAction func stopButton(_ sender: Any)
-
Perform the MultiPass FIDO registeration process.
Declaration
Swift
@IBAction func sendRegister()
-
Perform the MultiPass FIDO authentication verification.
Declaration
Swift
@IBAction func sendAuthenticate()
-
Fetch MultiPass FIDO Certificate informations into the Database and set these into registerAPDU instance.
Declaration
Swift
func fetchCertificate()
-
Push the value true for the key fidotoken into the Database.
Declaration
Swift
func pushFidoRegistered()
-
Perform the buttons change in case of if the fido is registered or not.
Declaration
Swift
func isFidoRegistered(_ bool: Bool)


Tear the cloth with your mouse. Coming soon! Your mouse cursor will becoome an animated kitten when tearing the cloth.


rotten > Library > Cryptozoology > Aliens
Aliens
Aliens Who Abduct Too Much,
And The Abductees Who Love Them
There are few more pressing questions for our time: Who are the aliens that walk among us? Where are they from? What do they want? And why are they so fascinated with our asses?
Scientific opinion on extraterrestrial life often runs along the lines of Arthur C. Clarke's relentlessly logical vision of placid creatures with jellyfish bodies and a cow-like disposition that float aimlessly on hot-air updrafts from a gas-giant planet like Jupiter.
That's all well and good, and while there may be many such delightful and fascinating aliens populating this awe-striking wonder-work of a universe, these scientific theories inevitably generate a consistent and predictable human response: Boooorrrring! If that's the best the universe can come up with, why bother?
No, screw those friendly space jellyfish, who have earned the sober approval of such luminaries as Carl Sagan. These are not the aliens with which we Earthlings must be concerned. Neither the floaters, nor the fascinating and friendly silicon-based Horta of Star Trek, nor even the furry but formidable Wookies of Star Wars.
 Their interrogation rooms make Abu Ghraib look like a day in the park (and remarkably they themselves resemble the waifish Lynndie England in both stature and demeanor).
People who are trying way too hard will tell you that the aliens have been coming for us since the days of prehistory, citing the existence of the pyramids, pictures of rockets in Mayan hieroglyphics, obscure biblical passages in which angels spawn freakish giant children with the daughters of men.
While we can't conclusively determine whether these accounts are rigidly accurate historical tracts or the rantings of demented perverts, there really isn't much of a consistent historical legend about alien beings.
There are scattered reports of alien sightings from the 1950s, which surely are only coincidentally timed to the first appearance of horrific alien creatures in Hollywood movies and the arrival of Martians in the War of the Worlds hoax. Most of the 1950s sightings are unique, each featuring a type of alien which has never been reported elsewhere. One of the most famous of these is the "Flatwoods Monster," which resembled a cross between a Dalek and the Flying Nun.
The modern era of alien encounters can only be conclusively documented back to 1961. That's the year of the first official "alien abduction," an event that changed the world in weird and confusing ways. After the Barney and Betty Hill case, the nature of the "alien threat" settled down into some very predictable configurations.
The Hills were a charming, well-adjusted couple who, whilst minding their own business as well-adjusted people are wont to do, were unexpectedly set upon by a UFO in September 1961. A light in the sky zoomed over their car on a lonely back road. The UFO landed some yards in front of them. Barney Hill looked at it through binoculars and saw shadowy figures within. The couple fled.
They were 20 minutes from their house, but when they got home it was two hours later. (All of this is strictly according to their own account, by the way.) Over the course of the next few weeks, they began to have recurring nightmares about "people" doing medical tests on them.
They went to a Boston psychiatrist, who hypnotized them in an effort to unlock their trauma. He found quite a bit to unlock. The story told by the Hills would become a kind of new mythology, one that swept the world with the same force and fervor that once favored Christianity.
Under hypnosis, the Hills "revealed" that they had been imprisoned by aliens on the UFO they had seen.  The aliens described by the Hills bore a striking resemblance to aliens featured on an episode of The Outer Limits just a week earlier.
The aliens performed medical tests on the Hills, they "recalled" under hypnosis, and showed Betty Hill a star map indicating where they came from. Later, enthusiasts would "identify" the location on the map as Zeta Reticuli, a relatively nearby solar system.
As you have probably inferred by the abundance of quotation marks in the preceding passages, much of the Hills' account is questionable to a reasonable person. The "encounter" was only the first, however, in a long line of reports that would eventually sprawl into an inescapable social zeitgeist. The sightings of grey aliens slowly increased until they became part of the popular culture.
There have been hundreds of sightings of Greys, as they are called. As initial reports spread and were discussed, more came to light. A mythology began to develop around the beings, as popular culture absorbed the images of the aliens and regurgitated them into movies and TV, which then led (amazingly enough) to more "sightings."
Somewhat surprisingly, many of these sources have slowly converged into a somewhat consistent, if still fairly ridiculous, storyline.
The alpha-primate among UFO theories holds that there are a handful of alien species duking it out in our skies (and up our anuses) in a never-ending battle for alien truth, alien justice and the alien American way. There are three major types of alien, according to this school of thought:
The Greys
With big heads, small mouths and big black eyes that wrap around the back of their heads, the waifish Greys are the standard issue alien in modern culture. They resemble giant fetuses, which is probably no accident.
The Greys are considered to be the major abductors among alien species, and they're the aliens who abducted the Hills to get the whole ball rolling. They are presumed to come from Zeta Reticuli. A crackpot minority (relative to the majority crackpots) have speculated that the Greys might be human beings from the far future who have traveled back in time to visit us cavemen.
 They seem to take a special delight in inserting their instruments into rectums and noses. (South Park's Eric Cartman encountered the Greys in the premier episode, "Cartman Gets An Anal Probe.")
Greys are said to implant metallic objects in the bodies of abductees, but there haven't been any decisive scientific discoveries related to these so-called implantations. The Greys are also believed to be responsible for cattle mutilation, and a Grey-type alien was featured as the star performer in the ill-begotten alien autopsy hoax. According to lore, it was a Grey spacecraft that crashed at Roswell.
Greys get their name from their skin color, which is gray. However, there are numerous variations on the theme in the annals of UFO abduction, including blue Greys and white Greys.
The Greys played a starring role in The X-Files and have been lampooned in everything from The Simpsons to Star Trek.
The Nordics
If the Greys seem like the result of some dark psychosexual-zeitgeist-fantasy, then the Nordics are perhaps less in need of explanation.
Taller than average humans, and a damn sight better looking, the Nordics also abduct people for the purposes of science, but also for hot monkey love. While they perform all the usual experiments and probes, they also engage in good, old-fashioned intercourse. A testimonial from a woman abductee on one investigator's Web site said "It went far beyond anything we can do on this earth plane. The orgasms were constant, fast, furious, and savage."
Nordics are sometimes seen with Greys. Some abductees report that the two work together. Other "sources" say the two races are at war over some indeterminate issue. Nordics also occasionally deliver messages of inspiration and doom in a manner similar to that attributed to the Virgin Mary.
The Reptilians
Some Reptilians look very similar to the Greys, except scaly and uglier. Others look more like a traditional Star Trek-type low-budget lizard man special effect. Reptilians tend to provoke an extremely negative emotional reaction in human beings (despite being mostly innocuous in their terrestrial form).
Well, OK, in most people anyway. Pamela Stonebrook, a jazz singer described by L.A. Weekly as "the favorite entertainer of the late Gene Roddenberry," claims to have been abducted by Reptilian aliens. As she explained on her Web site:
[My forthcoming book] will tell about my reptilian encounters, a subject that very few women are prepared to go public with or speak openly about. I praise the courage of the few that already have - and endured public ridicule as a result. Reptilians are not a politically correct species in the UFO community, and to admit to having sex with one - much less enjoying it - is beyond the pale as far as the more conservative members of that community are concerned. [...]
Certainly, the reptilians use sex to control people in various ways. They have the ability to shape-shift and to control the mind of the experiencer, as well as to give tremendous pleasure through their mental powers. I have wrestled with all of these implications and the various levels of meaning and possibilities represented by my encounter experiences. I will say, however, as I have said before, that I feel a deep respect for the reptilian entity with whom I interacted, and a profound connection with this being.
Aside from Ms. Stonebrook's "not politically correct" view, one would be hard pressed to find anyone with anything good to say about the Reptilians. Some "experts" in such matters will tell you that the Reptilians are in charge of the Greys.
Some "investigators" claim the U.S. government has a secret treaty with the Greys which allows them free abduction privileges in exchange for their technological support of the Masonic-Illuminati conspiracy that rules the world. That would indeed be a scoop... One wonders how Woodward and Bernstein missed it.
The Truth Is In Here...
There is a remarkably large population of hardcore UFO believers, followed by an even larger group of people who are open to the notion that some or all of the stories of little gray men could be true.
Alien noise reached a near fever pitch a couple decades back, thanks to a series of developments, some of which require quote marks and others which must be italicized. In the latter category, Close Encounters of the Third Kind was released in 1977. A couple of years later, "researchers" "discovered" a "government" "cover-up" of a "UFO crash" in Roswell, New Mexico, which purportedly led to the creation of the "above top secret" "government organization" Majestic-12.
All this media attention led, not surprisingly, to an increase in the number of abduction reports. UFO enthusiasts will tell you that the news reports triggered suppressed memories. Skeptics will point to the sort of mob mentality that accompanies allegations of Satanic abuse.
Either way, the peak of this cultural feedback loop came with Whitley Strieber's Communion, published in 1987. Communion was the New Age epic to end all New Age epics, providing a lavish back story to the grey aliens and their ilk.
Strieber brought a scientological rigor to the discussion which had not previously manifested. Previously a successful fantasy novelist, Strieber told a tale of multiple abductions and laid out a rough sketch of experiences which has evolved into a widely discussed notion of alien conspiracy. How widely discussed? Strieber reports have received half a million letters from readers, many of whom reported similar experiences to his own.
In the wake of Communion, a series of ever more incredulous books began to emerge in which remote viewing, channeling and all manner of "deductive reasoning" and "investigative reporting" were brought to bear on the problem of alien visitors... such as the mind-boggling Above Top Secret, which related tales of unimaginable conspiracies and highly improbable leaps of logic with a poker face to die for.
Above Top Secret helped popularize some of the more ridiculous claims of UFOlogy (on the relative scale of ridiculousness that one must employ when looking at the body of material out there).
"Like everyone assumes there are little gray visitors here from Zeta Reticuli," Strieber commented in an interview. "What? How did we come to that? We came to that through a lot of very vague, hypnotized recall and examining star maps that may or may not have had some sort of relevance to the real world. And it became engraved in stone. But it's nothing. It's fantasy. It's superstition."
Which is, of course, a relative evaluation. While Strieber draws the line at Zeta Reticuli, there are many, many people who draw the line much more conservatively, including the vast majority of mainstream scientists.
While mainstream scientists are often a joyless, narrow-minded lot, they do have a point. Alien abduction stories are a tough sell, especially in light of the overwhelming absence of hard evidence to support these claims. UFO enthusiasts (or whatever you want to call them) claim that there have been more than 4 million abductions. It's extremely difficult to credit the idea that 4 million people have been abducted, probed and/or implanted without producing a single piece of conclusive physical evidence.
As appealing as the notion may be, a reasonable person could be forgiven for concluding that there are not any little gray men running around, nor a massive government conspiracy to cover up an alien invasion.
Moreover, the mainstream scientists can actually offer some reasonably compelling explanations of why so many people may report the same sort of experience, an explanation with a little more weight that just saying that somewhere between 500,000 and 4 million people all saw the same episode of The Outer Limits.
The most popular skeptical explanation of alien abduction phenomena has to do with a syndrome known as "sleep paralysis." Alien abductions almost always occur late at night. Abductees most often report being taken from their beds or being abducted on long, lonely roads.
Sleep paralysis is a fairly common syndrome, which affects as many of half of all people at some point in their lives. In sleep paralysis, someone who has just fallen asleep or is between sleep and waking, experiences a sudden feeling of heaviness and an inability to move. The same feeling is frequently reported by alien abductees, who blame the effect on some sort of rohypnol-laced lubricant designed to facilitate the insertion of objects into the abductee's rectum.
While sleep paralysis, combined with the onset of dreaming or a hypnotic trance produced by late night driving, might account for some of the tales of abduction, there isn't a strong body of evidence showing that sleep paralysis can be commonly associated with the sort of hallucinations required to produce a full-blown abduction experience.
A recent theory might be better suited to explaining the seeming universal notion of the abduction experience. Scientists (and casual users) have noted that the hallucinogenic drug DMT frequently produces a hallucination that closely matches the abduction experience, right down to the little gray men. When DMT is synthesized into a concentrated form (similar to how opium is refined into heroin), it produces an extremely consistent abduction experience.
DMT is a chemical produced naturally by the brain. Under normal circumstances, it exists in the brain only in trace amounts. But the same mechanism that regulates its presence in the brain is responsible for regulating sleep patterns via serotonin production. Tie sleep paralysis to DMT production, and you might be on to something.
Of course, there's still the question of why these experiences are so similar to each other. And there's the interesting quirk that the gland which regulates DMT in the brain is the pineal gland, which is located right where mystics believe you can find the "Third Eye" chakra. Happily, when science closes a door, the creative imagination jimmies a window. Maybe there are no little grey men, but that doesn't mean something even weirder isn't just around the corner.
The Wow Signal
Pornopolis |
Rotten |
Faces of Death |
Famous Nudes
|


US Secretary of State Rex Tillerson said North Korea must "earn its way back to the table" for negotiations between the two countries to restart.
Mr Tillerson said Pyongyang had to carry out a "sustained cessation" of weapons testing before any talks. It is a U-turn from comments made earlier this week, when Mr Tillerson said the US was "ready to talk any time time North Korea would like to talk". That remark was swiftly contradicted by the White House. What missiles does North Korea have? How North Korea could tip the balance North Korea has carried out repeated nuclear and ballistic missile tests this year, in defiance of global condemnation and increasingly heavy international sanctions. Earlier this week Mr Tillerson said: "Let's just meet and let's talk about the weather if you want and talk about whether it's going to be a square table or a round table if that's what you're excited about. "Then we can begin to lay out a map, a road map, of what we might be willing to work towards." Rex Tillerson at the UN security council meetingImage copyrightREUTERS Image caption There are reports that Mr Tillerson had planned to repeat his offer of talks without preconditions But his remarks, welcomed by China and Russia, were promptly greeted with a rebuttal from the White House and a reiteration that North Korea must admit to abandoning its nuclear weapons before talks resume. Within just a few hours press secretary Sarah Saunders released a statement to reporters saying Mr Trump's views "have not changed". "North Korea is acting in an unsafe way not only toward Japan, China, and South Korea, but the entire world," she said.


Education for All
Quality and access to education is the major concern in rural schools as there are fewer committed teachers, lack of proper text books and learning material in the schools.
Learn MoreQuality and access to education is the major concern in rural schools as there are fewer committed teachers, lack of proper text books and learning material in the schools.
Learn More


Bunch raises $3.85m toward development, expanded partnerships
Supercell, Tencent, and Riot Games among investors in mobile game chat and party app
by Rebekah ValentineMobile group chat and partying app Bunch has announced it raised $3.85 million in funding from several major gaming companies.
The funding round was led by companies including Supercell, Tencent, Riot Games, Miniclip, and Colopl Next, and will be used to expand the company's partnerships and integrations, as well as to continue development of the Bunch app.
In total, Bunch has raised $8.5 million in funding to date.
"At Supercell, we believe that the social experience has a crucial role in making games that players keep playing for years," said Supercell head of developer relations and strategic finance Jaakko Harlas. "We are constantly looking for ways to make playing our games with friends as easy and enjoyable as possible. Bunch is the most thoughtful experience we've seen that tackles this problem - and we are looking forward to working together closely."
Additionally, Bunch has hired former Facebook and Google senior product designer Jonas Christiansen as its new head of product, and Apple and Mobcrush veteran Greg Essig as head of business development.


Kees Dorst
A critique of design research, especially prescription for design processes. Contrasts design research and design practice and tries to understand the distance between them. Urges that design processes may need to match content, personalities and context.
They add 'naive' to the sequence of increasing expertise offered by Dreyfus; novice, advanced beginner, competent, expert, master, visionary.
Positions design research as the study and innovation of design practice.
"to describe an area of complex creative
human endeavour like design...one would need to describe the
object
of
this activity (in this case, the design problem
and the emerging design solution e the ‘con-
tent’), the
actor
(the designer or the design
team/designing organisation), the
context
in
which the activity takes place (as far as it impacts
upon the activity) and the structure and dynam-
ics of the complex of activities that is being stud-
ied (‘the
design process
’)."
"Within design research there are two fundamen-
tally different ways of looking at design process,
those of rational problem solving (a goal-
oriented problem solving process) and reflective
practice (a process of learning)."
"this total ignoring of
the design content, the designer and the design
context allows us to claim that we are construct-
ing models, methods and tools that will be valid
for every designer, dealing with every possible
kind of design problem, in any situation."
"Hubert Dreyfus...distinguishes six distinct
levels of expertise, corresponding with six ways
of perceiving, interpreting, structuring, and solv-
ing problems"
"Design researchers should
join design practitioners in co-creating the design
expertise and design practices of the future."
Dreyfus
Complete