In this project, you'll classify images from the CIFAR-10 dataset. The dataset consists of airplanes, dogs, cats, and other objects. You'll preprocess the images, then train a convolutional neural network on all the samples. The images need to be normalized and the labels need to be one-hot encoded. You'll get to apply what you learned and build a convolutional, max pooling, dropout, and fully connected layers. At the end, you'll get to see your neural network's predictions on the sample images.
Run the following cell to download the CIFAR-10 dataset for python.
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import problem_unittests as tests
import tarfile
cifar10_dataset_folder_path = 'cifar-10-batches-py'
# Use Floyd's cifar-10 dataset if present
floyd_cifar10_location = '/cifar/cifar-10-python.tar.gz'
if isfile(floyd_cifar10_location):
tar_gz_path = floyd_cifar10_location
else:
tar_gz_path = 'cifar-10-python.tar.gz'
class DLProgress(tqdm):
last_block = 0
def hook(self, block_num=1, block_size=1, total_size=None):
self.total = total_size
self.update((block_num - self.last_block) * block_size)
self.last_block = block_num
if not isfile(tar_gz_path):
with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:
urlretrieve(
'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',
tar_gz_path,
pbar.hook)
if not isdir(cifar10_dataset_folder_path):
with tarfile.open(tar_gz_path) as tar:
tar.extractall()
tar.close()
tests.test_folder_path(cifar10_dataset_folder_path)
The dataset is broken into batches to prevent your machine from running out of memory. The CIFAR-10 dataset consists of 5 batches, named data_batch_1
, data_batch_2
, etc.. Each batch contains the labels and images that are one of the following:
Understanding a dataset is part of making predictions on the data. Play around with the code cell below by changing the batch_id
and sample_id
. The batch_id
is the id for a batch (1-5). The sample_id
is the id for a image and label pair in the batch.
Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?". Answers to questions like these will help you preprocess the data and end up with better predictions.
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import helper
import numpy as np
# Explore the dataset
batch_id = 1
sample_id = 6
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)
Take an image x as input ---> return numpy array
def normalize(x):
"""
Normalize a list of sample image data in the range of 0 to 1
: x: List of image data. The image shape is (32, 32, 3) 3 means : RGB
: return: Numpy array of normalize data
"""
# TODO: Implement Function
return x / 256
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)
Just like the previous code cell, you'll be implementing a function for preprocessing. This time, you'll implement the one_hot_encode
function. The input, x
, are a list of labels. Implement the function to return the list of labels as One-Hot encoded Numpy array. The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode
. Make sure to save the map of encodings outside the function.
Hint: Don't reinvent the wheel.
def one_hot_encode(x):
"""
One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
: x: List of sample Labels
: return: Numpy array of one-hot encoded labels
"""
# TODO: Implement Function
tmpList = []
for i in range(len(x)):
encoded = np.zeros(10)
encoded[x[i]] = 1
tmpList.append(encoded)
retArray = np.array(tmpList)
return retArray
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)
As you saw from exploring the data above, the order of the samples are randomized. It doesn't hurt to randomize it again, but you don't need to for this dataset.
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)
This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper
# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))
For the neural network, you'll build each layer into a function. Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function. This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.
Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section. TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.
However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the
conv2d
class, tf.layers.conv2d, you would want to use the TF Neural Network version ofconv2d
, tf.nn.conv2d.
Let's begin!
The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions
neural_net_image_input
image_shape
with batch size set to None
.name
parameter in the TF Placeholder.neural_net_label_input
n_classes
with batch size set to None
.name
parameter in the TF Placeholder.neural_net_keep_prob_input
name
parameter in the TF Placeholder.These names will be used at the end of the project to load your saved model.
Note: None
for shapes in TensorFlow allow for a dynamic size.
import tensorflow as tf
def neural_net_image_input(image_shape):
"""
Return a Tensor for a batch of image input
: image_shape: Shape of the images
: return: Tensor for image input.
"""
# TODO: Implement Function
return tf.placeholder(tf.float32, (None,)+image_shape,name='x')
def neural_net_label_input(n_classes):
"""
Return a Tensor for a batch of label input
: n_classes: Number of classes
: return: Tensor for label input.
"""
# TODO: Implement Function
return tf.placeholder(tf.float32,(None,n_classes),name='y')
def neural_net_keep_prob_input():
"""
Return a Tensor for keep probability
: return: Tensor for keep probability.
"""
# TODO: Implement Function
return tf.placeholder(tf.float32,name='keep_prob')
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)
Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool
to apply convolution then max pooling:
conv_ksize
, conv_num_outputs
and the shape of x_tensor
.x_tensor
using weight and conv_strides
.pool_ksize
and pool_strides
.Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer, but you can still use TensorFlow's Neural Network package. You may still use the shortcut option for all the other layers.
def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
"""
Apply convolution then max pooling to x_tensor
:param x_tensor: TensorFlow Tensor
:param conv_num_outputs: Number of outputs for the convolutional layer
:param conv_ksize: kernal size 2-D Tuple for the convolutional layer
:param conv_strides: Stride 2-D Tuple for convolution
:param pool_ksize: kernal size 2-D Tuple for pool
:param pool_strides: Stride 2-D Tuple for pool
: return: A tensor that represents convolution and max pooling of x_tensor
"""
# TODO: Implement Function
#calcute the depth of channel
channel = x_tensor.get_shape().as_list() #5
#assemble list for calculating weight
argForWeight = list(conv_ksize + (channel[-1],) + (conv_num_outputs,))
#print(argForWeight)
#create weight
kernel_weight = tf.Variable(tf.truncated_normal(argForWeight,0,0.3))
#create bias
kernel_bias = tf.Variable(tf.zeros(conv_num_outputs))
#apply weight and bias to x_tensor as requirment.
convTensor = tf.nn.conv2d(x_tensor, kernel_weight, (1,)+conv_strides+(1,), padding='SAME')
#add bias to convTensor
convTensor = tf.nn.bias_add(convTensor, kernel_bias)
#apply relu non-linar to convTensor
convTensor = tf.nn.relu(convTensor)
#do max-pooling
#ref:udacity deep-learing course:
#For both ksize and strides, the batch and channel dimensions are typically set to 1.
#calculate ksize
ksize = (1,)+ pool_ksize +(1,)
strides = (1,) + pool_strides + (1,)
convTensor = tf.nn.max_pool(convTensor,ksize,strides,'SAME')
return convTensor
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)
Implement the flatten
function to change the dimension of x_tensor
from a 4-D tensor to a 2-D tensor. The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
def flatten(x_tensor):
"""
Flatten x_tensor to (Batch Size, Flattened Image Size)
: x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
: return: A tensor of size (Batch Size, Flattened Image Size).
 Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
def fully_conn(x_tensor, num_outputs):
"""
Apply a fully connected layer to x_tensor using weight and bias
: x_tensor: A 2-D tensor where the first dimension is batch size.
: num_outputs: The number of output that the new tensor should be.
: return: A 2-D tensor where the second dimension is num_outputs.
 Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Note: Activation, softmax, or cross entropy should not be applied to this.
def output(x_tensor, num_outputs):
"""
Apply a output layer to x_tensor using weight and bias
: x_tensor: A 2-D tensor where the first dimension is batch size.
: num_outputs: The number of output that the new tensor should be.
: return: A 2-D tensor where the second dimension is num_outputs.
 The function takes in a batch of images, x
, and outputs logits. Use the layers you created above to create this model:
keep_prob
. def conv_net(x, keep_prob):
"""
Create a convolutional neural network model
: x: Placeholder tensor that holds image data.
: keep_prob: Placeholder tensor that hold dropout keep probability.
: return: Tensor that represents logits
"""
# TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
# Play around with different number of outputs, kernel size and stride
# Function Definition from Above:
# conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
#conv_model = conv2d_maxpool(x, conv_num_outputs=18, conv_ksize=(4,4), conv_strides=(1,1),
#pool_ksize=(8,8), pool_strides=(1,1))
x = conv2d_maxpool(x, conv_num_outputs=18, conv_ksize=(1,1), conv_strides=(1,1),
pool_ksize=(2,2), pool_strides=(1,1))
#x = tf.nn.dropout(x, keep_prob)
x = conv2d_maxpool(x, conv_num_outputs=32, conv_ksize=(2,2), conv_strides=(1,1),
pool_ksize=(6,6), pool_strides=(2,2))
#x = tf.nn.dropout(x, keep_prob)
# TODO: Apply a Flatten Layer
# Function Definition from Above:
# flatten(x_tensor)
x = flatten(x)
# TODO: Apply 1, 2, or 3 Fully Connected Layers
# Play around with different number of outputs
# Function Definition from Above:
# fully_conn(x_tensor, num_outputs)
x = fully_conn(x,32) # 4-fold
x = tf.nn.dropout(x, keep_prob)
x = fully_conn(x,64) # 4-fold
x = tf.nn.dropout(x, keep_prob)
x = fully_conn(x,1024) # 4-fold
x = tf.nn.dropout(x, keep_prob)
# TODO: Apply an Output Layer
# Set this to the number of classes
# Function Definition from Above:
# output(x_tensor, num_outputs)
x = output(x,10)
# TODO: return output
return x
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
##############################
## Build the Neural Network ##
##############################
# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()
# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()
# Model
logits = conv_net(x, keep_prob)
# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')
# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)
# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')
tests.test_conv_net(conv_net)
Implement the function train_neural_network
to do a single optimization. The optimization should use optimizer
to optimize in session
with a feed_dict
of the following:
x
for image inputy
for labelskeep_prob
for keep probability for dropoutThis function will be called for each batch, so tf.global_variables_initializer()
has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
"""
Optimize the session on a batch of images and labels
: session: Current TensorFlow session
: optimizer: TensorFlow optimizer function
: keep_probability: keep probability
: feature_batch: Batch of Numpy image data
: label_batch: Batch of Numpy label data
"""
# TODO: Implement Function
#sess.run(x, feed_dict={x: 'Hello World'})
session.run(optimizer, feed_dict={x: feature_batch, y: label_batch, keep_prob: keep_probability})
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)
Implement the function print_stats
to print loss and validation accuracy. Use the global variables valid_features
and valid_labels
to calculate validation accuracy. Use a keep probability of 1.0
to calculate the loss and validation accuracy.
def print_stats(session, feature_batch, label_batch, cost, accuracy):
"""
Print information about loss and validation accuracy
: session: Current TensorFlow session
: feature_batch: Batch of Numpy image data
: label_batch: Batch of Numpy label data
: cost: TensorFlow cost function
: accuracy: TensorFlow accuracy function
"""
# TODO: Implement Function
loss = session.run(cost, feed_dict={x: feature_batch, y: label_batch, keep_prob:1.0})
valid_acc = sess.run(accuracy, feed_dict={x: valid_features,y: valid_labels, keep_prob: 1.0})
print('Loss: {:>10.4f} Validation Accuracy: {:.6f}'.format(loss,valid_acc))
Tune the following parameters:
epochs
to the number of iterations until the network stops learning or start overfittingbatch_size
to the highest number that your machine has memory for. Most people set them to common sizes of memory:keep_probability
to the probability of keeping a node using dropout# TODO: Tune Parameters
epochs = 50
batch_size = 384
keep_probability = 0.8
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy. Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
# Initializing the variables
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(epochs):
batch_i = 1
for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
print('Epoch {:>2}, CIFAR-10 Batch {}: '.format(epoch + 1, batch_i), end='')
print_stats(sess, batch_features, batch_labels, cost, accuracy)
Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'
print('Training...')
with tf.Session() as sess:
# Initializing the variables
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(epochs):
# Loop over all batches
n_batches = 5
for batch_i in range(1, n_batches + 1):
for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
print('Epoch {:>2}, CIFAR-10 Batch {}: '.format(epoch + 1, batch_i), end='')
print_stats(sess, batch_features, batch_labels, cost, accuracy)
# Save Model
saver = tf.train.Saver()
save_path = saver.save(sess, save_model_path)
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import tensorflow as tf
import pickle
import helper
import random
# Set batch size if not already set
try:
if batch_size:
pass
except NameError:
batch_size = 64
save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3
def test_model():
"""
Test the saved model against the test dataset
"""
test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
# Load model
loader = tf.train.import_meta_graph(save_model_path + '.meta')
loader.restore(sess, save_model_path)
# Get Tensors from loaded model
loaded_x = loaded_graph.get_tensor_by_name('x:0')
loaded_y = loaded_graph.get_tensor_by_name('y:0')
loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')
# Get accuracy in batches for memory limitations
test_batch_acc_total = 0
test_batch_count = 0
for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
test_batch_acc_total += sess.run(
loaded_acc,
feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
test_batch_count += 1
print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))
# Print Random Samples
random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
random_test_predictions = sess.run(
tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)
test_model()
You might be wondering why you can't get an accuracy any higher. First things first, 50% isn't bad for a simple CNN. Pure guessing would get you 10% accuracy. However, you might notice people are getting scores well above 80%. That's because we haven't taught you all there is to know about neural networks. We still need to cover a few more techniques.
When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_image_classification.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.
Running Environment: OS:macOS 10.12.6 CPU:i7 2.7GHz Memory:16G DDR3 No GPU Acceleration.
To Reviews: I have tried different hyperparameters values and asjust conv-net structure, but I even can't achieve 30% accurrency. I should try hader!!!


Here's the Problem
One of the problems at UNLV is the lack of encouragement for free speech around campus. Along the free speech walkway, there are different groups of people that share their messages. However, this is a miniscule portion of the UNLV campus. Because of this, students feel as though free speech is not encouraged all around campus. This can be problematic because if students feel as though they do not have a voice, then they will not be engaged in college life or their studies. Below are a couple graphs created from a survey taken asking students how they feel about free speech on campus.


WITNESS LIST County Affairs Committee April 14, 1999-8:00A HB 1756 For: Aguilar, Carlos (El Paso County) Falcone, Ernani (NW Neighborhood Alli, Nthside Neighborhoods for Orderly Dev, Dist. 9 Coalition of Neighborhood Assn) Guerra, Joe A. (Self) Krier, Cyndi (Self and County of Bexar) Peak, Howard (Self) Sinkin, Bill (Citizens for the Right to Vote) Wender, Charles Martin (Self and The Greater San Antonio Chamber of Commerce) Against: Boethel, Carey (Texas Association of Counties) On: Cook, Gordon W. (Self) HB 1854 On: Eckels, Robert (Self) HB 2009 Committee Substitute (Substitute by Chisum) For: Allison, Jim (County Judges & Commissioners Association of Texas) HB 2034 For: Brown, Curtiss (Galveston County Commissioners Court & County Judge Jim Yarbrough) HB 2214 For: Carreon, Rodrigo (Self and Fort Bend Interfaith Council) Eckels, Robert (Self and Harris County) Garcia, Maria Roberta (Self and Cumings Road Water Dist.) Hernandez, Hortensia (Cummings Road Water Supply Corporation) Prestage, Grady (Self) On: Steinberg, Jonathan (Texas Water Development Board) HB 2265 For: Eckels, Robert (Self and Harris County Commissioners Court) HB 2331 For: Allison, Jim (County Judges & Commissioners Association of Texas) Ray, Elizabeth (Harris County District Judges) Against: Meadows, Mindy (Texas Apartment Association) HB 2497 For: Allison, Jim (County Judges & Commissioners Association of Texas) HB 2654 For: Klein, Barry (Self and Houston Property Rights Association) Against: Allison, Jim (County Judges & Commissioners Association of Texas) Burgess, Anita E. (City of Lubbock) McNabb, Charlie (City of El Paso) HB 2693 For: Wetz, Fred (Self) HB 2787 For: Blanton, Tom (Texas Automobile Dealers Association) HB 2968 For: Allison, Jim (County Judges & Commissioners Association of Texas) Riester, Eric (Bexar County) HB 3114 On: Allison, Jim (County Judges & Commissioners Association of Texas) HB 3185 For: Brown, Curtiss (Galveston County Commissioners Court & County Judge Jim Yarbrough) HB 3262 For: Hutchins, Elbert C. (Texas Veterinary Medical Association) HB 3453 For: Blair, Nelda Luce (Self) Cooke, Walter (Self and South Montgomery County Woodlands Chamber of Commerce) Robb, Ed (Self) HB 3569 For: Miears, Bill (County & District Clerks Association, Chair person Cecelia Adair) Moore, Jr., J.R. (Self and The Tax Assessor-Collector's Association of Texas) Prindle, Sandy (Justices of the Peace and Constables Association of Texas) Scheske, Robert B. (Self) Against: Allison, Jim (County Judges & Commissioners Association of Texas) HJR 61 For: Aguilar, Carlos (El Paso County) Falcone, Ernani (NW Neighborhood Alli, Nthside Neighborhood for Orderly Dev, Dist 9 coalition of Neighborhood Assn) Guerra, Joe A. (Self) Krier, Cyndi (Self and County of Bexar) McNabb, Charles (Mayor Carlos Ramirez and the City of El Paso) Peak, Howard (Self) Peterson, Larry (Self) Sinkin, Bill (Citizens for the Right to Vote) Wender, Charles Martin (Self and The Greater San Antonio Chamber of Commerce) Against: Allison, Jim (County Judges & Commissioners Association of Texas) Boethel, Carey (Texas Association of Counties) Moore, J.R. (Self and The Tax Assessor-Collector's Association of Texas) Pardue, Craig (Dallas County) On: Cook, Gordon W. (Self) SB 624 For: Paniagua, Joe (Fort Worth City Council) On: Johnson, Jamie (Self)


Hello there!
My name is Al Aganya.I would like to be a pro programmer one day.
As you might notice from the background, I love comics.
I have a couple of projects already as of now.
If you would like to know more about me or view my projects so far,
feel free to do so by clicking the menu bar and navigate.
You can also see my social profile by asking the icons up there politely.
I hope you'll like it.


--- layout: default --- {% include github_ribbon.html %} {% include header.html %} {{content}} {% include team.html %}


Create your dream holiday with the help of our expert. The entire holidays are designed around requirement you send us.  Find out more
The travel experts have had in depth background on their specialist regions. You will have suggested holday in timely manner get the great save of your time and budget. You will have the same specialist throughout the planning process. Find out more
You are not sure where to plan your upcoming trips. The inspiration section provides highlights of month and destinations. You find hundreds of itinerary and destination ideas based on your travel preferences. Find out more
Best deals guaranteed. Check our lowest rates.
Get up to the moment deals from multiple trips.
Certified secure communication for safe transactions
Customer satisfaction is our greatest expertise.


Web Thing Controller
Control Web Things via EVRYTHNG.
My WoT Plug
Actions
_setStatus
Response
DEBUG (Show HTTP Response)
Properties
0.0
power
0
voltage
0
current
false
status


Azure Microservice Platforms Solution AKS
Overview
TODO kubernetes core
The services that (can) complement Kubernetes:
-
Provisioning
See Provisioning for general aspects how you can handle provisioing in Azure such as creating pipelines for creating infrastructure. Additional aspects such as building a containe image are described here.
-
Azure Container Registry
-
Azure AD
Azure Active Directory provides the service principal the pipelines run with.
-
Monitoring
For infrastructure monitoring see "Monitoring". For application monitoring specific tools exist which will be described here.
The picture illustrates the setup with the major dependencies:
Pattern Details
Geting Started
TODO if there are options what do I have to decide first e.g. organizations, projects and teams in case of Azure DevOps
Remaining goals
Points from summary Migration:
-
Deployment resources to Kubernetes
Instead of having to write separate YAML files for each application manually, you can simply create a Helm chart and let Helm deploy the application to the cluster for you. Helm charts contain templates for various Kubernetes resources that combine to form an application. A Helm chart can be customized when deploying it on different Kubernetes clusters. Helm charts can be created in such a way that environment or deployment-specific configurations can be extracted out to a separate file so that these values can be specified when the Helm chart is deployed. For example, you need not have separate charts for deploying an application in development, staging, and production environments.
Chart linting is an easy tool that you can add to your pipeline to ensure your deployments are valid and versioned correctly. For security reasons and improvement of Helm charts, it is useful to make use of at least one Helm linting tool. Why choosing Polaris as Linting Tool: For helm chart linting, there are several tools like Polaris, kube-score or config-lint available. With Polaris, checks and rules are already given by default, whereby other tools need a lot of custom rules configuration and are therefore more complex to setup. Polaris runs a variety of checks to ensure that Kubernetes pods and controllers are configured using best practices, helping to avoid problems in the future. Polaris can be either installed inside a cluster or as a command-line tool to analyze Kubernetes manifests statically.
The Helm operator provides an extension to Flux that automates Helm Chart releases. A Helm Chart release is described via a Kubernetes custom resource named HelmRelease. Flux synchronizes these resources from Git to the cluster, while the Helm operator makes sure Helm Charts are released as specified in the resources.
-
Configuration
Azure Key Vault to Kubernetes (akv2k8s) is used for our applications, to make Azure Key Vault secrets, certificates and keys available to use in a secure way. The goals of Azure Key Vault to Kubernetes are to avoid a direct program dependency for getting secrets, secure and low risk to transfer Azure Key Vault secrets and transparently inject secrets into applications. Per default secrets, configurations and certificates can be easily read and accessed by users in Kubernetes and access to them can only be restricted by setting access rights. This will be avoided and is a huge benefit of using akv2k8s with a simple setup of Azure Key Vault and the option to set more detailed restrictions and configurations.
Question: What about telling container what is the right key vault? (Injection needs to know about environment)
Tools for synchronizing configuration:
-
ArgoCD:
Continuous Delivery for Kubernetes: ArgoCD is a leader developed by Intuit and synchronizes changes in the code of applications, photos and cluster definitions, so the Git - repository , to the cluster. The solution is open source software, kept relatively simple and is one of the most important and oldest tools on the market.
-
Flux
Integration of Flux CD into the workflow: Flux basically does the same job as ArgoCD, i.e. synchronizes the repository and cluster in the course of continuous delivery. Flux is also open source and kept simple - the special thing: It comes from the GitOps inventor Weaveworks.
Question: Have heard about built in Flux support in AKS
-
-
Monitoring
Application monitoring: Prometheus Infrastructure: see Standard (Extend dashboarding)
Variations
The following alternatives exist:
-
Managed Openshift
Comment Openshift: Refers to Red Hat Openshift and not hosted offering TODO: Better variation for me managed OpenShift (Link)
Criteria Openshift Kubernetes Flexibility
Limited; opinionated components/ fnctionalities
greater flexibility
Installation
limited options
almost anywhere
Security
very strict; certain permission level for maintenance
Deployment
less flexible DeplyomentConfig
very flexile helm charts
External access
via Routers
via ingress conntrollers
Management
ImageStreams improve management
Management container images not that easy
Usere experience
Better user support
additional tool for user experience
Networking
native networking solution
some components don’t have networking + third party required
Service Catalog
When to use
When you need orchestration support due to a higher number of microservices. If you start with a single service then Function App reduces greatly the complexity. You can still later on move to Kubernetes by containerizing your function app code.


List of general configuration options
In Administration > Configuration there are a number of configuration values.
#
key
default
meaning
version
1
crypt.class.name
org.apache.openmeetings.util.crypt.SCryptImplementation
This Class is used for Authentification-Crypting. Be carefull what you do here! If you change it while running previous Pass of users will not be workign anymore! for more Information see https://openmeetings.apache.org/CustomCryptMechanism.html
1.9.x
2
allow.frontend.register
true
Is USER register available on login screen
1.8.x
3
allow.soap.register
true
Is user register available via SOAP/REST
3.0.x
4
allow.oauth.register
true
Is user register available via OAuth
3.0.x
5
#
default.group.id
1
1.8.x
6
mail.smtp.server
localhost
this is the smtp server to send messages
1.9.x
7
#
mail.smtp.port
25
this is the smtp server port normally 25
1.9.x
8
mail.smtp.system.email
<EMAIL_ADDRESS>all send e-mails by the system will have this address
1.9.x
9
mail.smtp.user
System auth email username
1.9.x
10
mail.smtp.pass
System auth email password
1.9.x
11
mail.smtp.starttls.enable
false
Enable TLS
1.9.x
12
#
mail.smtp.connection.timeout
30000
Socket connection timeout value in milliseconds. Default is 30 seconds (30000).
1.9.x
13
#
mail.smtp.timeout
30000
Socket I/O timeout value in milliseconds. Default is 30 seconds (30000).
1.9.x
14
application.name
OpenMeetings
Name of the Browser Title window
3.0.x
15
#
default.lang.id
1
Default System Language ID see languages.xml
1.8.x
16
#
document.dpi
150
dpi for conversion of PDF to images (should be an integer between 50 and 600 with a default value of 150 dpi)
2.0.x
17
#
document.quality
90
compression quality for conversion of PDF to images (should be an integer between 1 and 100, with a default value of 90)
2.0.x
18
path.imagemagick
Path to ImageMagick tools
2.0.x
19
path.sox
Path To SoX-Tools
2.0.x
20
path.ffmpeg
Path To FFMPEG
2.0.x
21
path.office
The path to OpenOffice/LibreOffice (optional) please set this to the real path in case jodconverter is unable to find OpenOffice/LibreOffice installation automatically
2.0.x
22
dashboard.rss.feed1
https://mail-archives.apache.org/mod_mbox/openmeetings-user/?format=atom
Feed URL 1
1.9.x
23
dashboard.rss.feed2
https://mail-archives.apache.org/mod_mbox/openmeetings-dev/?format=atom
Feed URL 2
1.9.x
24
send.email.at.register
false
User get a EMail with their Account data.
2.0.x
25
send.email.with.verfication
false
User must activate their account by clicking on the activation-link in the registering Email It makes no sense to make this(send.email.with.verfication) 'true' while send.email.at.register is 'false' cause you need to send a EMail.
2.0.x
26
application.base.url
https://localhost:5443/openmeetings/
Base URL your OPenmeetings installation will be accessible at.
3.0.2
27
sip.enable
false
Enable to enable the red5SIP integration
1.9.x
28
sip.room.prefix
400
Numerical prefix for OM rooms created inside the SIP
1.9.x
29
sip.exten.context
rooms
Enable to enable the red5SIP integration
1.9.x
30
default.timezone
Europe/Berlin
This is the default timezone if nothing is specified
1.9.x
31
#
screensharing.default.quality
1
Default selection in ScreenSharing Quality: 0 - bigger frame rate, no resize 1 - no resize 2 - size == 1/2 of selected area 3 - size == 3/8 of selected area
3.0.3
32
#
screensharing.default.fps
10
Default selection in ScreenSharing FPS
3.0.3
33
screensharing.fps.show
true
Is screensharing FPS should be displayed or not
3.0.3
34
screensharing.allow.remote
true
Is remote control will be enabled while screensharing. Allowing remote control will be not possible in case it is set to 'false'
3.0.4
35
dashboard.show.myrooms
true
Show 'My Rooms' widget on dashboard
1.9.x
36
dashboard.show.chat
true
Show 'Global Chat' outside the room
1.9.x
37
dashboard.show.rssfeed
false
Show RSS widget on dashboard
1.9.x
38
#
max.upload.size
104857600
Maximum size of upload file (bytes)
1.8.x
39
#
number.minutes.reminder.send
15
The number of minutes before reminder emails are send. Set to 0 to disable reminder emails
1.9.x
40
#
user.login.minimum.length
4
Number of chars needed in a user login
1.9.x
41
#
user.pass.minimum.length
8
Number of chars needed in a user password
1.9.x
42
#
calendar.conference.rooms.default.size
50
Default number of participants conference room created via calendar
1.9.x
43
video.arrange.keycode
Shift+F8
A hot key code for arrange video windows functionality
2.0.x
44
mute.others.keycode
Shift+F12
A hot key code for the 'mute others' functionality
2.0.x
45
mute.keycode
Shift+F7
A hot key code for the 'mute/unmute audio' functionality
2.0.x
46
#
default.ldap.id
0
Ldap domain selected by default in the login screen
1.9.x
47
inviter.email.as.replyto
true
Set inviter's email address as ReplyTo in email invitations
2.0.x
48
default.landing.zone
user/dashboard
Area to be shown to the user after login. Possible values are: user/dashboard, user/calendar, user/record, rooms/my, rooms/group, rooms/public, profile/messages, profile/edit, profile/search, profile/invitation, profile/widget, admin/user, admin/connection, admin/group, admin/room, admin/config, admin/lang, admin/ldap, admin/oauth2, admin/backup, admin/email
2.1.x
49
oauth2.ignore.bad.ssl
false
Set "yes" or "no" to enable/disable ssl certifications checking for OAuth2 WARNING: it is not secure
3.0.x
50
redirect.url.for.external.users
Users entered the room via invitationHash or secureHash will be redirected to this URL on connection lost
3.0.x
51
google.analytics.code
Code for Google Analytics
3.1.0
52
#
external.process.ttl
20
Time to live in minutes for external processes such as conversion via ffmpeg (default 20 minutes)
3.3.0
53
personal.rooms.enabled
true
Users are allowed to create personal rooms
3.3.2
54
reminder.message
Reminder message to notify about upcoming appointment, generated message will be used if not set
2.0.x
55
#
mp4.audio.rate
22050
Audio sampling rate (in Hz) for MP4 video
4.0.1
56
mp4.audio.bitrate
32k
Audio bitrate for MP4 video
4.0.1
57
rest.allow.origin
List of addresses browser Ajax REST requests are allowed from
4.0.2
58
#
user.fname.minimum.length
4
Number of chars needed in a user first name
4.0.4
59
#
user.lname.minimum.length
4
Number of chars needed in a user last name
4.0.4
60
chat.send.on.enter
false
Controls if chat message will be set on Enter (default: send on Ctrl+Enter)
4.0.5
61
mp4.video.preset
medium
Preset (encoder optimization settings) to be used while performing mp4 conversion.Valid values are: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow
4.0.5
62
#
cam.fps
30
Camera FPS, should be positive number in range (0, 60]
5.0.0
63
#
mic.rate
22
The rate at which the microphone should capture sound, in kHz. The default value is 22 kHz.
5.0.0
64
mic.echo.cancellation
true
Whether or not echo cancellation is preferred and/or required.
5.0.0
65
mic.noise.suppression
true
Whether noise suppression is preferred and/or required.
5.0.0
66
display.name.editable
false
Is user will be able to edit his/her display name (default false).
4.0.7
67
start.quickpoll.keycode
Ctrl+Alt+KeyQ
A hot key code to start quick poll
4.0.10
68
auto.open.sharing
false
Whether shared screen should be auto-opened.
5.0.0
69
video.arrange.resize.keycode
Ctrl+Shift+KeyA
A hot key code to arrange video windows bottom-to-top with resize to 120x90
5.0.0
70
header.csp.font
https://fonts.gstatic.com
Value for 'font-src' directive of 'Content-Security-Policy' header (default: https://fonts.gstatic.com), more info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy
5.0.0
71
header.csp.frame
'self'
Value for 'frame-src' directive of 'Content-Security-Policy' header (default: SELF), more info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy
5.0.0
72
header.csp.image
'self',data:
Value for 'image-src' directive of 'Content-Security-Policy' header (default: 'self',data:), more info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy
5.0.0
73
header.csp.media
'self',data:
Value for 'media-src' directive of 'Content-Security-Policy' header (default: 'self',data:), more info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy
5.0.0
74
header.csp.script
'strict-dynamic'
Value for 'script-src' directive of 'Content-Security-Policy' header (default: STRICT_DYNAMIC), more info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy
5.0.0
75
header.csp.style
https://fonts.googleapis.com/css
Value for 'style-src' directive of 'Content-Security-Policy' header (default: https://fonts.googleapis.com/css), more info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy
5.0.0
76
mail.smtp.ssl.enable
false
Enable SSL
5.0.0
77
header.csp.enabled
true
Whether or not CSP secure headers are enabled
5.0.0
78
user.pass.check.upper
true
Whether or not Password MUST contain uppercase characters
5.0.1
79
user.pass.check.digit
true
Whether or not Password MUST contain numeric
5.0.1
80
user.pass.check.special
true
Whether or not Password MUST contain special character
5.0.1
81
#
appointment.pre.start.minutes
5
How many minutes before the start the room should be open (default: 5)
5.0.1
82
recording.enabled
true
Whether or not recording functionality is enabled
6.0.0
83
ui.theme
Sandstone
UI theme, possible values are Cerulean, Cosmo, Cyborg, Darkly, Flatly, Journal, Litera, Lumen, Lux, Materia, Minty, Pulse, Sandstone, Simplex, Sketchy, Slate, Solar, Spacelab, Superhero, United, Yeti
6.1.0


| LIN | NSN | Item Name | PBIC | Type | DND | OH Qty |
|---|---|---|---|---|---|---|
| SOLAR SYSTEM TY I-(5-4-09696) | H | LTT | N | 1 | ||
| TANK,FABRIC,COLLAPS-(50K GALLON | N | TPE | N | 4 | ||
| GEN ST DSL MEP-802A | H | LTT | N | 1 | ||
| HEATER D 120000BTU/HR | H | LTT | N | 2 | ||
| FILT SEP DL13217E9320 | N | TPE | N | 11 | ||
| METER ASSY-(PRESOL 40) | H | LTT | N | 4 | ||
| PUMP ADC 1500 | N | TPE | N | 2 | ||
| PUMP ASSY FLAMBL LIQ | H | LTT | N | 1 | ||
| PUMP ASSY FLAMBL LIQ | N | TPE | N | 5 | ||
| PUMP ASSEMB LC350AGPM | N | TPE | N | 1 | ||
| PUMPING ASSEMBLY,FL | N | TPE | N | 2 | ||
| PUMPING ASSEMBLY,FL-(350 GPM) | H | LTT | N | 1 | ||
| PUMPING ASSEMBLY,FL-(350 GPM) | N | TPE | N | 2 | ||
| PUMPING ASSEMBLY,FL-(350GPM) | N | TPE | N | 7 | ||
| TEST KIT PETRO FUEL | H | LTT | N | 1 | ||
| TANK ASSY 20000 PETRO | H | LTT | N | 1 | ||
| TR T W/W M978P1 | N | TPE | N | 1 | ||
| TRK TK FS M978 W/W | N | TPE | Y | 5 | ||
| TRUCK TANK M978A2 | N | TPE | Y | 1 | ||
| TRK TK FS M978 | N | TPE | Y | 3 | ||
| ALL TERRAIN VEHICLE M-GATOR: DIE | N | TPE | N | 1 |


Type Erasure with Small Buffer Optimization
===========================================
C++ London Meetup
May 2018
Gašper Ažman
Credits
-------
1. Basics: Sean Parent's _Better Code_.
2. Steven Watanabe's `boost::type_erasure`
3. My `liberasure` https://github.com/atomgalaxy/liberasure/
4. Louis Dionne's _Dyno_ is a newer take on this.
How To: Achieve Runtime Polymorphism with Regular Types
-------------------------------------------------------
(Inheritance + virtual functions force reference semantics)
(Refresher - what's Regular)
----------------------------
The term comes from Elements of Programming by Stepanov and McJones.
It's basically "like an int"
- DefaultConstructible (caveats)
- Copyable - and the copies don't alias!
- CopyConstructible
- CopyAssignable
- Movable
- MoveConstructible
- MoveAssignable
- Destructible
- EqualityComparable (and == is an equivalence relation)
"Solution": Inheritance + virtual functions
-------------------------------------------
Pros:
- natural interface
- people understand it
Cons:
- must inherit from specific base class
- need to provide "clone" or some other virtual function for copy.
- No way to model Regular.
- *really* hard to not do on the heap
- you pay for virtual method calls even when you _know_ the type
- no support for free functions
"Open" polymorphism
Solution: `std::variant`
-------------------------------------
Pros:
- contained types are freeform
- no variant, no runtime cost.
- really flexible
- no (inherent) heap allocation
Cons:
- fixed number of types
- contained types are baked in at variant definition
- awkward `std::visit`-based interface.
- no methods, only free functions (with visit)
- cannot model same concept as contained types
- easy to use, but basically impossible to make your own
- explaining variant takes a bit
"Closed" runtime polymorphism.
Solution: Classic Type Erasure -- Sean Parent-style
-----------------------------------------------------------
(like `std::function`)
Pros:
- "natural" interface
- free functions and member functions
- contained types not declared in advance
- contained types do not need to inherit from anything (but must support required interface)
- no wrapper, no runtime cost.
- can model same concept as contained types
- can act as a "lift" of static polymorphism to runtime
- the _standard co-worker_ can undestand it with 10 minutes of free time
- this has been tested
Cons:
- allocates on the heap
"Open" runtime polymorphism
Solution: Type Erasure + Small Buffer
-------------------------------------
"Closed" runtime polymorphism (to types smaller than buffer)
Pros:
- no heap allocation (for types smaller than buffer)
- natural interface
- free functions and methods
- contained types must support interface, but no need to inherit
- no wrapper, no runtime cost.
- can model same concept as contained types
- can act as a "lift" of static polymorphism to runtime
- the standard co-worker can undestand it with 15 minutes of free time
Cons:
- buffer is fixed size.
- types a bit constrained.
"Semi-Open" runtime polymorphism
Example: Simple JSON-like Language
----------------------------------
- elements:
- ints
- strings
- arrays of elements
- maps of string -> element
The Elements
------------
- `struct element;` (fwd declare)
- `struct null {};`
- `int`
- `std::string`
- `using array_t = std::vector<element>`
- `using map_t = std::map<std::string, element>`
Notice:
- We have no control over inheritance of elements.
- _int_ isn't even a class.
Write Into: `null`
-----------------
```cpp
void write_into(std::ostream& o, null x) {
o << "null";
}
```
Write Into: `int`
-----------------
```cpp
void write_into(std::ostream& o, int x) {
o << x;
}
```
Looks like: `5`
Write Into: `std::string`
-------------------------
```cpp
void write_into(std::ostream& o, std::string const& x) {
o << '"' << x << '"';
}
```
Looks like: `"foo"`
Write Into: `array_t`
---------------------
```cpp
void write_into(std::ostream& o, array_t const& x) {
o << '[';
for (auto const& elem : x) {
write_into(o, elem);
o << ", ";
}
o << ']';
}
```
Looks like: `[1, 2, "bar",]`
Write Into: `map_t`
-------------------
```cpp
void write_into(std::ostream& o, map_t const& x) {
o << '{';
for (auto const& elem : x) {
write_into(o, elem.first);
o << ": ";
write_into(o, elem.second);
o << ", ";
}
o << '}';
}
```
Looks like: `{1: [1, 2, "bar",], "foo": 3}`
Write Into: `element`?
----------------------
This one is more complicated. Let's see what `element` looks like.
Element: Interface
------------------
```cpp
struct element {
element() noexcept; // construct null
element(element&&) noexcept; // move-ctr
element(element const&); // copy-ctr
element& operator=(element) noexcept; // copy+move assign
```
Creation:
---------
```cpp
/* struct element { */
element(null) noexcept : element() {};
element(int) noexcept;
element(std::string&&) noexcept;
element(char const*); // extra for string
element(array_t&&) noexcept;
element(map_t&&) noexcept;
```
`EqualityComparable`
--------------------
```cpp
/* struct element { */
// bonus slides
friend
bool operator==(element const&, element const&);
friend
bool operator!=(element const& x, element const& y) {
return !(x==y);
}
```
Extras:
```
friend
void write_into(std::ostream& o, element const& elem);
```
Storage
-------
```cpp
/* struct element { */
private:
std::aligned_union<???>::type space; // small buffer
// accessors
auto ptr() {
return std::launder( // trust me, here be dragons.
reinterpret_cast<concept*>(&space)(;
}
auto ptr() const {
return std::launder( // same here.
reinterpret_cast<concept const*>(&space));
}
}; // end struct element
```
Recap #1
---------
- `element` is `Regular` (do as the `int`s do)
- `DefaultConstructible`
- `NoexceptMovable`
- `Copyable` (and copies don't alias)
- `EqualityComparable`
Recap #2
--------
Additionally:
- It's constructible from all the 5 types it can hold
- `null`
- `int`
- `std::string` (plus the convenience `char*`)
- `array_t`
- `map_t`
- it has `write_into`.
Implementation
--------------
First, the definition of the vtable. We'll call it `concept`.
Pure abstract base class.
```cpp
struct concept {
friend element; // ... helps.
virtual write_into(std::ostream&) const = 0;
/* ... */
virtual ~concept() {};
};
```
This is how we'll forward `write_into(o)`.
Usage in `element`
------------------
```cpp
/* struct element { */
friend
void write_into(std::ostream& o, element const&) {
ptr()->write_into(o);
}
```
Remember:
```cpp
private:
std::aligned_union<???>::type space; // small buffer
auto ptr() const {
return std::launder(
reinterpret_cast<concept const*>(&space));
}
```
Implementation: Model
---------------------
This is the main bit:
```cpp
template <typename ValueType>
struct model : concept {
// vtable, plus one member:
ValueType value;
void write_into(std::ostream& o) const override {
// remember, we defined these for all our basic types
write_into(o, *this);
}
};
```
You can now say you know how type erasure works.
Hiding `model` and `concept`
----------------------------
`model` and `concept` are private to `element`.
The user should never see them.
```cpp
struct element {
// all of these constructors look the same
element(map_t&& x) {
void* at = &space;
new (at) model<map_t>(std::move(x));
}
}
```
This requires `space` to be sized appropriately.
Sizing the Buffer
-----------------
The actual objects stored in there are `model<T>`,
so we need to size it appropriately.
```cpp
struct element {
std::aligned_union<1,
model<null>,
model<int>,
model<std::string>,
model<array_t>,
model<map_t>>::type
space;
}
```
How to move?
------------
```cpp
struct element {
element(element&& other) {
other.ptr()->move_construct_into(&space);
}
};
struct concept {
virtual void move_construct_into(void* at) = 0;
};
template <typename ValueType>
struct model : concept{
void move_construct_into(void* at) override {
new (at) model(std::move(*this));
}
};
```
How to copy?
------------
```cpp
struct element {
element(element const& other) {
other.ptr()->copy_construct_into(&space);
}
};
struct concept {
virtual void copy_construct_into(void* at) const = 0;
};
template <typename ValueType>
struct model : concept{
void copy_construct_into(void* at) const override {
new (at) model(*this);
}
};
```
The Assignment Operator
-----------------------
We take element by value. That way we already get
a constructed one, always.
If you want to optimize this, go ahead, make it more complicated.
```cpp
struct element {
element& operator=(element new_elem) {
// it's *never* us.
ptr()->~concept();
new_elem.ptr()->move_construct_into(&space);
return *this;
}
};
```
The Destructor
--------------
We take element by value. That way we already get
a constructed one, always.
If you want to optimize this, go ahead, make it more complicated.
```cpp
struct element {
~element() { ptr()->~concept(); }
};
```
Recap
-----
We only needed all the special functions on `element`.
We did not need them on either `concept` or `model` - those get the
rule of 1 (virtual destructor).
Example Usage
-------------
```cpp
auto config_file =
map_t{
{"foo", array_t{"abc", 1, 2, "cde"}},
{"bar", "baz"}};
write_out(std::out, config_file);
```
Thank You
=========
Questions?
Find the library at https://github.com/atomgalaxy/liberasure.


Lye Calculator
Step 1 - Type / Units
Choose your soap type and measurement units.
Type
Solid
Liquid
Unit
Ounces
Grams
Add Ingredients
Step 2 - Ingredients
What are your ingredient amounts by weight?
Almond Oil
Avocado Butter
Castor Oil
Kokum Butter
Mango Butter
Get Results
Step 3 - Results
Lye & Liquid
Amount
Lye (
)
of liquid
Total
Oils & Fats
Amount
%
Total
100 %
Totals
Amount
Lye & Liquid
Oils & Fats
Total Batch Yield
Recipe - Title
Recipe - Comments / Notes
Print Results


# NNPACK Contrib Installation
[NNPACK](https://github.com/Maratyszcza/NNPACK) is an acceleration package for neural network computations, which can run on x86-64, ARMv7, or ARM64 architecture CPUs. Using NNPACK, higher-level libraries like _MXNet_ can speed up the execution on multi-core CPU computers, including laptops and mobile devices.
*Note*: AS TVM already has natively tuned schedules, NNPACK is here mainly for reference and comparison purpose. For regular use prefer native tuned TVM implementation.
_TVM_ supports NNPACK for forward propagation (inference only) in convolution, max-pooling, and fully-connected layers. In this document, we give a high level overview of how to use NNPACK with _TVM_.
## Conditions The underlying implementation of NNPACK utilizes several acceleration methods, including [fft](https://arxiv.org/abs/1312.5851) and [winograd](https://arxiv.org/abs/1509.09308). These algorithms work better on some special batch size, kernel size, and stride settings than on other, so depending on the context, not all convolution, max-pooling, or fully-connected layers can be powered by NNPACK. When favorable conditions for running NNPACKS are not met,
NNPACK only supports Linux and OS X systems. Windows is not supported at present.
## Build/Install NNPACK
If the trained model meets some conditions of using NNPACK, you can build TVM with NNPACK support. Follow these simple steps: * Build NNPACK shared library with the following commands. _TVM_ will link NNPACK dynamically.
Note: The following NNPACK installation instructions have been tested on Ubuntu 16.04.
### Build [Ninja](https://ninja-build.org/)
NNPACK need a recent version of Ninja. So we need to install ninja from source.
`bash
git clone git://github.com/ninja-build/ninja.git
cd ninja
./configure.py --bootstrap
`
Set the environment variable PATH to tell bash where to find the ninja executable. For example, assume we cloned ninja on the home directory ~. then we can added the following line in ~/.bashrc.
`bash
export PATH="${PATH}:~/ninja"
`
### Build [NNPACK](https://github.com/Maratyszcza/NNPACK)
The new CMAKE version of NNPACK download [Peach](https://github.com/Maratyszcza/PeachPy) and other dependencies alone
```bash git clone –recursive https://github.com/Maratyszcza/NNPACK.git cd NNPACK # Add PIC option in CFLAG and CXXFLAG to build NNPACK shared library sed -i “s|gnu99|gnu99 -fPIC|g” CMakeLists.txt sed -i “s|gnu++11|gnu++11 -fPIC|g” CMakeLists.txt mkdir build cd build # Generate ninja build rule and add shared library in configuration cmake -G Ninja -D BUILD_SHARED_LIBS=ON .. ninja sudo ninja install
# Add NNPACK lib folder in your ldconfig echo “/usr/local/lib” > /etc/ld.so.conf.d/nnpack.conf sudo ldconfig ```
## Build TVM with NNPACK support
`bash
git clone --recursive https://github.com/dmlc/tvm
`
Set set(USE_NNPACK ON) in config.cmake.
Set NNPACK_PATH to the $(YOUR_NNPACK_INSTALL_PATH)
after configuration use make to build TVM
`bash
make
`


|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object | +--org.apache.catalina.core.ContainerBase
Abstract implementation of the Container interface, providing common
functionality required by nearly every implementation. Classes extending
this base class must implement getInfo()
, and may implement
a replacement for invoke()
.
All subclasses of this abstract base class will include support for a
Pipeline object that defines the processing to be performed for each request
received by the invoke()
method of this class, utilizig the
"Chain of Responsibility" design pattern. A subclass should encapsulate its
own processing functionality as a Valve
, and configure this
Valve into the pipeline by calling setBasic()
.
This implementation fires property change events, per the JavaBeans design
pattern, for changes in singleton properties. In addition, it fires the
following ContainerEvent
events to listeners who register
themselves with addContainerListener()
:
| Type | Data | Description |
|---|---|---|
addChild |
Container |
Child container added to this Container. |
addValve |
Valve |
Valve added to this Container. |
removeChild |
Container |
Child container removed from this Container. |
removeValve |
Valve |
Valve removed from this Container. |
start |
null |
Container was started. |
stop |
null |
Container was stopped. |
| Field Summary | |
protected java.util.HashMap |
children
The child Containers belonging to this Container, keyed by name. |
protected Cluster |
cluster
The cluster with which this Container is associated. |
protected int |
debug
The debugging detail level for this component. |
protected LifecycleSupport |
lifecycle
The lifecycle event support for this component. |
protected java.util.ArrayList |
listeners
The container event listeners for this Container. |
protected Loader |
loader
The Loader implementation with which this Container is associated. |
protected Logger |
logger
The Logger implementation with which this Container is associated. |
protected Manager |
manager
The Manager implementation with which this Container is associated. |
protected Mapper |
mapper
The one and only Mapper associated with this Container, if any. |
protected java.lang.String |
mapperClass
The Java class name of the default Mapper class for this Container. |
protected java.util.HashMap |
mappers
The set of Mappers associated with this Container, keyed by protocol. |
protected java.lang.String |
name
The human-readable name of this Container. |
protected Container |
parent
The parent Container to which this Container is a child. |
protected java.lang.ClassLoader |
parentClassLoader
The parent class loader to be configured when we install a Loader. |
protected Pipeline |
pipeline
The Pipeline object with which this Container is associated. |
protected Realm |
realm
The Realm with which this Container is associated. |
protected javax.naming.directory.DirContext |
resources
The resources DirContext object with which this Container is associated. |
protected static StringManager |
sm
The string manager for this package. |
protected boolean |
started
Has this component been started? |
protected java.beans.PropertyChangeSupport |
support
The property change support for this component. |
| Fields inherited from interface org.apache.catalina.Container |
ADD_CHILD_EVENT, ADD_MAPPER_EVENT, ADD_VALVE_EVENT, REMOVE_CHILD_EVENT, REMOVE_MAPPER_EVENT, REMOVE_VALVE_EVENT |
| Fields inherited from interface org.apache.catalina.Lifecycle |
START_EVENT, STOP_EVENT |
| Constructor Summary | |
ContainerBase()
|
| Method Summary | |
void |
addChild(Container child)
Add a new child Container to those associated with this Container, if supported. |
void |
addContainerListener(ContainerListener listener)
Add a container event listener to this component. |
protected void |
addDefaultMapper(java.lang.String mapperClass)
Add a default Mapper implementation if none have been configured explicitly. |
void |
addLifecycleListener(LifecycleListener listener)
Add a lifecycle event listener to this component. |
void |
addMapper(Mapper mapper)
Add the specified Mapper associated with this Container. |
void |
addPropertyChangeListener(java.beans.PropertyChangeListener listener)
Add a property change listener to this component. |
void |
addValve(Valve valve)
Add a new Valve to the end of the pipeline associated with this Container. |
Container |
findChild(java.lang.String name)
Return the child Container, associated with this Container, with the specified name (if any); otherwise, return null |
Container[] |
findChildren()
Return the set of children Containers associated with this Container. |
Mapper |
findMapper(java.lang.String protocol)
Return the Mapper associated with the specified protocol, if there is one. |
Mapper[] |
findMappers()
Return the set of Mappers associated with this Container. |
void |
fireContainerEvent(java.lang.String type,
java.lang.Object data)
Notify all container event listeners that a particular event has occurred for this Container. |
Valve |
getBasic()
Return the Valve instance that has been distinguished as the basic Valve for this Pipeline (if any). |
Cluster |
getCluster()
Return the Cluster with which this Container is associated. |
int |
getDebug()
Return the debugging detail level for this component. |
abstract java.lang.String |
getInfo()
Return descriptive information about this Container implementation and the corresponding version number, in the format <description>/<version> . |
Loader |
getLoader()
Return the Loader with which this Container is associated. |
Logger |
getLogger()
Return the Logger with which this Container is associated. |
Manager |
getManager()
Return the Manager with which this Container is associated. |
java.lang.String |
getName()
Return a name string (suitable for use by humans) that describes this Container. |
Container |
getParent()
Return the Container for which this Container is a child, if there is one. |
java.lang.ClassLoader |
getParentClassLoader()
Return the parent class loader (if any) for this web application. |
Pipeline |
getPipeline()
Return the Pipeline object that manages the Valves associated with this Container. |
Realm |
getRealm()
Return the Realm with which this Container is associated. |
javax.naming.directory.DirContext |
getResources()
Return the resources DirContext object with which this Container is associated. |
Valve[] |
getValves()
Return the set of Valves in the pipeline associated with this Container, including the basic Valve (if any). |
void |
invoke(Request request,
Response response)
Process the specified Request, to produce the corresponding Response, by invoking the first Valve in our pipeline (if any), or the basic Valve otherwise. |
protected void |
log(java.lang.String message)
Log the specified message to our current Logger (if any). |
protected void |
log(java.lang.String message,
java.lang.Throwable throwable)
Log the specified message and exception to our current Logger (if any). |
protected java.lang.String |
logName()
Return the abbreviated name of this container for logging messsages |
Container |
map(Request request,
boolean update)
Return the child Container that should be used to process this Request, based upon its characteristics. |
void |
removeChild(Container child)
Remove an existing child Container from association with this parent Container. |
void |
removeContainerListener(ContainerListener listener)
Remove a container event listener from this component. |
void |
removeLifecycleListener(LifecycleListener listener)
Remove a lifecycle event listener from this component. |
void |
removeMapper(Mapper mapper)
Remove a Mapper associated with this Container, if any. |
void |
removePropertyChangeListener(java.beans.PropertyChangeListener listener)
Remove a property change listener from this component. |
void |
removeValve(Valve valve)
Remove the specified Valve from the pipeline associated with this Container, if it is found; otherwise, do nothing. |
void |
setBasic(Valve valve)
Set the Valve instance that has been distinguished as the basic Valve for this Pipeline (if any). |
void |
setCluster(Cluster cluster)
Set the Cluster with which this Container is associated. |
void |
setDebug(int debug)
Set the debugging detail level for this component. |
void |
setLoader(Loader loader)
Set the Loader with which this Container is associated. |
void |
setLogger(Logger logger)
Set the Logger with which this Container is associated. |
void |
setManager(Manager manager)
Set the Manager with which this Container is associated. |
void |
setName(java.lang.String name)
Set a name string (suitable for use by humans) that describes this Container. |
void |
setParent(Container container)
Set the parent Container to which this Container is being added as a child. |
void |
setParentClassLoader(java.lang.ClassLoader parent)
Set the parent class loader (if any) for this web application. |
void |
setRealm(Realm realm)
Set the Realm with which this Container is associated. |
void |
setResources(javax.naming.directory.DirContext resources)
Set the resources DirContext object with which this Container is associated. |
void |
start()
Prepare for active use of the public methods of this Component. |
void |
stop()
Gracefully shut down active use of the public methods of this Component. |
| Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Field Detail |
protected java.util.HashMap children
protected int debug
protected LifecycleSupport lifecycle
protected java.util.ArrayList listeners
protected Loader loader
protected Logger logger
protected Manager manager
protected Cluster cluster
protected Mapper mapper
protected java.util.HashMap mappers
protected java.lang.String mapperClass
protected java.lang.String name
protected Container parent
protected java.lang.ClassLoader parentClassLoader
protected Pipeline pipeline
protected Realm realm
protected javax.naming.directory.DirContext resources
protected static StringManager sm
protected boolean started
protected java.beans.PropertyChangeSupport support
| Constructor Detail |
public ContainerBase()
| Method Detail |
public int getDebug()
public void setDebug(int debug)
debug
- The new debugging detail levelpublic abstract java.lang.String getInfo()
<description>/<version>
.
getInfo
in interface Container
public Loader getLoader()
null
.
getLoader
in interface Container
public void setLoader(Loader loader)
setLoader
in interface Container
loader
- The newly associated loaderpublic Logger getLogger()
null
.
getLogger
in interface Container
public void setLogger(Logger logger)
setLogger
in interface Container
logger
- The newly associated Loggerpublic Manager getManager()
null
.
getManager
in interface Container
public void setManager(Manager manager)
setManager
in interface Container
manager
- The newly associated Managerpublic Cluster getCluster()
null
.
getCluster
in interface Container
public void setCluster(Cluster cluster)
setCluster
in interface Container
public java.lang.String getName()
getName
in interface Container
public void setName(java.lang.String name)
setName
in interface Container
name
- New name of this container
java.lang.IllegalStateException
- if this Container has already been
added to the children of a parent Container (after which the name
may not be changed)public Container getParent()
null
.
getParent
in interface Container
public void setParent(Container container)
setParent
in interface Container
container
- Container to which this Container is being added
as a child
java.lang.IllegalArgumentException
- if this Container refuses to become
attached to the specified Containerpublic java.lang.ClassLoader getParentClassLoader()
getParentClassLoader
in interface Container
public void setParentClassLoader(java.lang.ClassLoader parent)
setParentClassLoader
in interface Container
parent
- The new parent class loaderpublic Pipeline getPipeline()
public Realm getRealm()
null
.
getRealm
in interface Container
public void setRealm(Realm realm)
setRealm
in interface Container
realm
- The newly associated Realmpublic javax.naming.directory.DirContext getResources()
null
.
getResources
in interface Container
public void setResources(javax.naming.directory.DirContext resources)
setResources
in interface Container
resources
- The newly associated DirContextpublic void addChild(Container child)
setParent()
method must be called, with this
Container as an argument. This method may thrown an
IllegalArgumentException
if this Container chooses not
to be attached to the specified Container, in which case it is not added
addChild
in interface Container
child
- New child Container to be added
java.lang.IllegalArgumentException
- if this exception is thrown by
the setParent()
method of the child Container
java.lang.IllegalArgumentException
- if the new child does not have
a name unique from that of existing children of this Container
java.lang.IllegalStateException
- if this Container does not support
child Containerspublic void addContainerListener(ContainerListener listener)
addContainerListener
in interface Container
listener
- The listener to addpublic void addMapper(Mapper mapper)
addMapper
in interface Container
mapper
- The corresponding Mapper implementation
java.lang.IllegalArgumentException
- if this exception is thrown by
the setContainer()
method of the Mapperpublic void addPropertyChangeListener(java.beans.PropertyChangeListener listener)
addPropertyChangeListener
in interface Container
listener
- The listener to addpublic Container findChild(java.lang.String name)
null
findChild
in interface Container
name
- Name of the child Container to be retrievedpublic Container[] findChildren()
findChildren
in interface Container
public Mapper findMapper(java.lang.String protocol)
null
.
findMapper
in interface Container
protocol
- Protocol for which to find a Mapperpublic Mapper[] findMappers()
findMappers
in interface Container
public void invoke(Request request, Response response) throws java.io.IOException, javax.servlet.ServletException
invoke
in interface Container
request
- Request to be processedresponse
- Response to be produced
java.lang.IllegalStateException
- if neither a pipeline or a basic
Valve have been configured for this Container
java.io.IOException
- if an input/output error occurred while
processing
javax.servlet.ServletException
- if a ServletException was thrown
while processing this requestpublic Container map(Request request, boolean update)
null
instead.
map
in interface Container
request
- Request being processedupdate
- Update the Request to reflect the mapping selection?public void removeChild(Container child)
removeChild
in interface Container
child
- Existing child Container to be removedpublic void removeContainerListener(ContainerListener listener)
removeContainerListener
in interface Container
listener
- The listener to removepublic void removeMapper(Mapper mapper)
removeMapper
in interface Container
mapper
- The Mapper to be removedpublic void removePropertyChangeListener(java.beans.PropertyChangeListener listener)
removePropertyChangeListener
in interface Container
listener
- The listener to removepublic void addLifecycleListener(LifecycleListener listener)
addLifecycleListener
in interface Lifecycle
listener
- The listener to addpublic void removeLifecycleListener(LifecycleListener listener)
removeLifecycleListener
in interface Lifecycle
listener
- The listener to removepublic void start() throws LifecycleException
start
in interface Lifecycle
java.lang.IllegalStateException
- if this component has already been
started
LifecycleException
- if this component detects a fatal error
that prevents it from being startedpublic void stop() throws LifecycleException
stop
in interface Lifecycle
java.lang.IllegalStateException
- if this component has not been started
LifecycleException
- if this component detects a fatal error
that needs to be reportedpublic void addValve(Valve valve)
setContainer
method must be called, with this Container
as an argument. The method may throw an
IllegalArgumentException
if this Valve chooses not to
be associated with this Container, or IllegalStateException
if it is already associated with a different Container.
addValve
in interface Pipeline
valve
- Valve to be added
java.lang.IllegalArgumentException
- if this Container refused to
accept the specified Valve
java.lang.IllegalArgumentException
- if the specifie Valve refuses to be
associated with this Container
java.lang.IllegalStateException
- if the specified Valve is already
associated with a different Containerpublic Valve getBasic()
Return the Valve instance that has been distinguished as the basic Valve for this Pipeline (if any).
getBasic
in interface Pipeline
public Valve[] getValves()
getValves
in interface Pipeline
public void removeValve(Valve valve)
removeValve
in interface Pipeline
valve
- Valve to be removedpublic void setBasic(Valve valve)
Set the Valve instance that has been distinguished as the basic
Valve for this Pipeline (if any). Prioer to setting the basic Valve,
the Valve's setContainer()
will be called, if it
implements Contained
, with the owning Container as an
argument. The method may throw an IllegalArgumentException
if this Valve chooses not to be associated with this Container, or
IllegalStateException
if it is already associated with
a different Container.
setBasic
in interface Pipeline
valve
- Valve to be distinguished as the basic Valveprotected void addDefaultMapper(java.lang.String mapperClass)
mapperClass
- Java class name of the default Mapperpublic void fireContainerEvent(java.lang.String type, java.lang.Object data)
type
- Event typedata
- Event dataprotected void log(java.lang.String message)
message
- Message to be loggedprotected void log(java.lang.String message, java.lang.Throwable throwable)
message
- Message to be loggedthrowable
- Related exceptionprotected java.lang.String logName()
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |


Pokemon: The First Movie 0 Stars (Out of 4) Reviewed by Mac VerStandig<EMAIL_ADDRESS>http://www.moviereviews.org November 7, 1999 USA Release Date: November 10, 1999
A copy of this review can be found at = http://www.moviereviews.org/pokemon_the_first_movie.htm
"I have created a monster." -Mary Shelley, Frankenstein
There are only two explanations for why anyone would make Pokemon: The First Movie: alien abduction or drugs.
Children were running around, waiting to get in hours before the movie started; one parent openly contemplated "How are we going to keep track of our kids?" Then the film began and near-deafening levels of applause were heard at the very sight of the opening credits. From there it was all downhill. Before long, people were walking out and cries of "I want to go home" replaced the children's glee. To say that the movie was a bomb would not do it justice. The movie was more like a bomb that is accidentally dropped at the wrong time over a country's closest ally, who just happens to have a massive nuclear arsenal and short temper.
The movie actually begins with a short film meant to introduce the audience to the world of Pokemon; a place filled with fictional characters, strange powers and human masters. Unfortunately, the film appeared to be little more than a fairy tale gone horribly astray. The characters' language was absolutely undecipherable and sheer pandemonium dominated the moment. The morals displayed for the film's young audience were despicable, including scenes where all the characters would point, stare and laugh at some poor creature who had fallen down. Worst of all, the piece doesn't accomplish its goal. Not only is the audience no closer to understanding the world of Pokemon, but the all-important first impression sets a negative tone that is constant throughout the feature film.
The nadir of all kids' movies this decade has a plot that makes that of Mission: Impossible seem self-explanatory. From the little that can be gathered, the Pokemon universe is in grave danger.  The creature comes with the standard villain-sized ego, boasting "Behold my powers! I am the strongest Pokemon in the world." This creation then takes on all other Pokemon in a battle that is haplessly drawn out over much of the film's duration.
The production does use some techniques displayed in older, and more successful, movies. Gloomy colors and equally dark music act as constant tip-offs that evil is approaching. Some sewer scenes wreak of borrowed elements from the old Ninja-Turtles television series.  And the memorable moment in Star Wars, that was tastefully recycled in Men in Black, of a bunch of strange looking creatures hanging around nonchalantly (Star Wars was in a bar, Men in Black in headquarters), is not so tastefully exhibited here.
At the end of the film, Mewtwo suggests to all the other Pokemon "Perhaps for you, these events are best forgotten." So be it.
---EDITORIAL CRITICISM OF THE MPAA---
The Motion Picture Association of America (MPAA) rated Pokemon: The First Movie "G." That is the lowest possible rating and means that the film may be seen by anyone and everyone.
Pokemon: The First Movie displays considerable amounts of violence. The entire plot is dependant on hand-to-hand style combat.  It must also be pointed out that the numerous people who will walk out mid-movie will never receive this anti-violence message anyway.
Another film opening this weekend, The Insider, was rated "R". That means that anyone under 17 who wishes to see the film must do so with a parent or guardian. The Insider is a true story that people read about in unrated newspapers a few years ago. The only remotely questionable aspect of the movie is foul language.
I guess what the MPAA and its president, Jack Valenti, are trying to say is that fighting in a deranged and far-fetched film is okay for everyone to see but a real life story with real life language should only be seen by adults.
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Protocols
The following protocols are available globally.
-
A notification dispatch mechanism protocol that enables the broadcast of information to registered observers.
See moreDeclaration
Swift
public protocol NotificationCenterProtocol
-
Represents a single state in a state machine.
See moreDeclaration
Swift
public protocol State


| Interface | Description |
|---|---|
| BytesMessage |
A
BytesMessage object is used to send a message containing a stream of uninterpreted bytes. |
| CompletionListener |
A
CompletionListener is implemented by the application and may be specified when a message is sent
asynchronously. |
| Connection |
A
Connection object is a client's active connection to its Jakarta Messaging provider. |
| ConnectionConsumer |
For application servers,
Connection objects provide a special facility for creating a
ConnectionConsumer (optional). |
| ConnectionFactory |
A
ConnectionFactory object encapsulates a set of connection configuration parameters that has been defined by
an administrator. |
| ConnectionMetaData |
A
ConnectionMetaData object provides information describing the Connection object. |
| DeliveryMode |
The delivery modes supported by the Jakarta Messaging API are
PERSISTENT and NON_PERSISTENT . |
| Destination |
A
Destination object encapsulates a provider-specific address. |
| ExceptionListener |
If a Jakarta Messaging provider detects a serious problem with a
Connection object, it informs the Connection
object's ExceptionListener , if one has been registered. |
| JMSConsumer |
A client using the simplified Jakarta Messaging API introduced for Jakarta Messaging 2.0 uses a
JMSConsumer object to receive messages
from a queue or topic. |
| JMSContext |
A
JMSContext is the main interface in the simplified Jakarta Messaging API introduced for Jakarta Messaging 2.0. |
| JMSProducer |
A
JMSProducer is a simple object used to send messages on behalf of a JMSContext . |
| MapMessage |
A
MapMessage object is used to send a set of name-value pairs. |
| Message |
The
Message interface is the root interface of all Jakarta Messaging messages. |
| MessageConsumer |
A client uses a
MessageConsumer object to receive messages from a destination. |
| MessageListener |
A
MessageListener object is used to receive asynchronously delivered messages. |
| MessageProducer |
A client uses a
MessageProducer object to send messages to a destination. |
| ObjectMessage |
An
ObjectMessage object is used to send a message that contains a serializable object in the Java programming
language ("Java object"). |
| Queue |
A
Queue object encapsulates a provider-specific queue name. |
| QueueBrowser |
A client uses a
QueueBrowser object to look at messages on a queue without removing them. |
| QueueConnection |
A
QueueConnection object is an active connection to a point-to-point Jakarta Messaging provider. |
| QueueConnectionFactory |
A client uses a
QueueConnectionFactory object to create QueueConnection objects with a point-to-point
Jakarta Messaging provider. |
| QueueReceiver |
A client uses a
QueueReceiver object to receive messages that have been delivered to a queue. |
| QueueSender |
A client uses a
QueueSender object to send messages to a queue. |
| QueueSession |
A
QueueSession object provides methods for creating QueueReceiver , QueueSender ,
QueueBrowser , and TemporaryQueue objects. |
| ServerSession |
A
ServerSession object is an application server object that is used by a server to associate a thread with a
Jakarta Messaging session (optional). |
| ServerSessionPool |
A
ServerSessionPool object is an object implemented by an application server to provide a pool of
ServerSession objects for processing the messages of a ConnectionConsumer (optional). |
| Session |
A
Session object is a single-threaded context for producing and consuming messages. |
| StreamMessage |
A
StreamMessage object is used to send a stream of primitive types in the Java programming language. |
| TemporaryQueue |
A
TemporaryQueue object is a unique Queue object created for the duration of a Connection . |
| TemporaryTopic |
A
TemporaryTopic object is a unique Topic object created for the duration of a Connection . |
| TextMessage |
A
TextMessage object is used to send a message containing a java.lang.String . |
| Topic |
A
Topic object encapsulates a provider-specific topic name. |
| TopicConnection |
A
TopicConnection object is an active connection to a publish/subscribe Jakarta Messaging provider. |
| TopicConnectionFactory |
A client uses a
TopicConnectionFactory object to create TopicConnection objects with a
publish/subscribe Jakarta Messaging provider. |
| TopicPublisher |
A client uses a
TopicPublisher object to publish messages on a topic. |
| TopicSession |
A
TopicSession object provides methods for creating TopicPublisher , TopicSubscriber , and
TemporaryTopic objects. |
| TopicSubscriber |
A client uses a
TopicSubscriber object to receive messages that have been published to a topic. |
| XAConnection |
The
XAConnection interface extends the capability of Connection by providing an XASession
(optional). |
| XAConnectionFactory |
The
XAConnectionFactory interface is a base interface for the XAQueueConnectionFactory and
XATopicConnectionFactory interfaces. |
| XAJMSContext |
The
XAJMSContext interface extends the capability of JMSContext by adding access to a Jakarta Messaging provider's
support for the Java Transaction API (JTA) (optional). |
| XAQueueConnection |
An
XAQueueConnection provides the same create options as QueueConnection (optional). |
| XAQueueConnectionFactory |
An
XAQueueConnectionFactory provides the same create options as a QueueConnectionFactory (optional). |
| XAQueueSession |
An
XAQueueSession provides a regular QueueSession , which can be used to create QueueReceiver ,
QueueSender , and QueueBrowser objects (optional). |
| XASession |
The
XASession interface extends the capability of Session by adding access to a Jakarta Messaging provider's
support for the Java Transaction API (JTA) (optional). |
| XATopicConnection |
An
XATopicConnection provides the same create options as TopicConnection (optional). |
| XATopicConnectionFactory |
An
XATopicConnectionFactory provides the same create options as a TopicConnectionFactory (optional). |
| XATopicSession |
An
XATopicSession provides a regular TopicSession . |
| Class | Description |
|---|---|
| QueueRequestor |
The
QueueRequestor helper class simplifies making service requests. |
| TopicRequestor |
The
TopicRequestor helper class simplifies making service requests. |
| Exception | Description |
|---|---|
| IllegalStateException |
This exception is thrown when a method is invoked at an illegal or inappropriate time or if the provider is not in an
appropriate state for the requested operation.
|
| IllegalStateRuntimeException |
This unchecked exception is thrown when a method is invoked at an illegal or inappropriate time or if the provider is
not in an appropriate state for the requested operation, and the method signature does not permit a
IllegalStateRuntimeException to be thrown. |
| InvalidClientIDException |
This exception must be thrown when a client attempts to set a connection's client ID to a value that is rejected by a
provider.
|
| InvalidClientIDRuntimeException |
This unchecked exception must be thrown when a client attempts to set a connection's client ID to a value that is
rejected by a provider, and the method signature does not permit a
InvalidClientIDException to be thrown. |
| InvalidDestinationException |
This exception must be thrown when a destination either is not understood by a provider or is no longer valid.
|
| InvalidDestinationRuntimeException |
This unchecked exception must be thrown when a destination either is not understood by a provider or is no longer
valid, and the method signature does not permit a
InvalidDestinationException to be thrown. |
| InvalidSelectorException |
This exception must be thrown when a Jakarta Messaging client attempts to give a provider a message selector with invalid syntax.
|
| InvalidSelectorRuntimeException |
This unchecked exception must be thrown when a Jakarta Messaging client attempts to give a provider a message selector with invalid
syntax, and the method signature does not permit a
InvalidSelectorException to be thrown. |
| JMSException |
This is the root class of all checked exceptions in the Jakarta Messaging API.
|
| JMSRuntimeException |
This is the root class of all unchecked exceptions in the Jakarta Messaging API.
|
| JMSSecurityException |
This exception must be thrown when a provider rejects a user name/password submitted by a client.
|
| JMSSecurityRuntimeException |
This unchecked exception must be thrown when a provider rejects a user name/password submitted by a client, or for
any case where a security restriction prevents a method from completing, and the method signature does not permit a
JMSSecurityException to be thrown. |
| MessageEOFException |
This exception must be thrown when an unexpected end of stream has been reached when a
StreamMessage or
BytesMessage is being read. |
| MessageFormatException |
This exception must be thrown when a Jakarta Messaging client attempts to use a data type not supported by a message or attempts to
read data in a message as the wrong type.
|
| MessageFormatRuntimeException |
This unchecked exception must be thrown when a Jakarta Messaging application attempts to use a data type not supported by a message
or attempts to read data in a message as the wrong type, and the method signature does not permit a
MessageFormatException to be thrown. |
| MessageNotReadableException |
This exception must be thrown when a Jakarta Messaging client attempts to read a write-only message.
|
| MessageNotWriteableException |
This exception must be thrown when a Jakarta Messaging client attempts to write to a read-only message.
|
| MessageNotWriteableRuntimeException |
This unchecked exception must be thrown when a Jakarta Messaging client attempts to write to a read-only message.
|
| ResourceAllocationException |
This exception is thrown when a provider is unable to allocate the resources required by a method.
|
| ResourceAllocationRuntimeException |
This unchecked exception is thrown when a provider is unable to allocate the resources required by a method.
|
| TransactionInProgressException |
This exception is thrown when an operation is invalid because a transaction is in progress.
|
| TransactionInProgressRuntimeException |
This unchecked exception is thrown when an operation is invalid because a transaction is in progress.
|
| TransactionRolledBackException |
This exception must be thrown when a call to
Session.commit results in a rollback of the current transaction. |
| TransactionRolledBackRuntimeException |
This unchecked exception must be thrown when a call to
JMSContext.commit results in a rollback of the current
transaction. |
| Annotation Type | Description |
|---|---|
| JMSConnectionFactory |
This annotation may be used on a field to specify the JNDI lookup name of a
jakarta.jms.ConnectionFactory to be
used when injecting a jakarta.jms.JMSContext object. |
| JMSConnectionFactoryDefinition |
An application may use this annotation to specify a Jakarta Messaging
ConnectionFactory resource that it requires in its operational environment. |
| JMSConnectionFactoryDefinitions |
Specifies one or more
JMSConnectionFactoryDefinition annotations. |
| JMSDestinationDefinition |
An application may use this annotation to specify a Jakarta Messaging
Destination resource that it requires in its operational environment. |
| JMSDestinationDefinitions |
Specifies one or more
JMSDestinationDefinition annotations. |
| JMSPasswordCredential |
This annotation may be used to specify the userName and password to be used when injecting a
jakarta.jms.JMSContext object. |
| JMSSessionMode |
This annotation may be used to specify the session mode to be used when injecting a
jakarta.jms.JMSContext
object. |


This template is a Scala-based, real-time word count example, using Apache Storm. We demonstrate how Scala can be used with ease to implement Storm components, thanks to its convenience of interoperation with Java.
This template contains two Storm topologies: one that uses in-memory maps and stateful dispatching to process and store word counts, and one that uses Hazelcast as an in-memory data grid.
Apache Storm is distributed, scalable and fault-tolerant open source real-time computation system. Storm has a simple API to create graphs of distributed components, called topologies.
To do real-time computation on Storm, you create what are called "topologies". A topology is a graph of computation. Each node in a topology contains processing logic, and links between nodes indicate how data should be passed around between nodes.
One of the core abstractions in Storm is the concept of a stream, which is an unbounded sequence of tuples. Each stream starts with a component called a spout that is responsible of emitting data tuples into the stream. The spout is typically combined with any number of bolts to process and transform the data.
Each node in a Storm topology executes in parallel. In your topology, you can specify how much parallelism you want for each node, and then Storm will spawn that number of threads across the cluster to do the execution.
A topology runs forever, or until you kill it. Storm will automatically reassign any failed tasks. Additionally, Storm guarantees that there will be no data loss, even if machines go down and messages are dropped.
Hazelcast is a scalable Java-based open source in-memory data grid. In-memory data grids are often used with databases in order to improve performance of applications, to distribute data and computation across servers, clusters and geographies, and to manage very large data sets or high data ingest rates.
Hazelcast can be used for the following use cases:
Storm uses tuples as its data model. A tuple is a named list of values, and a field in a tuple can be an object of any type. Out of the box, Storm supports all the primitive types, strings, and byte arrays as tuple field values. To use an object of another type, you just need to implement a serializer for the type.
Every node in a topology must declare the output fields for the tuples it emits. Here is an example:
override def declareOutputFields(declarer: OutputFieldsDeclarer) {
declarer.declare(new Fields("word", "count"))
}
Spout nodes in Storm topologies are responsible for emiting tuples into the topology. Spouts typically integrate with external data sources, like persistence data stores, messaging systems or web services. In this template we implemented a simple spout that sources data from a internal collection.
override def nextTuple(): Unit = {
collector.emit(List[AnyRef](nextSentence).asJava)
}
Bolt nodes in Storm topologies are responsible for processing the data received in tuples and either emitting tuples back into the topology or outside of the topology, into persistent data store, messaging system, web service, etc. We use Hazelcast as an external data store in HazelcastWordCounterBolt.
override def execute(input: Tuple) = {
val word = input.getStringByField("word")
val count = input.getIntegerByField("count")
wordCount.lock(word)
try {
val total = Option(wordCount.get(word))
wordCount.put(word, total.getOrElse(0) + count)
} finally {
wordCount.unlock(word)
}
collector.ack(input)
}
Topologies are graphs of stream transformations that can be submitted for the execution to Storm clusters.
TopologyBuilder
can be used to compose a topology by adding spouts and bolts accompanied by the number of desired instances of these nodes
across the Storm cluster and the type of grouping to be used for distributing data tuples along topology graph edges.
val builder = new TopologyBuilder
builder.setSpout("generator", new RandomSentenceGeneratorSpout, 1)
builder.setBolt("splitter", new SentenceSplitterBolt, 3).shuffleGrouping("generator")
builder.setBolt("counter", new InMemoryWordCounterBolt, 3).fieldsGrouping("split", new Fields("word"))
val topology: StormTopology = builder.createTopology
In this template, we demonstrate how to implement a word count functionality using Storm's distributed computation capabilities.
Counting words requires a source of text. RandomSentenceGeneratorSpout is a spout that emits a continuous stream of sentences being randomly picked from the predefined list.
The next stage in the processing flow is splitting sentences into words. This is accomplished by SentenceSplitterBolt
that splits sentences into words and emits tuples consisting of the word and the count of one (used for further aggregation).
Tasks in Storm are executed in parallel, so we need to tell storm how tuples should be routed between multiple nodes.
In case of SentenceSplitterBolt
we can use the shuffle grouping that will distribute sentence tuples evenly between all splitter tasks.
The last step of the topology involves counting all the word tuples.
InMemoryWordCounterBolt uses a mutable.Map
to count words.
We need to use field grouping on the word field to make sure word tuples are directed by Storm to the right bolt for counting.
If we opted for shuffle grouping each of the multiple InMemoryWordCounterBolt
s would be storing partial counts
and we would need to aggregate all the counts somehow (this is what the topology using Hazelcast addresses).
To see the basic word count topology in action run the application using InMemoryWordCountTopology main class.
In the basic implementation of the word-counting topology we use a mutable.Map
to store word counts in
InMemoryWordCounterBolt,
which combined with field grouping allows to effectively count words across a number of bolt instances.
The downside of this approach is that word counts are stored in memory across multiple mutable.Map
s
and there is no easy way to collate them as the topology processes the stream of sentences.
Apache Storm usually utilises external persistent stores or messaging solutions to push the outcome of stream processing to. In this template we use Hazelcast as a distributed in-memory cache to keep word counts outside of the Storm topology. HazelcastWordCounterBolt connects to the Hazelcast cluster and uses a distributed map to store word counts, which also means that we can use shuffle grouping. This map can be interrogated after the topology has been stopped to retrieve the final counts for all the words.
This is exactly what HazelcastWordCountTopology does, so if you run the application using this main class you will get the total word counts in the console output, even after the Storm cluster has been terminated.
Testing distributed computation systems is not an easy task, given the complexity such systems usually involve. Storm is not an exception in that regard, but luckily provides some facilities that help testing topologies in a controlled environment.
Before we discuss testing topologies, it definitely helps to verify the behaviour of their building blocks, spouts and bolts.
This template includes a RandomSentenceGeneratorSpoutSpec
that exercises the spout generating random sentences, as well as SentenceSplitterBoltSpec
that tests the bolt responsible for splitting sentences into words. We are using ScalaTest's WordSpec
to make sure test cases
are sufficiently descriptive and well structured. Mockito is used for mocking dependencies in unit tests.
When it come to testing word-counting bolts, we have to consider two variants and apply appropriate testing approach to both of them.
InMemoryWordCounterBoltSpec
tests the in-memory version of the bolt, and asserts that word counts are stored in mutable.Map
internal to the bolt class.
In HazelcastWordCounterBoltSpec
we need to take a different approach as the bolt communicates to the Hazelcast cluster over TCP and mocking out
any dependencies outside of the Storm API is not possible due to the distributed nature of Storm (only serializable
config properties can be used). The test creates a Hazelcast cluster and passes connection details to the bolt using the Storm
configuration mechanism. Hazelcast is then used to assert that the word counts that the bolt stores in the distributed map are correct.
With unit tests we can exercise spouts and bolts in isolation (or with minimal dependencies) but this alone won't give us full confidence that when all nodes are put together and working in parallel (as they do in Storm) the entire topology will behave as expected. HazelcastWordCountTopologyIntegrationSpec uses Storm's testing framework to run topology in a controlled environment and with the ability to inject test data as opposed to using our sentence generating spout. In this test we start a Hazelcast cluster again and validate the word counts after the topology has completed processing all the tuples injected by the framework.


Rising Up To The Challenge
Working within the IT Industry for over 15 years as a Business/Pre-Sales Consultant my roles have been focused on identification of business requirements needed to implement a strategic and tactical initiatives.
Returning to work after having my two beautiful children, I decided it was time to rise to the challenge and learn to 'build' that optimal solution.
So here I am studying to be a Full Stack Web Developer. With hard work and dedication anything is possible.
I love working in collaboration with my clients to build and deliver an optimal solution to meet their business needs.