class: center, middle # Building Data Pipelines with Apache Kafka ## Yaroslav Tkachenko --- # About me .left-column.column-30[ ![](http://1.gravatar.com/avatar/565c77691f1676e0bab82b4881361f55?size=200) - Data Pipelines - Big Data - Microservices - Distributed Systems - DevOps - ... [and more](http://sap1ens.com/files/resume-yaroslav-tkachenko.pdf) ] .right-column.column-70[ .text-center[ **Yaroslav (Slava) Tkachenko**, Vancouver, Canada ]
**Activision Blizzard**, 2+ years *Senior Software/Data Engineer → Software Architect* **Mobify**, 6 months *Senior Software Engineer, Lead* **Bench Accounting**, 5 years *Software Engineer → Engineering Lead → Director of Engineering* **Freelance**, 4 years *Web Developer* ] ??? - Ask everyone for 10 seconds introductions - Thank for participation - Feel free to interrupt me anytime and ask questions --- # Agenda - Data Pipelines 101: ingestion, storage, processing - Kafka fundamentals: topics, partitions, brokers, replication, etc. - Producer and Consumer APIs - Kafka Streams as a processing layer - Kafka Connect for integrating with external systems - Kafka best practices and tuning # Requirements - Basic Java or Scala knowledge - Docker / Docker Compose for running examples locally --- # Important links - Slides: https://sap1ens.com/workshop/craftconf-2019 - GitHub: https://github.com/sap1ens/kafka-workshop Once you've cloned GitHub repo: - `docker-compose -f docker-compose.connect.yml pull` - `docker pull maven:3.5.0-jdk-8` --- class: center, middle # Data Pipelines 101 --- # Data Pipelines 101
.text-bigger[ ***Data Pipeline*** *is a system capable of ingesting, processing and persisting data in a way that allows data transformation, exploration and analysis in accordance with business requirements.* ] --- # Data Pipelines 101 Message: - Metadata: timestamps, context, user, etc. - Payload: business or operational event (user registered, order shipped, log produced) --- # Data Pipelines 101 Data producers: - Web clickstream - Application logs - IoT - Streaming video/audio - ... Data consumers: - Reporting - Triggers - Visualization - Analytics - Data Science --- # Data Pipelines 101 Why not just send data directly from A to B? - Separation of concerns, less coupling - Scalability - Recovery from failures - Dynamic and/or flexible routing - Sampling, throttling, circuit breaking --- # Data Pipelines 101 - Log
.center[
### Log ] --- # Data Pipelines 101 - Log Why log? - Immutable - Maintains ordering - Many consumers - Can go back and forth in history Very different from a work queue: - Typically unordered - Messages are removed once consumed - Harder to support multiple consumers --- # Data Pipelines 101 .center[ ![](http://sap1ens-archive.s3-website-us-east-1.amazonaws.com/workshop-data-pipelines/pipeline.png) ### Data Pipeline ] --- # Data Pipelines 101 - Ingestion Traditionally telemetry data is ingested through some sort of **endpoint**, like HTTP API. Main requirements for this endpoint: - High scalability and reliability - Autoscaling - Throttling and circuit breaking - Minimum amount of 3rd-party calls (sometimes it means no auth!) - Efficient communication with the Log - Persistent TCP connections - Batching - Compression - Retries As any other API it may have a set of API clients for different platforms / languages.
Other ways could include: - Hourly or daily jobs for importing database snapshots - Bulk uploads from external services --- # Data Pipelines 101 - Processing Historically, **batch** and **stream** processing have been used in data pipelines. Batch: - Hourly or daily jobs that consume a big chunk of data and perform ETL (extract-transform-load) operations - Should have robust scheduling and workflow managment - Able to efficiently deduplicate and reorder data - Typically very high latency (hours) Stream: - Constantly running processing performing transformations on messages - Record-at-a-time or microbatch approach - Should be able to handle always changing traffic patterns - Typically very low latency (seconds, minutes)
Modern **lambda** and **kappa** architectres have strong emphasis on stream processing --- # Data Pipelines 101 - Storage Typically storage is handled by multiple different components: - Data Lake: raw data, partitioned and indexed - Data Warehouse: integrated collection of cleaned, processed and transformed data - Data Mart: simplified Data Warehouse view on a specific business entity Due to variety of Big Data frameworks and NoSQL there are multiple dimensions to consider: - OLTP vs OLAP - *Hot* vs *Cold* data - Facts vs Dimensions - Normalized vs Denormalized - Star Schema vs Snowflake Schema Real world example using AWS (Amazon Web Services): - S3 as a Data Lake - Hive on EMR / Redshift as a Data Warehouse (and Data Marts) - DynamoDb / Elasticache / Elasticsearch as additional caches and data stores --- # Data Pipelines 101 - Log .center[
] \* also Apache Pulsar, Amazon Kinesis, Google Pub/Sub, etc. --- class: center, middle # Kafka Fundamentals --- # Kafka Fundamentals
.center[
] --- # Kafka Fundamentals Kafka cluster: - Consists from one or many **brokers** - Requires Zookeeper cluster for *discovery* and *metadata persistence* - One of the brokers is elected as a *controller*. It's just a role that can be assigned to anyone - Kafka doesn't really have a "master" or "leader" node, so there is no SPOF - Consumers and producers can use *any* broker to bootstrap. Once they discover full cluster topology they start talking to the *right* broker(s) ??? Draw 3-node ZK cluster and 6-node Kafka cluster --- # Kafka Fundamentals
.center[
Topics are *partitioned* and *replicated* data streams ] --- # Kafka Fundamentals | Partition | Leaders | Replicas | In Sync Replicas | |-----------|---------|---------------|------------------| | 0 | 101 | (101,102,103) | (101,102,103) | | 1 | 102 | (102,103,104) | (102,103,104) | | 2 | 103 | (103,104,101) | (103,104) | | 3 | 104 | (104,101,102) | () | --- # Kafka Fundamentals Let's start Zookeeper and Kafka 🤞 Save this as `docker-compose.yml` and run `docker-compose up`. ```yaml version: '2.1' services: zookeeper: container_name: zookeeper image: 'confluentinc/cp-zookeeper:5.1.2' ports: - 2181:2181 environment: - ZOOKEEPER_CLIENT_PORT=2181 kafka: container_name: kafka image: 'confluentinc/cp-kafka:5.1.2' ports: - 9092:9092 depends_on: - zookeeper environment: - KAFKA_BROKER_ID=1 - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 - KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 - KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 ``` --- # Kafka Fundamentals Connect to the running Kafka container: ```sh docker exec -it kafka bash ``` Let's create the first topic: ```sh kafka-topics --zookeeper zookeeper --create --replication-factor 1 --partitions 10 --topic demo-topic ``` And another one with a single partition (you'll see why): ```sh kafka-topics --zookeeper zookeeper --create --replication-factor 1 --partitions 1 --topic demo-topic-single-partition ``` List & describe topics: ```sh kafka-topics --zookeeper zookeeper --list kafka-topics --zookeeper zookeeper --describe --topic demo-topic kafka-run-class kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic demo-topic ``` --- class: center, middle # Kafka Producers & Consumers --- # Kafka Producers & Consumers Kafka Message: - Key - Value - Headers Kafka Producer: - Sends a message to a specified topic Kafka Consumer: - Consumes messages from a specified topic (or topics) --- # Kafka Producers & Consumers Simple Java producer: ```java String topic = "demo-topic"; Integer count = 100; Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("acks", "all"); props.put("retries", 0); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); try (KafkaProducer
producer = new KafkaProducer<>(props)) { for(Integer i = 0; i < count; i++) { ProducerRecord
data = new ProducerRecord<>(topic, i.toString(), "value-" + i); producer.send(data); } } ``` --- # Kafka Producers & Consumers Simple Scala producer: ```scala val topic = "demo-topic" val count = 100 val props = new Properties props.put("bootstrap.servers", "localhost:9092") props.put("acks", "all") props.put("retries", 0) props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer") props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer") val producer = new KafkaProducer[String, String](props) try { for (i <- 0 to count) { val data = new ProducerRecord[String, String](topic, i.toString, "value-" + i) producer.send(data) } } finally { producer.close() } ``` --- # Kafka Producers & Consumers Simple Java consumer: ```java String topic = "demo-topic"; Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("auto.offset.reset", "earliest"); props.put("group.id", "demo-consumer"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); KafkaConsumer
consumer = new KafkaConsumer<>(props); consumer.subscribe(Collections.singletonList(topic)); while (true) { ConsumerRecords
records = consumer.poll(Long.MAX_VALUE); for (ConsumerRecord
record : records) { System.out.println(record.key() + "\t" + record.value()); } } ``` --- # Kafka Producers & Consumers Simple Scala consumer: ```scala val topic = "demo-topic" val props = new Properties props.put("bootstrap.servers", "localhost:9092") props.put("auto.offset.reset", "earliest") props.put("group.id", "demo-consumer") props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") val consumer = new KafkaConsumer[String, String](props) consumer.subscribe(Collections.singletonList(topic)) while (true) { val records = consumer.poll(Long.MaxValue).asScala for (record <- records) { println(record.key() + "\t" + record.value) } } ``` --- # Kafka Producers & Consumers Consumer **group id** is very important! - Controls how to distribute messages (Point-to-Point vs Publish-Subcribe) - Used as a unique id when persisting offsets
.center[ ![](https://kafka.apache.org/21/images/consumer-groups.png) ] --- # Kafka Producers & Consumers *Producing to Kafka directly* vs *using a proxy* Direct connection: - Simple and straightforward - Low latency - Can use DNS for *some* maintenance work Proxy: - Very flexible - Can do all kinds of maintenance behind the scenes - Some enrichment is possible --- # Kafka Producers & Consumers Things to consider when building a reliable producer: - Retries (handled by Kafka) - Batching (can be handled by Kafka or the application) - Throttling (Kafka provides quotas) - Sampling --- # Kafka Producers & Consumers It's also important to understand delivery guarantees and potential workarounds! At most once: - May or may not be received - No duplicates - Probably missing data At least once: - Delivery guaranteed - Possible duplicates - No missing data Exactly once: - Delivery guaranteed - No duplicates - No missing data --- # Kafka Producers & Consumers Kafka provides **at least once** delivery by default (via producer retries). Idempotence and transactions were introduced in 0.11, so **exactly once** delivery is also possible (and can be enabled in Kafka Streams with a single config change). ```java KafkaProducer producer = ... producer.initTransactions(); KafkaConsumer consumer = ... consumer.subscribe("inputTopic")); ConsumerRecords records = consumer.poll(Long.MAX_VALUE); try { producer.beginTransaction(); for (ConsumerRecord record : records) { producer.send(processAndProduceRecord("outputTopic", record)); } producer.sendOffsetsToTransaction(currentOffsets(consumer), groupId); producer.commitTransaction(); } catch (Exception e) { producer.abortTransaction(); } ``` --- # Kafka Producers & Consumers **Message headers** is a very useful concept in messaging and Kafka introduced message header support in 0.11. They're supported on the protocol level and recognized by most of the clients now. ```java List
headers = Arrays.asList( new RecordHeader("hkey1", "hvalue1".getBytes()), new RecordHeader("hkey2", "hvalue2".getBytes()) ); new ProducerRecord<>("topic", 0, "key", "value", headers); ``` --- # Kafka Producers & Consumers - Formats & Schemas ```java public interface Serde
{ Serializer
serializer(); Deserializer
deserializer(); } ``` ```java public interface Serializer
{ byte[] serialize(String topic, T data); default byte[] serialize(String topic, Headers headers, T data) { return serialize(topic, data); } } ``` ```java public interface Deserializer
{ T deserialize(String topic, byte[] data); default T deserialize(String topic, Headers headers, byte[] data) { return deserialize(topic, data); } } ``` --- # Kafka Producers & Consumers - Formats & Schemas ```java public class StringSerializer implements Serializer
{ private String encoding = "UTF8"; @Override public byte[] serialize(String topic, String data) { try { if (data == null) return null; else return data.getBytes(encoding); } catch (UnsupportedEncodingException e) { throw new SerializationException("Error when serializing string to byte[] due to unsupported encoding " + encoding); } } } ``` Other examples: - [JsonSerializer](https://github.com/apache/kafka/blob/trunk/connect/json/src/main/java/org/apache/kafka/connect/json/JsonSerializer.java), [JsonDeserializer](https://github.com/apache/kafka/blob/trunk/connect/json/src/main/java/org/apache/kafka/connect/json/JsonDeserializer.java) - [GenericAvroSerde](https://github.com/confluentinc/schema-registry/blob/master/avro-serde/src/main/java/io/confluent/kafka/streams/serdes/avro/GenericAvroSerde.java) - [KafkaProtobufSerde](https://github.com/daniel-shuy/kafka-protobuf-serde/blob/master/kafka-protobuf-serde/src/main/java/com/github/daniel/shuy/kafka/protobuf/serde/KafkaProtobufSerde.java) --- # Kafka Producers & Consumers - Formats & Schemas We often have some sort of metadata that should be passed together with a message. Message headers for metadata: - Very simple standard interface - Consumers don't need to fully deserialize the payload, which can be very helpful for filtering/routing - Very weak support in integration/archival solutions Message envelope for metadata: - Requires custom format (JSON, Protobuf, Avro, etc.) - Consumers must deserialize the payload in order to get metadata - Metadata will always be passed together with the payload Ideal solution: **combination of both** Also possible: custom text or binary protocol / wire format, [example](https://docs.confluent.io/current/schema-registry/serializer-formatter.html#wire-format): - Very efficient - Consumers don't need to fully deserialize the payload - Requires custom format --- # Kafka Producers & Consumers - Formats & Schemas **Schema Registry** is a data store and an API to upload and fetch message schemas. Most of the common formats like Avro or Protocol Buffers require schemas for serializing and deserializing messages. API could look like this: - POST /schema/**$guid** $payload - GET /schema/**$guid** Good Schema Registry supports **immutability**, **versioning** and **validation** for schemas. [Confluent Schema Registry](https://github.com/confluentinc/schema-registry) is a popular open-source Schema Registry from Confluent. **Guid** is typically transported as metadata (as a header, envelope field or custom format field) with every message. --- # Kafka Producers & Consumers - Formats & Schemas When modifying schemas, as long as a new **guid** is generated for a new schema, Kafka consumers will be able to deserialize it. Things might get complicated downstream from that, for example when writing records to an external data store. Most of the data formats have a way to deal with **Schema Evolution**: - Avro supports different reader and writer schemas - Protocol buffers use tag numbers to keep schemas compatible --- class: center, middle # Kafka Streams --- # Kafka Streams Stream processing: - Constantly running processing performing transformations on messages/records - Record-at-a-time or microbatch approach - Should be able to handle always changing traffic patterns - Typically very low latency (seconds, minutes) Main advantages: - Low latency gives business agility - Stateful processing allows windowing, joining and aggregations - Can use Kafka as input AND output Challenges: - Requires different mindset - Operational complexity (reliability and scalability) - Hard to recalculate / reprocess historical data (for example 1+ year) --- # Kafka Streams Why **Kafka Streams**? Other popular stream processing frameworks include **Apache Spark** and **Apache Flink**, however Kafka Streams has one very significant distinction:
.center[ _Kafka Streams is just a library_ ]
It doesn't need a special scheduler like YARN, it doesn't rely on any Hadoop dependencies, there is no "cluster". It simply uses Kafka as a data store and it leverages Kafka Consumer protocol internally. It can be added to any existing JVM application. However, most of the concepts and operations described next would generally apply to any stream processing framework, like Apache Flink or Apache Spark. --- # Kafka Streams - Concepts It's possible to run one or many **applications**. Each application can have multiple **instances**. Each instance runs a JVM process that uses Kafka Streams *library*. Each instance defines a **processor topology**: - can be defined with *Streams DSL* or low-level *Processor API* - can be stateless or stateful --- # Kafka Streams - Concepts ```java Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-app-v1"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, "org.apache.kafka.common.serialization.Serdes$StringSerde"); props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, "org.apache.kafka.common.serialization.Serdes$StringSerde"); props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once"); StreamsBuilder builder = new StreamsBuilder(); // topology KafkaStreams streams = new KafkaStreams(builder.build(), props); streams.start(); ``` --- # Kafka Streams - Concepts ```java KStream
stream = builder.stream("input-topic"); KStream
transformed = stream.map( (key, value) -> KeyValue.pair(value.toLowerCase(), value.length())); stream.to("output-topic", Produced.with(Serdes.String(), Serdes.Integer()); ``` ```scala import org.apache.kafka.streams.scala.Serdes._ import org.apache.kafka.streams.scala.ImplicitConversions._ val stream: KStream[String, String] = builder.stream("input-topic") val transformed: KStream[String, Integer] = stream.map((_, value) => (value.toLowerCase, value.length)) stream.to("output-topic") ``` Stateless transformations: *map, mapValues, flatMap, flatMapValues, filter, selectKey, foreach, peek* --- # Kafka Streams - Concepts The stream-table duality describes the close relationship between **streams** and **tables**: - Stream as Table: A stream can be considered a _changelog_ of a table, where each data record in the stream captures a state change of the table - Table as Stream: A table can be considered a _snapshot_, at a point in time, of the latest value for each key in a stream --- # Kafka Streams - Concepts .center[
] --- # Kafka Streams - Concepts Simple counter implementation, let's look at an example: ```java KStream
stream = builder.stream("input-topic"); KTable
aggregatedTable = stream .groupByKey() .count(); aggregatedTable .toStream() .to("output-topic"); ``` Other aggregations: *aggregate, reduce* --- # Kafka Streams - Concepts Time in stream processing: - Event-time - Processing-time - Ingestion-time Timestamp extractor instructs Kafka Streams what timestamp to use. Examples of built-in extractors: _FailOnInvalidTimestamp_, _WallclockTimestampExtractor_. It's pretty common to implement a custom extractor: ```java public interface TimestampExtractor { long extract(ConsumerRecord
record, long previousTimestamp); } ``` --- # Kafka Streams - Concepts Windowing: - **Hopping**: fixed-size, overlapping windows - **Tumbling**: fixed-size, non-overlapping, gap-less windows - **Sliding**: fixed-size, overlapping windows that work on differences between record timestamps - **Session**: dynamically-sized, non-overlapping, data-driven windows Hopping & tumbling windows are very similar: ```java .windowedBy(TimeWindows.of(Duration.ofMinutes(5).advanceBy(Duration.ofMinutes(1)))) .windowedBy(TimeWindows.of(Duration.ofMinutes(5))) ``` Sliding windows are only applicable to joins: ```java JoinWindows.of(Duration.ofMinutes(5)) ``` Session windows allow creating per-key sessions: ```java .windowedBy(SessionWindows.with(Duration.ofMinutes(5))) ``` --- # Kafka Streams - Concepts Now counter with windows: ```java KStream
stream = builder.stream("input-topic"); KTable
aggregatedTable = stream .groupByKey() .windowedBy(TimeWindows.of(Duration.ofSeconds(15))) .count(); aggregatedTable .toStream() .to("output-topic"); ``` --- # Kafka Streams - Concepts How to handle out of order messages? Kafka Streams indtroduces grace window period: ```java .windowedBy(TimeWindows.of(Duration.ofHours(1)).grace(Duration.ofMinutes(10))) ``` More on this topic: - [Streaming 101: The world beyond batch](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101) - [Streaming 102: The world beyond batch](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102) > *A **watermark** is a notion of input completeness with respect to event times. A watermark with a value of time X makes the statement: “all input data with event times less than X have been observed.” As such, watermarks act as a metric of progress when observing an unbounded data source with no known end.* --- # Kafka Streams - Concepts Finally, windowed counter with suppression of intermediate updates: ```java KStream
stream = builder.stream("input-topic"); KTable
aggregatedTable = stream .groupByKey() .windowedBy(TimeWindows.of(Duration.ofSeconds(15)).grace(Duration.ofSeconds(5))) .count() .suppress(Suppressed.untilWindowCloses(unbounded())); aggregatedTable .toStream() .to("output-topic"); ``` --- # Kafka Streams - Concepts | Join operands | Type | (INNER) JOIN | LEFT JOIN | OUTER JOIN | |-------------------------|--------------|---------------|---------------|---------------| | KStream-to-KStream | Windowed | Supported | Supported | Supported | | KTable-to-KTable | Non-windowed | Supported | Supported | Supported | | KStream-to-KTable | Non-windowed | Supported | Supported | Not Supported | | KStream-to-GlobalKTable | Non-windowed | Supported | Supported | Not Supported | | KTable-to-GlobalKTable | N/A | Not Supported | Not Supported | Not Supported | --- # Kafka Streams - Concepts Joining mechanics: - Joins are performed on message keys. It's possible to select a different message key by using one of the following methods: *selectKey* (obviously), *map*, *flatMap* - All input topics must be **co-partitioned** (have the same number of partitions and use the same partitioning strategy). This doesn't apply to joins with `GlobalKTable` - Input messages with null keys are typically ignored ```java KStream
left = ...; KTable
right = ...; KStream
joined = left.leftJoin(right, (leftValue, rightValue) -> "left=" + leftValue + ", right=" + rightValue, Joined.keySerde(Serdes.String()).withValueSerde(Serdes.String()) ); ``` --- # Kafka Streams - Concepts - **At-least-once** semantics, enabled by default (`processing.guarantee="at_least_once"`). Rely on basic Kafka Producer & Consumer functionality - **Exactly-once** semantics, has to be enabled explicitly (`processing.guarantee="exactly_once"`). Rely on Kafka transactions and idempotent producers (covered before) --- # Kafka Streams - Concepts Kafka Streams provides low-level API for defining: - **Transformers**, via either `Transformer` or `ValueTransformer` interface - **Processor**, via `Processor` interface - They have access to `ProcessorContext`, which contains low-level Kafka metadata like partition and offset - When implementing stateful Transformers or Processors, **State Store** is used for persisting required state: - Can store data in memory or built-in RocksDB database - Can be windowed - Backed by Kafka changelog --- # Kafka Streams - Concepts ```java public class PageEmailNotificationAlert implements Processor
{ private ProcessorContext context; @Override public void init(ProcessorContext context) { this.context = context; // any additional initialization } @Override void process(PageId pageId, PageMetadata metadata) { // formatting and sending emails } @Override void close() { // cleanup if required } } ``` Using in DSL: ```java .process(() -> new PageEmailNotificationAlert()); ``` --- # Kafka Streams - Concepts Kafka Streams may create internal topics, typically following `
-
-
` convention. - **Repartition** topics are created every time a message key is changed and required for further stateful processing (for example for aggregation or joining) - **Changelog** topics are created for backing state stores (for example for aggregation, windowing and or when using a custom state store in Processor/Transformer) **These topics could add significant overhead to Kafka Cluster!** It's important to monitor and tweak them if needed. Sometimes rewriting complicated DSL topology with a lot of stateful operators with a simple Processor or Transformer can reduce that overhead significantly. --- # Kafka Streams - Concepts .center[
] --- # Kafka Streams - Concepts Kafka Streams uses the concepts of **stream partitions** and **stream tasks** as logical units of its parallelism model. There are close links between Kafka Streams and Kafka in the context of parallelism: - Each **stream partition** is a totally ordered sequence of data records and maps to a Kafka topic partition. - A data record in the stream maps to a Kafka message from that topic. - The keys of data records determine the partitioning of data in both Kafka and Kafka Streams, i.e., how data is routed to specific partitions within topics. So, the maximum parallelism at which your application may run is bounded by the maximum number of stream tasks, which itself is determined by **maximum** number of partitions of the input topic(s) the application is reading from. --- # Kafka Streams - Concepts .center[
] --- # Kafka Streams - Concepts .center[
] --- # Kafka Streams - Example Now let's combine what we learned together: ```java KTable
users = builder.table("users"); KStream
usersActivity = builder.stream("user_activity"); KTable
, Long> activityByConsole = usersActivity .leftJoin(users, (activity, user) -> { if (user != null) { Map
headers = activity.getMetadata(); headers.put("console", user.getConsole().toString()); } return activity; }) .map((userId, activity) -> new KeyValue<>(activity.getMetadata().get("console"), activity)) .groupByKey() .windowedBy(TimeWindows.of(Duration.ofSeconds(15))) .count(); activityByConsole .toStream((windowedConsole, count) -> windowedConsole.toString()) .to("user_activity_by_console", Produced.with(Serdes.String(), Serdes.Long())); ``` --- # Kafka Streams **Interactive queries** is a powerful technique to access the state of the application (without using any external data stores). - Local state is easy to query programmatically: ```java ReadOnlyKeyValueStore
keyValueStore = streams.store("CountsKeyValueStore", QueryableStoreTypes.keyValueStore()); keyValueStore.get("hello"); // also, 'range', 'all', etc. ``` - Querying global / remote state over RPC (HTTP): - Configure each *instance* of an application with `application.server` (`host:port`) - *Implement* the RPC endpoint - Now instances can discover each other's state stores! And forward request/responses appropriately --- # Kafka Streams - Applications in Data Pipelines - Data transformation - ETL, analytics, ML - Deduplication - Using custom state stores - Enrichment - Using joins - Running aggregations --- class: center, middle # Kafka Connect --- # Kafka Connect .center[
] --- # Kafka Connect .center[ ![](http://sap1ens-archive.s3-website-us-east-1.amazonaws.com/workshop-data-pipelines/connect.png) ] --- # Kafka Connect - Data integration framework, open-sourced and Kafka-native - Stop writing custom adhoc integration components between system A and B! Instead, use connectors: - **Source connector** for sending data from a data store to Kafka - **Sink connector** for sending data from Kafka to a data store - State is kept in Kafka - Configuration driven - REST API is provided - Tons of connectors are available already - Flexible, scalable and extendable --- # Kafka Connect - Concepts - Workers and tasks - Connectors - Converters - Transforms --- # Kafka Connect - Concepts - Standalone vs Distributed Workers - Kafka Connect is just a Worker JVM process - Can be configured with Property files - Custom extensions (your converters, 3rd-party connectors, etc.) should be available in Java Classpath - After that they can be referred with a full classname, like `com.blueapron.connect.protobuf.ProtobufConverter` - Workers create Tasks responsible for actual work - Tasks are coordinated & rebalanced using a protocol based on consumer groups --- # Kafka Connect - Concepts .center[ ![](http://sap1ens-archive.s3-website-us-east-1.amazonaws.com/workshop-data-pipelines/workers.png) ] --- # Kafka Connect - Concepts - Kafka Connect uses [internal data format](https://github.com/apache/kafka/tree/trunk/connect/api/src/main/java/org/apache/kafka/connect/data), it supports primitives and complex types like structs, maps and arrays - All messages coming to and from Kafka must be converted to/from that internal format using a specified *Converter* - **Why?** In a truly generic system you don't want to implement JSON Elasticsearch connector or Avro S3 connector... If serialization format is generic enough, all connectors can be reused! And it makes possible to apply generic *Transforms*. .center[ ![](http://sap1ens-archive.s3-website-us-east-1.amazonaws.com/workshop-data-pipelines/connect-converters.png) ] --- # Kafka Connect - Concepts - *Transforms* allows to use simple, declarative DSL to define lightweight transformations on data. Can be used with source connectors for pre-processing or sink connectors for post-processing - Built-in transforms: Cast, Drop, ExtractField, ExtractTopic, Flatten, HoistField, InsertField, MaskField, RegexRouter, ReplaceField, SetSchemaMetadata, TimestampConverter, TimestampRouter, ValueToKey - You can implement your own Transforms! Just compile and put them in Java classpath - Example: ``` transforms=InsertKey, CastLong transforms.InsertKey.type=org.apache.kafka.connect.transforms.ValueToKey transforms.InsertKey.fields=id transforms.CastLong.type=org.apache.kafka.connect.transforms.Cast$Key transforms.CastLong.spec=int64 ``` --- # Kafka Connect - Example Let's use Kafka Connect and try to send a text file to and from Kafka. Use `docker-compose.connect.yml` to start up everything that's required with `docker-compose -f docker-compose.connect.yml up` --- # Kafka Connect - Applications in Data Pipelines - SQL and NoSQL *data capture* with **source** connectors (*Change Data Capture*) - Example: MySQL connector capturing and streaming updates for a set of tables - Delivering processed data to various destinations (Data Marts, BI, application integration) via **sink** connectors - Example: using ElasticSearch connector for sending messages to run adhoc queries and search; using InfluxDB connector for sending metrics to build dashboards - Data *archival* with **sink** connectors - Example: capturing and saving data after various transformations in the pipeline to AWS S3 or HDFS --- class: center, middle # Running Kafka in Production --- # Running Kafka in Production Kafka brokers tuning and optimizations: - Kafka likes hardware with good IO, so pay attention to good disks (however SSDs are not required), use XFS or EXT4 with some tuning (https://kafka.apache.org/documentation/#generalfs) - If you're not bottlenecked on disk IO, you're bottlenecked on network IO - File descriptor limits should be updated - Consider running Kafka brokers with 16 CPU cores and using 8 `num.io.threads` and 8 `num.network.threads` (defaults are 8 and 3) - Kafka brokers don't need a lot of heap, but they use pagecache a lot - Kafka broker startup scripts have basic JVM optimizations already - Large messages require special tuning --- # Running Kafka in Production Kafka exposes hundreds of metrics over JMX. Some of them should be used for dashboarding and alerting: - Incoming msg/s, bytes/s, msg/s per topic, bytes/s per topic - Outgoing bytes/s - Leader and partition counts - **Offline and under-replicated partitions** - **Active controller count** - ISR expand and shrink rates - Leader and follower replication byte rates - **CPU, RAM and JVM metrics** - **Disk usage, read and write throughput** - **Network and IO threads idle %** - **Zookeeper disconnects** --- # Running Kafka in Production **End-to-end** testing for Kafka involves sending a series of messages to the input endpoint, making sure they're passed through the whole pipeline and recording them at the end of the pipeline. No duplicates or lost data is expected. [kafka-monitor](https://github.com/linkedin/kafka-monitor) is one of the potential solutions. --- # Running Kafka in Production Kafka has a powerful built-in CLI, however sometimes we want to use UI too (can be helpful when exposing Kafka to other teams with less experience). [kafka-manager](https://github.com/yahoo/kafka-manager) is a de-facto standard: .center[
] --- # Running Kafka in Production Typical Kafka maintenance tasks include: - **Rolling restarts** for appliying broker configuration changes. Simply restarting brokers one by one by triggering a controlled JVM shutdown. With replication and min-ISR > 2 nothing should be affected - **Blue-green** deployments could be used for applying bigger changes (for example going from 0.8 to 0.10) - **Preferred replica leader election** is a mechanism for ensuring that a leader for a given partition should be a "preferred replica" (first in a list of replicas). It can be triggered with CLI - **Partition reassignment** is required when a broker is completely lost. Simply adding new broker doesn't rebalance existing partitions, it's a manual process that can be triggered with CLI --- class: center, middle # Data Pipelines 201 --- # Data Pipelines 201 If we have time we can talk about: - Kafka topic naming conventions - Routing (merging and splitting, filtering) - Getting message schemas right is important! - Data governance and linage - Self-serve ("on-demand") streams - Data reprocessing and deduplication - Hive streaming ingestion; handling small files problem --- # Links - https://kafka.apache.org/documentation/ - https://www.confluent.io/resources/kafka-the-definitive-guide/ - https://docs.confluent.io/current/build-applications.html - https://github.com/confluentinc/kafka-streams-examples - https://www.manning.com/books/kafka-streams-in-action - http://kappa-architecture.com - https://dataintensive.net - https://www.enterpriseintegrationpatterns.com - https://www.manning.com/books/big-data - http://streamingsystems.net --- class: center, middle # We did it! Thank you! Please answer quick servey: https://goo.gl/forms/xR9CNke7mhMdq0qk2


The first time, you need to wait for the permission pop-up (might take some time) and accept it. Then, set the key and location to get the current temperature.
Temperature: ℃


Snowball: Quick introduction |
Links |
|
You can use this site at a number of levels: - You can look at the stemming algorithm definitions themselves, and use them as templates for coding your own versions of stemmers in the computer language of your choice. - You can use the various ANSI C and Java stemmers in programs of your own, without bothering yourself with the Snowball system that generated them. To do that, download either the C or the Java version of the libstemmer library, and follow the instructions contained in the README files within these tarballs. The tarballs also contain simple example programs which allow you to run the stemmers from the command line. - You can get involved in Snowball itself. This is particularly worthwhile if you want to adjust the stemmers or develop new stemmers. A typical reason for adjusting the stemmers is that you are working with a different encoding of accented letters from the ISO Latin I encoding assumed in most of the scripts here. Then you need to make your own version of the Snowball compiler and work with the Snowball scripts.
|


Full name
Username
+23400008893
Login
Register
Profile
Appointment Records
Health Record
Drug Record
Meet Experts
Sign Out
Setting
Profile
Profile name
email@address
+234
Medical Expert Inform
Medical Expert name
Office Line
+234


Languages have been my passion for the past 15 years! On my path of self discovery I have stepped into the realm of software engineering and applied my honed skills as a language teacher to acquiring technical languages. I have gained valuable experience using Javascript, HTML, CSS, React, Node, Heroku, PostgreSQL, and API Management.
Each learning experience in coding is amazing and possibilities are endless! Working with teams, teaching, and learning come natural given my experience as an educator and leader. Why am I a Resilient Developer? Because regardless of what challenge I am faced with I work through it with success! The sky is the limit!
View ResumeThis Full Stack App was built to allow clients to store their information such as a username and password, register and sign into their account, and upload and submit photos. Once clients submit photos, the Clarifai API returns photos with all faces detected within a box.
Robofriends is a totally responsive app that demonstrates how to use a random robot generator and the JSON placeholder API to generate cards with random robot pictures and ID information.
This app was created to incorporate Angular and Spring Boot skills learned in my Java Bootcamp into one full stack working application. There are login, welcome, task, and log out page components as well as web security via Json Web Token. Users are able to update, add and delete tasks and those changes are updated in the backend H2 database.
<EMAIL_ADDRESS>Call to Action


Chris Perkins
Partner
Chris Perkins is an expert in quantitative and qualitative analysis. He’s considered one of the most accurate pollsters and top public opinion experts in the United States. Since 2010, Chris has polled for nearly one hundred winning Republican candidates for Governor, U.S. Senate, U.S. Congress, and a variety of other offices. Chris has also been a key resource for numerous organizations tackling public policy issues, including Fortune 500s, trade associations, and non-profit organizations.
When he's not wading through petabytes of data, Chris can usually be found at one of his kids’ baseball games, hosting friends to some of the best block parties in Austin, or explaining the virtues of Austin barbecue to the uninitiated. And rest assured, while Chris’s barbecue is never over seasoned, his highly detailed memory of every campaign he's ever touched intimidates even his most seasoned employees.


Removal Issues!
-
heggaton
I decided after a quick eval, I didn't want to keep winswitch. Unfortunately (on my OS X.4.3 system) whenever I restart, my user switching menu never returns!
I tried to re-install and same thing. If I log directly into my account, the user switch menu doesn't come up.
Now, if I log into the other account on this computer, before I log into mine, and then log into my account, it all works fine.
Help is needed! this is painful!
-
Greg Hurrell
1. Command-drag WinSwitch out of your menu bar.
2. In the "Accounts" system preference pane, turn on Fast User Switching (to add the Apple Fast User Switching menu extra back into the menu bar).
-
heggaton
Tried that, and it comes back. But the above reoccurs
-
Greg Hurrell
Sounds like you have a problem unrelated to WinSwitch. WinSwitch doesn't touch the Accounts preference pane in any way, so if you're having a problem with it then it means something else is the cause.
Reply
This topic is now closed.


twilio.rest.preview.hosted_numbers.authorization_document package¶
Submodules¶
twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order module¶
This code was generated by / _ _ _| _ _
(_)/(_)(_|/| |(/_ v1.0.0 / /
Bases:
twilio.base.instance_resource.InstanceResource
PLEASE NOTE that this class contains preview products that are subject to change. Use them with caution. If you currently do not have developer preview access, please contact<EMAIL_ADDRESS>Bases:
object
Bases:
object
Returns: Account sid. Return type: unicode
Returns: Address sid. Return type: unicode
Returns: Seconds (0-30) to delay ownership verification call by. Return type: unicode
Returns: A mapping of phone number capabilities. Return type: unicode
Returns: A list of emails. Return type: list[unicode]
Returns: The date this HostedNumberOrder was created. Return type: datetime
Returns: The date this HostedNumberOrder was updated. Return type: datetime
Returns: Email. Return type: unicode
Returns: Phone extension to use for ownership verification call. Return type: unicode
Returns: Why a hosted_number_order reached status “action-required” Return type: unicode
Returns: A human readable description of this resource. Return type: unicode
Returns: IncomingPhoneNumber sid. Return type: unicode
Returns: An E164 formatted phone number. Return type: unicode
Returns: HostedNumberOrder sid. Return type: unicode
Returns: LOA document sid. Return type: unicode
Returns: The Status of this HostedNumberOrder. Return type: DependentHostedNumberOrderInstance.Status
Returns: A unique, developer assigned name of this HostedNumberOrder. Return type: unicode
Returns: The number of attempts made to verify ownership of the phone number. Return type: unicode
Returns: List of IDs for ownership verification calls. Return type: list[unicode]
Returns: The digits passed during the ownership verification call. Return type: unicode
Returns: Verification Document Sid. Return type: unicode
Returns: The method used for verifying ownership of the number to be hosted. Return type: DependentHostedNumberOrderInstance.VerificationType
Bases:
twilio.base.list_resource.ListResource
PLEASE NOTE that this class contains preview products that are subject to change. Use them with caution. If you currently do not have developer preview access, please contact<EMAIL_ADDRESS>Retrieve a specific page of DependentHostedNumberOrderInstance records from the API. Request is executed immediately
Parameters: target_url (str) – API-generated URL for the requested results page Returns: Page of DependentHostedNumberOrderInstance Return type: twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderPage
Lists DependentHostedNumberOrderInstance records from the API as a list. Unlike stream(), this operation is eager and will load limit records into memory before returning.
Parameters: - status (DependentHostedNumberOrderInstance.Status) – The Status of this HostedNumberOrder.
- phone_number (unicode) – An E164 formatted phone number.
- incoming_phone_number_sid (unicode) – IncomingPhoneNumber sid.
- friendly_name (unicode) – A human readable description of this resource.
- unique_name (unicode) – A unique, developer assigned name of this HostedNumberOrder.
- limit (int) – Upper limit for the number of records to return. list() guarantees never to return more than limit. Default is no limit
- page_size (int) – Number of records to fetch per request, when not set will use the default value of 50 records. If no page_size is defined but a limit is defined, list() will attempt to read the limit with the most efficient page size, i.e. min(limit, 1000)
Returns: Generator that will yield up to limit results
Return type: list[twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderInstance]
Retrieve a single page of DependentHostedNumberOrderInstance records from the API. Request is executed immediately
Parameters: - status (DependentHostedNumberOrderInstance.Status) – The Status of this HostedNumberOrder.
- phone_number (unicode) – An E164 formatted phone number.
- incoming_phone_number_sid (unicode) – IncomingPhoneNumber sid.
- friendly_name (unicode) – A human readable description of this resource.
- unique_name (unicode) – A unique, developer assigned name of this HostedNumberOrder.
- page_token (str) – PageToken provided by the API
- page_number (int) – Page Number, this value is simply for client state
- page_size (int) – Number of records to return, defaults to 50
Returns: Page of DependentHostedNumberOrderInstance
Return type:
Streams DependentHostedNumberOrderInstance records from the API as a generator stream. This operation lazily loads records as efficiently as possible until the limit is reached. The results are returned as a generator, so this operation is memory efficient.
Parameters: - status (DependentHostedNumberOrderInstance.Status) – The Status of this HostedNumberOrder.
- phone_number (unicode) – An E164 formatted phone number.
- incoming_phone_number_sid (unicode) – IncomingPhoneNumber sid.
- friendly_name (unicode) – A human readable description of this resource.
- unique_name (unicode) – A unique, developer assigned name of this HostedNumberOrder.
- limit (int) – Upper limit for the number of records to return. stream() guarantees to never return more than limit. Default is no limit
- page_size (int) – Number of records to fetch per request, when not set will use the default value of 50 records. If no page_size is defined but a limit is defined, stream() will attempt to read the limit with the most efficient page size, i.e. min(limit, 1000)
Returns: Generator that will yield up to limit results
Return type: list[twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderInstance]
Bases:
twilio.base.page.Page
PLEASE NOTE that this class contains preview products that are subject to change. Use them with caution. If you currently do not have developer preview access, please contact<EMAIL_ADDRESS>Build an instance of DependentHostedNumberOrderInstance
Parameters: payload (dict) – Payload response from the API Returns: twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderInstance Return type: twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderInstance
Module contents¶
This code was generated by / _ _ _| _ _
(_)/(_)(_|/| |(/_ v1.0.0 / /
Bases:
twilio.base.instance_context.InstanceContext
PLEASE NOTE that this class contains preview products that are subject to change. Use them with caution. If you currently do not have developer preview access, please contact<EMAIL_ADDRESS>Access the dependent_hosted_number_orders
Returns: twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderList Return type: twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderList
Fetch the AuthorizationDocumentInstance
Returns: The fetched AuthorizationDocumentInstance Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance
Update the AuthorizationDocumentInstance
Parameters: - hosted_number_order_sids (list[unicode]) – A list of HostedNumberOrder sids.
- address_sid (unicode) – Address sid.
- email (unicode) – Email.
- cc_emails (list[unicode]) – A list of emails.
- status (AuthorizationDocumentInstance.Status) – The Status of this AuthorizationDocument.
- contact_title (unicode) – Title of signee of this Authorization Document.
- contact_phone_number (unicode) – Authorization Document’s signee’s phone number.
Returns: The updated AuthorizationDocumentInstance
Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance
Bases:
twilio.base.instance_resource.InstanceResource
PLEASE NOTE that this class contains preview products that are subject to change. Use them with caution. If you currently do not have developer preview access, please contact<EMAIL_ADDRESS>Bases:
object
Returns: Address sid. Return type: unicode
Returns: A list of emails. Return type: list[unicode]
Returns: The date this AuthorizationDocument was created. Return type: datetime
Returns: The date this AuthorizationDocument was updated. Return type: datetime
Access the dependent_hosted_number_orders
Returns: twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderList Return type: twilio.rest.preview.hosted_numbers.authorization_document.dependent_hosted_number_order.DependentHostedNumberOrderList
Returns: Email. Return type: unicode
Fetch the AuthorizationDocumentInstance
Returns: The fetched AuthorizationDocumentInstance Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance
Returns: The links Return type: unicode
Returns: AuthorizationDocument sid. Return type: unicode
Returns: The Status of this AuthorizationDocument. Return type: AuthorizationDocumentInstance.Status
Update the AuthorizationDocumentInstance
Parameters: - hosted_number_order_sids (list[unicode]) – A list of HostedNumberOrder sids.
- address_sid (unicode) – Address sid.
- email (unicode) – Email.
- cc_emails (list[unicode]) – A list of emails.
- status (AuthorizationDocumentInstance.Status) – The Status of this AuthorizationDocument.
- contact_title (unicode) – Title of signee of this Authorization Document.
- contact_phone_number (unicode) – Authorization Document’s signee’s phone number.
Returns: The updated AuthorizationDocumentInstance
Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance
Returns: The url Return type: unicode
Bases:
twilio.base.list_resource.ListResource
PLEASE NOTE that this class contains preview products that are subject to change. Use them with caution. If you currently do not have developer preview access, please contact<EMAIL_ADDRESS>Create the AuthorizationDocumentInstance
Parameters: - hosted_number_order_sids (list[unicode]) – A list of HostedNumberOrder sids.
- address_sid (unicode) – Address sid.
- email (unicode) – Email.
- contact_title (unicode) – Title of signee of this Authorization Document.
- contact_phone_number (unicode) – Authorization Document’s signee’s phone number.
- cc_emails (list[unicode]) – A list of emails.
Returns: The created AuthorizationDocumentInstance
Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance
Constructs a AuthorizationDocumentContext
Parameters: sid – AuthorizationDocument sid. Returns: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentContext Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentContext
Retrieve a specific page of AuthorizationDocumentInstance records from the API. Request is executed immediately
Parameters: target_url (str) – API-generated URL for the requested results page Returns: Page of AuthorizationDocumentInstance Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentPage
Lists AuthorizationDocumentInstance records from the API as a list. Unlike stream(), this operation is eager and will load limit records into memory before returning.
Parameters: - email (unicode) – Email.
- status (AuthorizationDocumentInstance.Status) – The Status of this AuthorizationDocument.
- limit (int) – Upper limit for the number of records to return. list() guarantees never to return more than limit. Default is no limit
- page_size (int) – Number of records to fetch per request, when not set will use the default value of 50 records. If no page_size is defined but a limit is defined, list() will attempt to read the limit with the most efficient page size, i.e. min(limit, 1000)
Returns: Generator that will yield up to limit results
Return type: list[twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance]
Retrieve a single page of AuthorizationDocumentInstance records from the API. Request is executed immediately
Parameters: - email (unicode) – Email.
- status (AuthorizationDocumentInstance.Status) – The Status of this AuthorizationDocument.
- page_token (str) – PageToken provided by the API
- page_number (int) – Page Number, this value is simply for client state
- page_size (int) – Number of records to return, defaults to 50
Returns: Page of AuthorizationDocumentInstance
Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentPage
Streams AuthorizationDocumentInstance records from the API as a generator stream. This operation lazily loads records as efficiently as possible until the limit is reached. The results are returned as a generator, so this operation is memory efficient.
Parameters: - email (unicode) – Email.
- status (AuthorizationDocumentInstance.Status) – The Status of this AuthorizationDocument.
- limit (int) – Upper limit for the number of records to return. stream() guarantees to never return more than limit. Default is no limit
- page_size (int) – Number of records to fetch per request, when not set will use the default value of 50 records. If no page_size is defined but a limit is defined, stream() will attempt to read the limit with the most efficient page size, i.e. min(limit, 1000)
Returns: Generator that will yield up to limit results
Return type: list[twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance]
Bases:
twilio.base.page.Page
PLEASE NOTE that this class contains preview products that are subject to change. Use them with caution. If you currently do not have developer preview access, please contact<EMAIL_ADDRESS>Build an instance of AuthorizationDocumentInstance
Parameters: payload (dict) – Payload response from the API Returns: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance Return type: twilio.rest.preview.hosted_numbers.authorization_document.AuthorizationDocumentInstance


dd
Convert and copy a file.
More information: home page
- Make a bootable USB drive from an isohybrid file (such like `archlinux-xxx.iso`) and show the progress:
dd if=path/to/file.iso of=/dev/usb_drive status=progress
- Clone a drive to another drive with 4 MiB block, ignore error and show the progress:
dd if=/dev/source_drive of=/dev/dest_drive bs=4M conv=noerror status=progress
- Generate a file of 100 random bytes by using kernel random driver:
dd if=/dev/urandom of=path/to/random_file bs=100 count=1
- Benchmark the write performance of a disk:
dd if=/dev/zero of=path/to/file_1GB bs=1024 count=1000000
- Generate a system backup into an IMG file and show the progress:
dd if=/dev/drive_device of=path/to/file.img status=progress
- Restore a drive from an IMG file and show the progress:
dd if=path/to/file.img of=/dev/drive_device status=progress
- Check the progress of an ongoing dd operation (run this command from another shell):
kill -USR1 $(pgrep ^dd)


--- layout: post title: Truck....? date: 2015-01-13 04:54:32.000000000 -08:00 type: post background: '/img/posts/bg.jpg' parent_id: '0' published: true password: '' status: publish categories: - travel tags: [] meta: _edit_last: '2' _thumbnail_id: '1013' permalink: "/travel/truck/" ---
I am all huddled up in bed and yes, you'll notice the time stamp. I'm banking on tons of sleep and an early morning. Time to get switched over to a more energetic me and stock up on the missing sleep from last weekend.
I'm truck shopping. So far this has entailed looping in my brother and my father in endless nervous phone calls and simultaneous smart phone tow weight capacity calculations. This purchase, unlike my rather affordable tiny house/trailer/camper, feels much bigger. It has all of the responsibility of keeping the house safe and always with me while I take 8 feet by 31 feet and tens of thousands of lbs on the freeway at 65.
I keep trying to talk myself into a kind of trucker like bravado, but the plain truth of it is that I'm a little freaked out. I'm hoping to get the one Randy test drove with me today. It costs almost four times as much as the trailer and I'm hoping it will be worth every gasoline-sucking penny. A whole lotta practice and just plain driving it should really make a big difference. Maybe I'll get those cowboy boots after all.


'The Proton Game' is a console-based data-crunching game for younger and older data scientists. Act as a data-hacker and find Slawomir Pietraszko's credentials to the Proton server. You have to solve four data-based puzzles to find the login and password. There are many ways to solve these puzzles. You may use loops, data filtering, ordering, aggregation or other tools. Only basics knowledge of R is required to play the game, yet the more functions you know, the more approaches you can try. The knowledge of dplyr is not required but may be very helpful. This game is linked with the ,,Pietraszko's Cave” story available at http://biecek.pl/BetaBit/Warsaw. It's a part of Beta and Bit series. You will find more about the Beta and Bit series at http://biecek.pl/BetaBit.
| Version: | 1.0 |
| Depends: | R (≥ 3.0.0) |
| Imports: | digest |
| Published: | 2015-11-30 |
| Author: | Przemysław Biecek [aut, cre], Witold Chodor [trl], Foundation SmarterPoland.pl [cph] |
| Maintainer: | Przemysław Biecek <przemyslaw.biecek at gmail.com> |
| License: | GPL-2 |
| NeedsCompilation: | no |
| CRAN checks: | proton results |
| Reference manual: | proton.pdf |
| Package source: | proton_1.0.tar.gz |
| Windows binaries: | r-devel: proton_1.0.zip, r-release: proton_1.0.zip, r-oldrel: proton_1.0.zip |
| OS X binaries: | r-release: proton_1.0.tgz, r-oldrel: proton_1.0.tgz |
Please use the canonical form https://CRAN.R-project.org/package=proton to link to this page.


I am tooltip 1
I am tooltip 2
I should be a panel
one
two
three
Click Me
Click Me
Click Me
Click Me
Click Me
I should be a panel with a shadow
show modal
I should be a panel with a shadow
I am the content for one
I am the content for two
I am the content for three
I am the content for four
foo
Theme Button
one
two
three


Before Change
with open("HISTORY.rst") as history_file:
history = history_file.read()
requirements = [
"numpy>=1.13.0",
"matplotlib>=2.0.2",
"hyperopt==0.1",
"ipyparallel==6.0.0",
"Keras==2.0.4",
"pandas>=0.18.0",
"scikit-learn==0.18.1",
"scipy>=0.17.0",
"Theano==0.9.0",
"xgboost==0.6a2",
"lightgbm==2.0.2"
]
test_requirements = [
"numpy>=1.13.0",
"matplotlib>=2.0.2",
"hyperopt==0.1",
"ipyparallel==6.0.0",
"Keras==2.0.4",
"pandas>=0.18.0",
"scikit-learn==0.18.1",
"scipy>=0.17.0",
"Theano==0.9.0",
"xgboost==0.6a2",
"lightgbm==2.0.2"
]
setup(
name="mlbox",
version="0.2.2",
description="A powerful Automated Machine Learning python library ",
long_description=readme + "\n\n" + history,
author="Axel ARONIO DE ROMBLAY",
<EMAIL_ADDRESS>url="https://github.com/AxeldeRomblay/mlbox",
packages=[
"mlbox","mlbox.encoding","mlbox.model","mlbox.optimisation","mlbox.prediction",
"mlbox.preprocessing","mlbox.model.supervised","mlbox.model.supervised.classification",
"mlbox.model.supervised.regression","mlbox.preprocessing.drift"
],
package_dir={"mlbox":"mlbox",
"mlbox.encoding":"mlbox/encoding",
"mlbox.model":"mlbox/model",
"mlbox.optimisation":"mlbox/optimisation",
"mlbox.prediction":"mlbox/prediction",
"mlbox.preprocessing":"mlbox/preprocessing",
"mlbox.model.supervised":"mlbox/model/supervised",
"mlbox.model.supervised.classification":"mlbox/model/supervised/classification",
"mlbox.model.supervised.regression":"mlbox/model/supervised/regression",
"mlbox.preprocessing.drift":"mlbox/preprocessing/drift"
},
include_package_data=True,
install_requires=requirements,
zip_safe=False,
keywords="mlbox",
classifiers=[
"Development Status :: 3 - Pre-Alpha",
"Intended Audience :: Developers",
"Natural Language :: English",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 2.6",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
],
test_suite="tests",
tests_require=test_requirements
)
After Change
from pip.req import parse_requirements
install_reqs = parse_requirements("./requirements.txt", session=False)
reqs = [str(ir.req) for ir in install_reqs]
class OverrideInstallCommand(install):
def run(self):


Hotline : +8801308383301 | E-mail<EMAIL_ADDRESS>FIS Network over the last decade has consistently provided its customers with state of the art IT communication solutions. Our clients have been able to rely on our ability to provide stable and consistent connectivity solutions. Using the state of the art backbone and infrastructure, we have ensured that our clients have had minimal worry when it comes to the dependability of their IT Communication. That, in turn, has ensured us an enviable list of blue-chip customers. When Internet and related value-added services are the critical input to business, Corporate Bangladesh has but only one obvious choice FIS Network.
We Provide Home Internet
We Provide Dedicated Bandwidth


Welcome to Doctors,
Doctors is an online reference management system which allows you to easily maintain your references, store the document being referenced, define reference categories and category heirarchies, and centralise resource information.
Take the Online Tutorial - Get Help
Doctors is happily brought to you by:


Available on Github, built by Tyler
A lazy image loader designed to enforce progressive enhancement and valid HTML.
Not a framework, not a library, just a function.
loadMedia( element, onload, scroll )
element: CSS String | NodeList | Element (optional) – loads all images if not set
onload: Function (optional)
scroll: Boolean (optional) – loads image when visible
Declare a lazy image like this:

loadMedia()Run

By default, the function targets every noscript element on the page.
Any attributes the noscript image has (srcset, class, id, alt, etc) are kept.
Other lazy loaders promote invalid HTML by omitting the src attribute, or aren't compatible for users without javascript.
You can specify what images to load by passing a parameter into loadMedia of either

You can hook an onload function for every loaded image.
This may help animate the image by styling it onload.
Run:loadMedia(null, function() { this.classList.add('loaded') })Output:

There's a default function to load images when they're scrolled into view.
This is a general solution, creating your own scroll-based loading functionality may be more performant.
Will be updated to use intersection observers when it becomes standardized. Ex:
loadMedia(null, null, true)


class Solution: def furthestBuilding(self, heights: List[int], bricks: int, ladders: int) -> int: hq = [] s = 0 for i in range(len(heights) - 1): diff = heights[i + 1] - heights[i] if diff <= 0: continue s += diff heapq.heappush(hq, -diff) if s > bricks: if ladders <= 0: return i s += heapq.heappop(hq) ladders -= 1 return len(heights) - 1


Users list
Select a role
Users
status
User detail
User name
:
Email:
Password:
reset
Comment
:
Creation
:
Last Login
:
Is enabled
:
Is locked
:
×
Delete user
Confirm delete for user
:
?
This operation cannot be recovered
Add user
User name
:
Password:
Retype
password
Email:
Comment
:
Role
:
Update user
E mail:
Comment
:
Reset user password
New password:
password:


Code Coverage |
|||||||||
Lines |
Functions and Methods |
Classes and Traits |
|||||||
| Total | |
92.05% |
336 / 365 |
|
91.74% |
211 / 230 |
|
71.43% |
10 / 14 |
| Product | |
100.00% |
72 / 72 |
|
100.00% |
45 / 45 |
|
100.00% |
9 / 9 |
| Country.php | |
100.00% |
15 / 15 |
|
100.00% |
10 / 10 |
|
100.00% |
1 / 1 |
| Product.php | |
99.47% |
189 / 190 |
|
99.20% |
124 / 125 |
|
0.00% |
0 / 1 |
| Stock.php | |
11.11% |
1 / 9 |
|
16.67% |
1 / 6 |
|
0.00% |
0 / 1 |
| StockProducts.php | |
0.00% |
0 / 17 |
|
0.00% |
0 / 11 |
|
0.00% |
0 / 1 |
| User.php | |
95.16% |
59 / 62 |
|
93.94% |
31 / 33 |
|
0.00% |
0 / 1 |


Experiment Guidelines
Welcome to our economics experiment. In this experiment you will be presented with real introductory economics content, fill a quiz and complete a survey.
The duration of this experiment is approximately 60 minutes and participation is paid.
If necessary you may use the calculator app included in your computer
Content
You will be presented with a real university introductory economics chapter and you will receive 30 minutes to review this content.
The Content consists of:
- Textbook Chapter and Explanations
- Graphs
- Optional Practice Questions (Not graded)
This content will provide everything necessary to do well in the quiz and obtain higher payment.
If you have any questions you may raise your hand.
Quiz and Survey
The quiz will start once the 30 minutes to review the content has passed.
The quiz will have 7 questions, which will be based on the content you learn.
You will have 10 minutes to complete the quiz, but you can submit the quiz before the 10 minutes have passed.
The score you obtain in the quiz will determine your final earnings.
After finalising the quiz you will complete an anonymous survey.
Timers
The Content and Quiz sections of the experiment will show countdown timers displaying the time you have left.
If the time ends before you submit your quiz it will be automatically submitted for you with your current answers.
Payment
The payment amount depends on a show up fee plus a variable amount that depends on your quiz grade.
- Show Up Fee: $5 dollars (AUD)
- Fee for finishing the experiment: $15 dollars (AUD)
- Variable Amount based on quiz grade: Up to a maximum of $5 dollars (AUD)
If you want to check the instructions at any time just press the icon on the top right


| GEOS SDK TechDocs
|
| 6.7 Print Spooler and Printing
| 6.9 Communications
The GEOS kernel allows applications and libraries to set up different types of timers that can be used for various purposes. When a timer ticks, it calls a routine specified by the geode that initiated it. The different types of timers that can be set up include
-
one-shot
- A timer that executes the specified routine once.
-
continual
- A timer that calls the routine once every given time period.
-
message
- Either a one-shot or continual timer that sends a specified message rather than call a specified routine.
-
millisecond
- A one-shot timer with millisecond accuracy.
-
sleep
- A timer that puts the calling thread to sleep for a given length of time.
-
semaphore
- A timer that provides a "time-out" for a given semaphore queue (if the thread does not gain the semaphore in the given length of time, it is removed from the queue).
| GEOS SDK TechDocs
|
| 6.7 Print Spooler and Printing
| 6.9 Communications


My Favorite Places
These are my favorite places in the
whole wide world
Countries
USA
Italy
Canada
Cities
Portland
Paris
New York City
Montreal
Places
MOMA
My Bed
Epicodus


|
Table 1
|
Experimental Matrix for the First DoE Cycle |
|
Table 2
|
Rietveld Refinement Compositions Obtained for the First Batc ... |
|
Table 3
|
Magnetic Properties of the First Batch of Samples |
|
Table 4
|
Experimental Matrix for the Second DoE Cycle |
|
Table 5
|
Rietveld Refinement Compositions Obtained for the Second Bat ... |
|
Table 6
|
Magnetic Properties of the Second Batch of Samples |
|
Table 7
|
Comparison of the Results Obtained in This Work with the Lit ... |
|
Table 8
|
Experimental Matrix Definition |


<#location> weather
Latitude <#latitude> Longitude <#longitude> Elevation <#altitude>
Outside Temperature |
Wind Chill |
Heat Index |
small |
small |
small |
Dew Point |
Outside Humidity |
Rain today |
small |
small |
small |
Inside Temperature |
Inside Humidity |
Cloud Base |
small |
small |
Barometer |
Wind |
analog |
wind |
| Wind Direction | Wind Speed | Wind Distribution |
| :now::gauges::today::yesterday::this month::this year::records::monthly records::trends:<#forum><#webcam> |
Page updated <#update>
powered by
Cumulus v<#version> (<#build>)


Our mission statement
Meet the members
Check out photos from our show!
I mustache you a question...
Placeholder here...
[DESCRIPTION HERE] PC: Jisung Kim '19
[DESCRIPTION HERE]PC: Jisung Kim '19
A yellow workspace with some scissors, pencils, and other objects.


Bristol family holds biggest family Christmas get together in country
'It gets quite hard to keep track of everyone, and know who is who.'
by Sarah Lumley, Alexander BrockA Bristol family spanning five generations has held the biggest family Christmas get together in the country.
More than 100 members of the Cole family gather for the party every year to enjoy a range of festive activities with their many loved ones.
The enormous party, held this year on Sunday (November 17) in Shirehampton , saw a gathering of 120 family members from 82 to just six months old.
Brenda Cotterell, 58, said the bash was arranged by her eldest sister Shirley.
The two sisters said the get together was carrying on the tradition that their late father, Albert Cole, began 15 years ago.
View gallery
Brenda said: “My dad loved Christmas so, so much. You could feel his excitement building every year.
“My father was always the big personality. His great-grandchildren still call him Grandad Lucky Dip, because he would dress up as Father Christmas and would always have a lucky dip to play with them.”
"The Lucky Dip is something that the family have made sure to keep going in Albert's honour, as well as a raffle, with gifts and sweets donated by various family members.
Brenda added: “My father started this gathering every year after losing his son Andrew, my youngest brother, aged 22, and then losing my mum 28 years ago.
“As the family has grown, it gets quite hard to keep track of everyone, and know who is who.
“But I know how much my father would love to see us all together today."
Brenda's sister Shirley, who helped to organise this year's event, said: “It's brilliant to get everyone together.
“It can be a nightmare trying to visit everyone during the year."
Brenda's aunt Lin, who is married to Albert's brother Richard – one of four of Albert's surviving siblings – joked that not one person at the party knew everybody else there.
The great-grandmother of three said: “It's just life, for us. My family have grown up not knowing any differently – this is just normal.
“My daughter was talking to someone recently who only had two cousins, and she just couldn't believe it. She couldn't fathom them not having more cousins.
“It's just nice to come to an event like this and not have to worry about keeping an eye on the little ones, or worry about them annoying others, because you know somebody in this hall will be watching them.”
Brenda's uncle John, who has 32 great-grandkids of his own, added: "It's really good to all get together – but we won't get Christmas presents for everybody here."
For the latest news in and around Bristol, visit and bookmark Bristol Live's homepage .


HPL Related Links
The list of links below contains some relevant material to this
work. This list is provided for illustrative purposes, and should be
regarded as an initial starting point for the interested reader. This
list is by all means not meant to be exhaustive.
MPI is a library specification for message-passing, proposed as a
standard by a broadly based committee of vendors, implementors, and
users. Machine-specific (optimized) as well as freely available MPI
libraries are available for a large variety of systems. Browse the
Message Passing Interface (MPI)
standard web page for more information.
The BLAS are high quality
"building block" routines for performing basic vector and matrix
operations. A lot of "BLAS-related" information can be found at this
site. In particular, a reference implementation is available. This
reference implementation is not optimized for any
system, and it is therefore not recommended to use it
for benchmarking purposes.
However, machine-specific
optimized BLAS libraries are available for a variety of computer
systems. For further details, please contact your local vendor
representative. Alternatively, one may also consider using automatic
code generators such as ATLAS.
This tool automatically generates a complete and optimized BLAS
library for a large variety of modern systems.
VSIPL is an API defined by
an open standard comprised of embedded signal and image processing
hardware and software vendors, academia, users, and government labs. A
lot of "VSIPL-related" information can be found at this site. In
particular, a reference implementation is available. Machine-specific
optimized VSIPL libraries are available for a variety of computer
systems. For further details, please contact your local vendor
representative.
TOP 500 List
The TOP 500
is an ordered list of the 500 most powerful computer systems worldwide.
Computers are ranked in this list by their performance on the
LINPACK Benchmark.
Parallel Dense Linear Algebra Software Libraries
Browse the Netlib software repository
or the National HPCC Software Exchange
to find a large collection of freely available linear algebra libraries.
[Home]
[Copyright and Licensing Terms]
[Algorithm]
[Scalability]
[Performance Results]
[Documentation]
[Software]
[FAQs]
[Tuning]
[Errata-Bugs]
[References]
[Related Links]


Padding is the space between the content and the border. In this example, we can see that the content has padding of 10px on the left and right sides and have a padding of 15px for both top and bottom sides. The padding either grows or shrinks the elements size.
Margin is the space outside the border of an element. Margin defines the space from this border to the next outer element. Since the margin is the space outside the border, the margin can be used to “push away” other elements. In the example below, each element has a margin of 10 pixels. Margins are used to create whitespace between elements which give them room to breathe.
The border is the space between the padding and the margin. By default, the border does not have any width but can be set one using the CSS border property. The border style can be changed to many styles including the border width just like in the image below.
So the difference between margin and padding is that while padding deals with the inner space (the space between an element content and the border), margin deals with the outer space (the space outside of the element) to the next outer element. Thus, padding and margin have crucial differences that allows them to play key roles in styling a web page.


Tread Softly: Little Golden Retriever Tries to Sneak Up on Big Doggo
Every canine is a natural-born hunter, but even the most talented dogs need a lot of training to become a dangerous predator. It is important to be fast, strong, and smart – and, of course, to be secretive and cunning.
This little Golden Retriever is learning how to be a ninja, so it tries to sneak up on a big doggo without being detected. And the attempt was almost successful – but the little guy totally forgot that the big retriever is watching its every step.


COMING SOON!
Yoga at your fingertips
Whether you are a newbie or seasoned veteran, Go-Ga offers classes tailored to suit your needs. Offering classes specializing in strength, flexibility, and meditation in a variety of styles, we can help you reach your goals!
Begin a yoga practice
Everyone starts at the start. Learn the yoga basics to begin building a strong foundation.
Fit a class into your busy schedule
With classes starting at 15 minutes, you can create a yoga schedule tailored to your life.
Mobile and Web accessible classes
Sream classes from your computer to any streaming device, or from our mobile app. We go where you go!
Here is our team
Sonia Allen
Yoga Instructor and CEO
Sonia is a 200 hr RYT with the Yoga Alliance, working on her 500 hour certification. She specializes in Sivinanda, Ashtanga, Vinyasa Flow, and meditation instruction.
Want to work with us?
Send us a message if you'd like to be a part of our online yoga community!


Variable
| Static Public Summary | ||
| public |
|
|
| public |
|
|
| public |
|
|
| public |
|
|
| public |
FREQUENCY_BANDS: {"delta": *, "theta": *, "alpha": *, "beta": *, "gamma": *} |
|
| public |
|
|
| public |
|
|
| public |
|
|
| public |
|
|
| public |
|
|
| public |
|
Static Public
public CHARACTERISTIC: string source
import {CHARACTERISTIC} from 'eeg-pipes/src/constants/index.js'
public FREQUENCY_BANDS: {"delta": *, "theta": *, "alpha": *, "beta": *, "gamma": *} source
import {FREQUENCY_BANDS} from 'eeg-pipes/src/constants/index.js'