| - PattoSoft - | - Home | Software | Articles | About | Search | Help - |
Jump To
Section... | More
Than Just Megahertz | Pipelining & Instruction-Level Parallelism Deeper Pipelines - Superpipelining Multiple Issue - Superscalar Explicit Parallelism - VLIW & EPIC Instruction Dependencies & Latencies Branches & Branch Prediction Eliminating Branches with Predication Scheduling, Register Renaming & OOO The Brainiac Debate What About x86? Threads - SMT, Hyper-Threading & Multi-core Data Parallelism - SIMD Vector Instructions Caches & The Memory Hierarchy Cache Conflicts & Associativity Memory Bandwidth vs Latency Acknowledgements More Detailed Information? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
WARNING: This article is meant to be informal and fun! Okay, so you're a CS graduate and you did a hardware/assembly course as part of your degree, but perhaps that was a few years ago now and you haven't really kept up with the details of processor designs since then. In particular, you might not be aware of some key topics that developed rapidly in the past decade...
Fear not! This article will get you up to speed fast. In no time you'll be discussing the finer points of superscalar-vs-VLIW, the brainiac debate and its relationship to IA64 and Itanium. But be prepared – this article is brief and to-the-point. It pulls no punches and the pace is pretty fierce (really). Let's get into it... More Than Just MegahertzThe first issue that must be cleared up is the difference between clock speed and a processor's performance. They are not the same thing. Look at the results for processors of the recent past...
A 200 MHz MIPS R10000, a 300 MHz UltraSPARC and a 400 MHz Alpha 21164 are all about the same speed at running most programs, yet they differ by a factor of two in clock speed. A 300 MHz Pentium-II is also about the same speed for many things, yet it's about half that speed for floating-point code such as scientific number crunching. A PowerPC G3 at that same 300 MHz is somewhat faster than the others for integer code, but still far slower than the top 3 for floating-point. At the other extreme, an IBM POWER2 processor at just 135 MHz matches the 400 MHz Alpha 21164 in floating-point speed, yet it's only half as fast for normal integer programs. How can this be? Obviously, there's more to it than just clock speed – it's all about how much work gets done in each clock cycle. Which leads to... Pipelining & Instruction-Level ParallelismInstructions are executed one after the other inside the processor, right? Well, that makes it easy to understand, but that's not really what happens. In fact, that hasn't happened since the middle of the 1980's. Instead, several instructions are all partially executing at the same time. Consider how an instruction is executed – first it is fetched, then decoded, then executed by the appropriate functional unit, and finally the result is written into place. With this scheme, a simple processor might take 4 cycles per instruction (CPI = 4)...
Modern processors overlap these stages in a pipeline, like an assembly line. While one instruction is executing, the next instruction is being decoded, and the one after that is being fetched...
Now the processor is completing 1 instruction every cycle (CPI = 1). This is a four-fold speedup without changing the clock speed at all. Not bad, huh? From the hardware point of view, each pipeline stage consists of some combinatorial logic and possibly access to a register set and/or some form of high speed cache memory. The pipeline stages are separated by latches. A common clock signal synchronizes the latches between each stage, so that all the latches capture the results produced by the pipeline stages at the same time. In effect, the clock "pumps" instructions down the pipeline. At the beginning of each clock cycle, the data and control information for a partially processed instruction is held in a pipeline latch, and this information forms the inputs to the logic circuits of the next pipeline stage. During the clock cycle, the signals propagate through the combinatorial logic of the stage, producing an output just in time to be captured by the next pipeline latch at the end of the clock cycle...
Since the result from each instruction is available after the execute stage has completed, the next instruction ought to be able to use that value immediately, rather than waiting for that result to be committed to its destination register in the writeback stage. To allow this, forwarding lines called bypasses are added, going backwards along the pipeline...
Although the pipeline stages look simple, it is important to remember that the execute stage in particular is really made up of several different groups of logic (several sets of gates), making up different functional units for each type of operation that the processor must be able to perform...
The early RISC processors, such as IBM's 801 research prototype, the MIPS R2000 (based on the Stanford MIPS machine) and the original SPARC (derived from the Berkeley RISC project), all implemented a simple 5 stage pipeline not unlike the one shown above (the extra stage is for memory access, placed after execute). At the same time, the mainstream 80386, 68030 and VAX processors worked sequentially using microcode (it's easier to pipeline a RISC because the instructions are all simple register-to-register operations, unlike x86, 68k or VAX). As a result, a SPARC running at 20 MHz was way faster than a 386 running at 33 MHz. Every processor since then has been pipelined, at least to some extent. A good summary of the original RISC research projects can be found in this 1985 CACM research paper by David Patterson. Deeper Pipelines - SuperpipeliningSince the clock speed is limited by (among other things) the length of the longest stage in the pipeline, the logic gates that make up each stage can be subdivided, especially the longer ones, converting the pipeline into a deeper super-pipeline with a larger number of shorter stages. Then the whole processor can be run at a higher clock speed! Of course, each instruction will now take more cycles to complete (latency), but the processor will still be completing 1 instruction per cycle (throughput), and there will be more cycles per second, so the processor will complete more instructions per second (actual performance)...
The Alpha architects in particular liked this idea, which is why the early Alpha's had very deep pipelines and ran at such very high clock speeds for their era. The MIPS R4000 series was also superpipelined (in fact the R4000 was only superpipelined and not superscalar, see below). Today, most processors strive to keep the number of levels of logic down to just a handful for each pipeline stage (about 10-20 levels), and most have quite deep pipelines (4-7 in PowerPC G3/G4, 5-7 in MIPS R10000, 7-9 in Alpha 21164, 7-12 in PowerPC G4e, 8-10 in Itanium-II, 9 in UltraSPARC, 10-15 in Athlon, 12+ in Pentium-Pro/II/III, 14 in UltraSPARC-III, 16-25 in PowerPC G5, 20+ in Pentium-4). The x86 processors generally have deeper pipelines than the RISC's because they need to do extra work to decode the x86 instructions (more on this later). Multiple Issue - SuperscalarSince the execute stage of the pipeline is really a bunch of different functional units, each doing its own task, it seems tempting to try to execute multiple instructions in parallel, each in its own functional unit. To do this, the fetch and decode/dispatch stages must be enhanced so that they can decode multiple instructions in parallel and send them out to the "execution resources"...
Of course, now that there are independent pipelines for each functional unit, they can even be different lengths. This allows the simpler instructions to complete more quickly, reducing latency (which we'll get to soon). There are also a bunch of bypasses within and between the various pipelines, but these have been left out for simplicity. In the above example, the processor could potentially execute 3 different instructions per cycle – for example one integer, one floating-point and one memory operation. Even more functional units could be added, so that the processor might be able to execute two integer instructions per cycle, or two floating-point instructions, or whatever the target applications could best use. On a superscalar processor, the instruction flow looks something like...
This is great! There are now 3 instructions completing every cycle (CPI = 0.33, or IPC = 3). The number of instructions able to be issued or completed per cycle is called a processor's width. Note that the issue-width is less than the number of functional units – this is typical. There must be more functional units because different code sequences have different mixes of instructions. The idea is to execute 3 instructions per cycle, but those instructions are not always going to be 1 integer, 1 floating-point and 1 memory operation, so more than 3 functional units are required. The IBM POWER1 processor, the predecessor of PowerPC, was the first mainstream superscalar processor. Most of the RISC's went superscalar soon after (SuperSPARC, Alpha 21064). Intel even managed to build a superscalar x86 – the original Pentium – however the complex x86 instruction set was a real problem for them (more on this later). Of course, there's nothing stopping a processor from having both a deep pipeline and multiple instruction issue, so it can be both superpipelined and superscalar at the same time...
Today, virtually every processor is a superpipelined-superscalar, so they're just called superscalar for short. Strictly speaking, superpipelining is just pipelining with a deeper pipe anyway. The issue-widths of current processors range from 2-issue (MIPS R5000) to 3-issue (PowerPC G3/G4, Pentium-Pro/II/III/M, Athlon, Pentium-4 (well, sort-of)) or 4-issue (UltraSPARC, MIPS R10000, Alpha 21164 & 21264, PowerPC G4e) or 5-issue (PowerPC G5), or even 6-issue (Itanium-I/II, but it's a VLIW – see below). The exact number and type of functional units in each processor depends on its target market. Some processors have more floating-point execution resources (MIPS R8000, IBM's POWER line, Athlon1), others are more integer-biased (Pentium-Pro/II/III/M, PowerPC G3), some devote much of their resources towards SIMD vector instructions (PowerPC G4 & G4e), and many take the middle ground (UltraSPARC, MIPS R10000, Alpha 21164 & 21264, Pentium-4, Itanium-I/II, PowerPC G5).
In cases where backward compatibility is not an issue, it is possible for the instruction set itself to be designed to explicitly group instructions to be executed in parallel. This approach eliminates the need for complex dependency checking logic in the dispatch stage, which should make the processor easier to design (and easier to ramp up the clock speed over time, at least in theory). In this style of processor, the "instructions" are really groups of little sub-instructions, and thus the instructions themselves are very long (often 128 bits or more), hence the name VLIW – very long instruction word. Each instruction contains information for multiple parallel operations. A VLIW processor's instruction flow is much like a superscalar, except that the decode/dispatch stage is much simpler and only occurs for each group of sub-instructions...
Other than the simplification of the dispatch logic, VLIW processors are much like superscalar processors. This is especially so from a compiler's point of view (more on this later). It is worth noting, however, that most VLIW designs are not interlocked. This means they do not check for dependencies between instructions, and often have no way of stalling instructions other than to stall the whole processor on a cache miss. As a result, the compiler needs to insert the appropriate number of cycles between dependent instructions, even if there are no instructions to fill the gap, by using nops (no-operations) if necessary. This complicates the compiler somewhat, because it is doing something that a superscalar processor normally does at runtime, however the extra code in the compiler is minimal and it saves precious resources on the processor chip. No VLIW designs have yet been commercially successful, however Intel's IA64 architecture, which is now in production in the form of the Itanium-I/II processors, was intended to be the replacement for x86 (and may still end up that way, although this is looking increasingly unlikely). Intel chose to call IA64 an "EPIC" design, for "explicitly parallel instruction computing", but it's basically just a VLIW with clever grouping (to allow long-term compatibility) and predication (see below). Many graphics processors (often called GPU's) can in some ways be considered VLIW-like (although obviously they only provide single-purpose instruction sets), and there's also Transmeta (see the x86 section, coming up soon). Instruction Dependencies & LatenciesHow far can pipelining and multiple issue be taken? If a 5 stage pipeline is 5 times faster, why not build a 20 stage superpipeline? If 4-issue superscalar is good, why not go for 8-issue? For that matter, why not build a processor with a 50 stage pipeline which issues 20 instructions per cycle? Well, consider the following two instructions... a = b * c; d = a + 1; The second instruction depends on the first – the processor can't execute the second instruction until after the first has completed calculating its result. This is a serious problem, because instructions that depend on each other cannot be executed in parallel. Thus, multiple issue is impossible in this case. If the first instruction was a simple integer addition then this might still be okay in a pipelined single issue processor, because integer addition is quick and the result of the first instruction would be available just in time to feed it back into the next instruction (using bypasses). However in the case of a multiply, which will take several cycles to complete, there is no way the result of the first instruction will be available when the second instruction reaches the execute stage just one cycle later. So, the processor will need to stall the execution of the second instruction until its data is available, inserting a bubble into the pipeline where no work gets done. The number of cycles between when an instruction reaches the execute stage and when its result is available for use by other instructions is called the instruction's latency2. The deeper the pipeline, the more stages and thus the longer the latency. So a very deep pipeline is not much more effective than a short one, because a deep one just gets filled up with bubbles thanks to all those nasty instructions depending on each other.
Latencies for memory loads are particularly troublesome, in part because they tend to occur early within code sequences, which makes it difficult to fill their delays with useful instructions, and equally importantly because they are somewhat unpredictable – the load latency varies a lot depending on whether the access is a cache hit or not (we'll get to caches later). Branches & Branch PredictionAnother key problem for pipelining is branches. Consider the following code sequence... if (a > 5) b = c; else b = d; which compiles into something like... cmp a, 5 ; a > 5 ? ble L1 mov c, b ; b = c br L2 L1: mov d, b ; b = d L2: ... Now consider a pipelined processor executing this code sequence. By the time the conditional branch at line 2 reaches the execute stage in the pipeline, the processor must have already fetched and decoded the next couple of instructions. But which instructions? Should it fetch and decode the if branch (lines 3 & 4) or the else branch (line 5)? It won't really know until the conditional branch gets to the execute stage, but in a deeply pipelined processor that might be several cycles away. And it can't afford to just wait – the processor encounters a branch every six instructions on average, and if it was to wait several cycles at every branch then most of the performance gained by using pipelining in the first place would be lost. So the processor must make a guess. The processor will then fetch down the path it guessed and speculatively begin executing those instructions. Of course, it won't be able to actually commit (writeback) those instructions until the outcome of the branch is known. Worse, if the guess is wrong the instructions will have to be cancelled, and those cycles will have been wasted. But if the guess is correct the processor will be able to continue on at full speed. The key question is how the processor should make the guess. Two alternatives spring to mind. First, the compiler might be able to mark the branch to tell the processor which way to go. This is called static branch prediction. It would be ideal if there was a bit in the instruction format in which to encode the prediction, but for older architectures this is not an option, so a convention can be used instead (such as backward branches are predicted to be taken while forward branches are predicted not-taken). More importantly, however, this approach requires the compiler to be quite smart in order for it to make the correct guess, which is easy for loops but might be difficult for other branches. The other alternative is to have the processor make the guess at runtime. Normally, this is done by using an on-chip branch prediction table containing the addresses of recent branches and a bit indicating whether each branch was taken or not last time. In reality, most processors actually use two bits, so that a single not-taken occurrence doesn't reverse a generally taken prediction (important for loop back edges). Of course, this dynamic branch prediction table takes up valuable space on the processor chip, but branch prediction is so important that it's well worth it. Unfortunately, even the best branch prediction techniques are sometimes wrong, and with a deep pipeline many instructions might need to be cancelled. This is called the mispredict penalty. The Pentium-Pro/II/III is a good example – it has a 12+ stage pipeline and thus a mispredict penalty of 10-15 cycles. Even with a clever dynamic branch predictor that correctly predicts an impressive 90% of the time, this high mispredict penalty means about 30% of the Pentium-Pro/II/III's performance is lost due to mispredicts. Put another way, one third of the time the Pentium-Pro/II/III is not doing useful work but instead is saying "oops, wrong way". Eliminating Branches with PredicationConditional branches are so problematic that it would be nice to eliminate them altogether. Clearly, if statements cannot be eliminated from programming languages, so how can the resulting branches possibly be eliminated? The answer lies in the way some branches are used. Consider the above example once again. Of the five instructions, two are branches, and one of those is an unconditional branch. If it was possible to somehow tag the mov instructions to tell them to execute only under some conditions, the code could be simplified... cmp a, 5 ; a > 5 ? mov c, b ; b = c cmovle d, b ; if le, then b = d Here, a new instruction has been introduced called cmovle, for "conditional move if less than or equal". This instruction works by executing as normal, but only commits itself if its condition is true. This is called a predicated instruction because its execution is controlled by a predicate (a true/false test). Given this new predicated move instruction, two instructions have been eliminated from the code, and both were costly branches. In addition, by being clever and always doing the first mov then overwriting it if necessary, the parallelism of the code has also been increased – lines 1 and 2 can now be executed in parallel, resulting in a 50% speedup (2 cycles rather than 3). Most importantly, though, the possibility of getting the branch prediction wrong and suffering a large mispredict penalty has been eliminated. Of course, if the blocks of code in the if and else cases were longer, then using predication would mean executing more instructions than using a branch, because the processor is effectively executing both paths through the code. Whether it's worth executing a few more instructions to avoid a branch is a tricky decision – for very small or very large blocks the decision is simple, but for medium-sized blocks there are complex trade-offs which the optimizer must consider. The Alpha architecture has had a conditional move instruction since the very beginning. MIPS, SPARC and x86 added it later. With IA64, however, Intel has gone all-out and made almost every instruction predicated in the hope of dramatically reducing branching problems in inner loops, especially ones where the branches are unpredictable (such as compilers and OS kernels). It will be interesting to see whether this works out well or not for IA64. Interestingly, the ARM architecture used in many portable handheld devices was the first architecture with a fully predicated instruction set. This is even more intriguing given that the ARM processors only have short pipelines and thus relatively small mispredict penalties. Scheduling, Register Renaming & OOOIf branches and long latency instructions are going to cause bubbles in the pipeline(s), then perhaps those empty cycles can be used to do other work. To achieve this, the instructions in the program must be reordered so that while one instruction is waiting, other instructions can execute. For example, it might be possible to find a couple of other instructions from further down in the program and put them between the two instructions in the earlier multiply example. There are two ways to do this. One approach is to do the reordering in hardware at runtime. Doing dynamic instruction scheduling (reordering) in the processor means the dispatch logic must be enhanced to look at groups of instructions and dispatch them out of order as best it can to use the processor's functional units. Not surprisingly, this is called out-of-order execution, or just OOO for short (sometimes written OoO or OOE). If the processor is going to execute instructions out of order, it will need to keep in mind the dependencies between those instructions. This can be made easier by not dealing with the raw architecturally-defined registers, but instead using a set of renamed registers. For example, a store of a register into memory, followed by a load of some other piece of memory into the same register, represent different values and need not go into the same physical register. Furthermore, if these different instructions are mapped to different physical registers they can be executed in parallel, which is the whole point of OOO execution. So, the processor must keep a mapping of the instructions in flight at any moment and the physical registers they use. This process is called register renaming. As an added bonus, it becomes possible to work with a potentially larger set of real registers in an attempt to extract even more parallelism out of the code. All of this dependency analysis, register renaming and OOO execution adds a lot of complex logic to the processor, making it harder to design and potentially harder to ramp up the clock speed over time. On the other hand, it offers the advantage that software need not be recompiled to get at least some of the benefits of the new processor's design (though typically not all). Another approach to the whole problem is to have the compiler optimize the code by rearranging the instructions (called static, or compile-time, instruction scheduling). The rearranged instruction stream can then be fed to a processor with simpler in-order multiple-issue logic, relying on the compiler to "spoon feed" the processor with the best instruction stream. Avoiding the need for complex OOO logic should make the processor quite a lot easier to design, and should potentially make it easier to ramp up the clock speed over time. The compiler approach also has some other advantages over OOO hardware – it can see further down the program than the hardware, and it can speculate down multiple paths rather than just one (a big issue if branches are unpredictable). On the other hand, a compiler can't be expected to be psychic, so it can't necessarily get everything perfect all the time. Without OOO hardware, the pipeline will stall when the compiler fails to predict something like a cache miss. Most of the early superscalars were in-order designs (SuperSPARC, hyperSPARC, UltraSPARC, Alpha 21064 & 21164). Examples of OOO designs include the MIPS R10000, Alpha 21264 and to some extent the entire POWER/PowerPC line (with their reservation stations). UltraSPARC-III is the most notable recent design which has stayed in-order and not added any OOO execution hardware. The Brainiac DebateWhether compilers can do the task of instruction scheduling well enough or not is a hot question at the moment in the hardware industry. This is called the brainiac vs speed-demon debate. This simple (and fun) classification of design styles first appeared in a 1993 Microprocessor Report editorial by Linley Gwennap, and was made widely known by Dileep Bhandarkar's Alpha Implementations & Architecture book. Brainiac designs are at the smart-machine end of the spectrum, like IBM's POWER2 and the MIPS R10000, whereas speed-demon designs like the Alpha 21164 and UltraSPARC rely on a smart compiler. Clearly, OOO hardware should make it possible for more instruction-level parallelism to be extracted, because things will be known at runtime that cannot be predicted in advance (cache misses, for example). On the other hand, an in-order design is potentially able to run at faster clock speeds at any given point in time, due to reduced design complexity. Exactly which is the more important factor is currently open to hot debate. At present, it seems that both the benefits and the costs of OOO execution have been somewhat overstated. In terms of cost, appropriate pipelining of the dispatch and register renaming logic has allowed OOO processors to achieve clock speeds competitive with simpler designs – the aggressively OOO Alpha 21264, moderately OOO PowerPC G4e, and simpler in-order UltraSPARC-III are all available at comparable clock speeds, for example. This is a testament to some outstanding engineering by processor architects. Unfortunately, however, the effectiveness of OOO execution in dynamically extracting additional instruction-level parallelism has been disappointing, with only a relatively small improvement being seen3. OOO execution has also been unable to deliver the degree of schedule-insensitivity originally hoped for, with recompilation still producing large speedups even on aggressive OOO processors such as the MIPS R10000 and Alpha 21264.
DEC, for example, went primarily speed-demon with the first two generations of Alpha, then changed to brainiac for the third generation. MIPS did similarly. Sun, on the other hand, went brainiac with their first superscalar then switched to speed-demon for more recent designs. The PowerPC camp has also gradually moved away from brainiac designs over the years, although the reservation stations in all PowerPC designs do offer a degree of OOO execution between different functional units even if the instructions within each functional unit's queue are executed strictly in order. Intel is sort-of going both ways at once – x86 processors have no choice but to be at least somewhat brainiac due to limitations of the x86 architecture (though the Pentium-4 is about as speed-demon as possible for a decoupled x86 microarchitecture), but with IA64, Intel is betting solidly on the smart-compiler approach, with a simple but very wide design relying totally on static scheduling (at least in the first two generations). No matter which route is taken, the key problem is still the same – normal programs just don't have a lot of fine-grained parallelism in them. A 4-issue superscalar processor requires four independent instructions to be available, with all their dependencies and latencies met, at every cycle. In reality this is virtually never possible, especially with load latencies of two or three cycles (and increasing with every processor generation). Currently, real-world instruction-level parallelism for mainstream applications is limited to about 2 instructions per cycle at best. Certain types of applications do exhibit more parallelism, such as scientific code, but these are generally not representative of mainstream applications. There are also some types of code, such as pointer chasing, where even sustaining 1 instruction per cycle is extremely difficult. For those programs, the key problem is the memory system (which we'll get to later). What About x86?So where does x86 fit into all this, and how have Intel and AMD been able to remain competitive through all of these developments in spite of an architecture that's now more than 20 years old? While the Pentium, a superscalar x86, was an amazing piece of engineering, it was clear that the big problem was the complex and messy x86 instruction set. Complex addressing modes and a minimal number of registers meant that few instructions could be executed in parallel due to potential dependencies. For the x86 camp to compete with the RISC platforms, they needed to find a way to "get around" the x86 instruction set. The solution, invented independently (at about the same time) by engineers at both NexGen and Intel, was to dynamically decode the x86 instructions into simple, RISC-like micro-instructions, which can then be executed by a fast, RISC-style register-renaming OOO superscalar core. For these "decoupled" superscalar x86 processors, register renaming is absolutely critical due to the meager 8 registers of the x86 architecture. This differs strongly from the RISC architectures, where providing more registers via renaming only has a minor effect. Nonetheless, with clever register renaming, the full bag of RISC tricks become available to the x86 world, with the two exceptions of advanced static instruction scheduling (because the micro-instructions are hidden behind the x86 layer and thus are less visible to compilers) and the use of a large register set to avoid memory accesses (because x86 only has 8 architecturally-visible registers). The basic scheme works something like this... |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
One of the most interesting members of this RISC-style x86 group is the Transmeta Crusoe processor, which translates x86 instructions into an internal VLIW form, rather than internal superscalar, and uses software to do the translation at runtime, much like a Java virtual machine. This approach allows the processor itself to be a simple VLIW, without the complex x86 decoding and register renaming hardware of decoupled x86 designs, and without any superscalar dispatch or OOO logic either. The software-based x86 translation does reduce the system's performance compared to hardware translation (which occurs as additional pipeline stages and thus is almost free in performance terms), but the result is a very lean chip which runs fast and cool and uses very little power. A 600 MHz Crusoe processor could match a then-current 500 MHz Pentium-III running in its low-power mode (300 MHz clock speed) while using only a fraction of the power and generating only a fraction of the heat. This made it ideal for laptops and handheld computers, where battery life is crucial. Today, of course, processor variants designed specifically for low power use, such as the Pentium-M (aka: Centrino), have made the Transmeta-style software-based approach unnecessary. Threads - SMT, Hyper-Threading & Multi-coreAs already mentioned, the approach of exploiting instruction-level parallelism through superscalar execution is seriously weakened by the fact that most normal programs just don't have a lot of fine-grained parallelism in them. Because of this, even the most aggressively brainiac OOO superscalar processor, coupled with a smart and aggressive compiler to spoon feed it, will still almost never exceed an average of about 2 instructions per cycle when running most real-world software, due to a combination of load latencies, cache misses, branching and dependencies between instructions. Issuing many instructions in the same cycle only ever happens for short bursts of a few cycles at most, separated by many cycles of executing low-ILP code, so peak performance is not even close to being achieved. If additional independent instructions aren't available within the program being executed, there is another potential source of independent instructions – other running programs (or other threads within the same program). Simultaneous multithreading (SMT) is a processor design technique which exploits exactly this type of thread-level parallelism. Once again, the idea is to fill those empty bubbles in the pipelines with useful instructions, but this time rather than using instructions from further down in the same program (which are hard to come by), the instructions come from multiple threads running at the same time, all on the one processor core. So, an SMT processor appears to the rest of the system as if it were multiple independent processors, just like a true multiprocessor system. Of course, a true multiprocessor system also executes multiple threads simultaneously – but only one in each processor. This is also true for multi-core processors (eg: the POWER4 and upcoming UltraSPARC-IV), which place two or more processor cores onto a single chip, but are otherwise no different from traditional multiprocessor systems. In contrast, an SMT processor uses just one physical processor core to present two or more logical processors to the system. This makes SMT much more efficient than a multi-core processor in terms of chip space, fabrication cost, power usage and heat dissipation. And of course there's nothing preventing a multi-core implementation where each core is an SMT design. From a hardware point of view, implementing SMT requires duplicating all of the parts of the processor which store the "execution state" of each thread – things like the program counter, the architecturally-visible registers (but not the rename registers), the memory mappings held in the TLB, and so on. Luckily, these parts only constitute a tiny fraction of the overall processor's hardware. The really large and complex parts, such as the decoders and dispatch logic, the functional units, and the caches, are all shared between the threads. Of course, the processor must also keep track of which instructions and which rename registers belong to which threads at any given point in time, but it turns out that this only adds a small amount to the complexity of the core logic. So, for the relatively cheap design cost of around 10-20% more logic in the core (and an almost negligible increase in transistor count and final production cost), the processor can execute several threads simultaneously, hopefully resulting in a substantial increase in functional unit utilization and instructions-per-clock (and thus overall performance). The instruction flow of an SMT processor looks something like...
This is really great! Now that we can fill those bubbles by running multiple threads, we can justify adding more functional units than would normally be viable in a single-threaded processor, and really go to town with multiple instruction issue. In some cases, this may even have the side effect of improving single-thread performance (for particularly ILP-friendly code, for example). So 20-issue here we come, right? Unfortunately, the answer is no. SMT performance is a tricky business. First, the whole idea of SMT is built around the assumption that either lots of programs are simultaneously executing (not just sitting idle), or if just one program is running, it has lots of threads all executing at the same time. Experience with existing multiprocessor systems shows that this isn't always true. In practice, at least for desktops, laptops and small servers, it is rarely the case that several different programs are actively executing at the same time, so it usually comes down to just the one task that the machine is currently being used for. Some applications, such as database systems, image & video processing, audio processing, 3D graphics rendering and scientific code, do have obvious high-level (course-grained) parallelism available and easy to exploit, but unfortunately many of these applications have not been written to make use of multiple threads in order to exploit multiple processors. In addition, many of the applications which are inherently parallel in nature are primarily limited by memory bandwidth, not by the processor (eg: image & video processing, audio processing, scientific code), so adding a second thread or processor won't help them – unless memory bandwidth is also dramatically increased. Worse yet, many other applications such as web browsers, multimedia design tools, language interpreters, hardware simulations and so on, are simply not inherently parallel enough to make effective use of multiple processors. On top of this, the fact that the threads are all sharing just one processor core, and just one set of caches, has major performance downsides compared to a true multiprocessor (or multi-core). Within the pipelines of an SMT processor, if one thread saturates just one functional unit which the other threads need, it effectively stalls all the other threads, even if they only need relatively little use of that unit. Thus, balancing the progress of the threads becomes critical, and the most effective use of SMT is for applications with highly variable code mixtures (so that the threads don't constantly compete for the same hardware resources). Also, competition between the threads for cache space may produce worse results than letting just one thread have all the cache space available – particularly for applications where the critical working set is highly cache-size sensitive, such as hardware simulators/emulators, virtual machines and high quality video encoding (with a large motion prediction window). The bottom line is that without care, and even with care for some applications, SMT performance can actually be worse than single-thread performance and traditional context switching between threads. On the other hand, applications which are limited primarily by memory latency, such as database systems and 3D graphics rendering, benefit dramatically from SMT, since it offers an effective way of using the otherwise idle time during cache misses (we'll cover caches later). Thus, SMT presents a very complex and application-specific performance picture. This also makes it a difficult challenge for marketing – sometimes almost as fast as two "real" processors, sometimes more like two really lame processors, sometimes even worse than one processor, huh? The Pentium-4 was the first (and is still the only) processor to use SMT, which Intel calls "Hyper-Threading". Its design allows for 2 simultaneous threads (although earlier revisions of the Pentium-4 have the SMT feature disabled due to bugs). The Alpha 21464 was an SMT design supporting 4 threads, but alas the Alpha architecture was terminated before the 21464 reached production. Several future processors will be SMT designs, including UltraSPARC-V and POWER5 (both of which will also be multi-core). Data Parallelism - SIMD Vector InstructionsIn addition to instruction parallelism, there is another source of parallelism in many programs – data parallelism. Rather than looking for ways to execute groups of instructions in parallel, the idea is to look for ways to make one instruction apply to a group of values in parallel. This is sometimes called SIMD parallelism (single instruction, multiple data). More often, it's called vector processing. Supercomputers used to use vector processing a lot, with very long vectors, because the types of scientific programs which are run on supercomputers are quite amenable to vector processing. Today, however, vector supercomputers have long since given way to multiprocessor designs where each processing unit is a commodity RISC CPU. So why revive vector processing? In many situations, especially in imaging, video and multimedia applications, a program needs to execute the same instruction for a small group of related values, usually a short vector (a simple structure or small array). For example, an image processing application might want to add groups of 8-bit numbers, where each 8-bit number represents one of the red, green, blue or alpha (transparency) values of a pixel...
What's happening here is exactly the same operation as a 32-bit addition, except that every 8th carry is not being propagated. Also, it might be desirable for the values not to wrap to zero once all 8 bits are full, and instead to hold at 255 as a maximum value in those cases (called saturation arithmetic). In other words, every 8th carry is not carried across but instead triggers an all-ones result. So, the vector addition operation shown above is really just a modified 32-bit add. From the hardware point of view, adding these types of vector instructions is not terribly difficult – existing registers can be used and in many cases the functional units can be shared with existing integer or floating-point units. Other useful packing and unpacking instructions can also be added, for byte shuffling and so on, and a few predicate-like instructions for bit-masking etc. With some thought, a small set of vector instructions can enable some impressive speedups. Of course, there's no reason to stop at 32 bits. If there happen to be some 64-bit registers, which architectures usually have for floating-point (at least), they could be used to provide 64-bit vectors, thereby doubling the parallelism (UltraSPARC VIS and x86 MMX do this). If it is possible to define entirely new registers, then they might as well be even wider (MMX2/SSE adds 128-bit registers for SIMD floating-point (only), while PowerPC AltiVec provides a full set of 32 new 128-bit registers4). The registers can also be divided up in other ways, for example as 16-bit integers or as 32-bit floating-point values. With AltiVec, for example, it is possible to execute a 4-way parallel floating-point multiply-add as a single, fully pipelined instruction.
Unfortunately, it's quite difficult for a compiler to automatically make use of vector instructions when working from normal source code, except in trivial cases. The key problem is that the way programmers write programs tends to serialize everything, which makes it difficult for a compiler to prove that two given operations are independent and can be done in parallel. Progress is slowly being made in this area, but at the moment programs must basically be rewritten by hand to take advantage of vector instructions (except for simple FORTRAN loops in scientific code). Luckily, however, rewriting just a small amount of code in key places within the graphics and video/audio libraries of your favorite operating system has a widespread effect across many applications. Today, most OS's have enhanced their key library functions in this way, so that virtually all multimedia and 3D graphics applications do make use of these highly effective vector instructions. Chalk up yet another win for abstraction! Almost every architecture has now added SIMD vector extensions, including SPARC (VIS), x86 (MMX/SSE, 3DNow!), PowerPC (AltiVec) and Alpha (MVI). Only relatively recent processors from each architecture can execute these new instructions, however, which raises backward compatibility issues. The MIPS architecture is notable for not adding SIMD vector instructions. Caches & The Memory HierarchyAs mentioned earlier, latency is a big problem for pipelined processors, and latency is especially bad for loads from memory, which make up about a quarter of all instructions. Loads tend to occur near the beginning of code sequences (basic blocks), with most of the other instructions depending on the data being loaded. This causes all the other instructions to stall, and makes it difficult to obtain large amounts of instruction-level parallelism. Things are even worse than they might first seem, because in practice most superscalar processors can still only issue one, or at most two, memory instructions per cycle. The core problem with memory access is that building a fast memory system is very difficult because of fixed limits, like the speed of light. These impose delays while a signal is transferred out to RAM and back. Nothing can change this fact of nature – we must learn to work around it. For example, access latency for main memory, even using a modern SDRAM with a CAS latency of 2, will typically be around 9 cycles of the memory system clock – 1 to send the address to the chipset (north bridge), 1 more to get it to the DIMM, RAS-to-CAS delay of 2 (assuming a page miss), CAS latency of 2, another 1 to get the data to the output buffer of the DIMM, 1 to send the data back to the chipset, and a final 1 to send the data up to the E-cache and processor5.
Furthermore, although a DDR SDRAM memory system transfers data on both the rising and falling edges of the clock signal (ie: at "double data rate"), the true clock speed of the memory system is still only half that, and it is the true clock speed which applies for control signals. So the latency of a DDR memory system is the same as a non-DDR system, even though the bandwidth is doubled (more on the difference between bandwidth and latency later). Also note that a substantial portion of memory latency (2 of the 9 bus cycles) involves the transfer of data between the processor and the chipset on the motherboard. One way to reduce this is to not change the memory system clock but dramatically increase the speed of the frontside bus (FSB) between the processor and the chipset (eg: 200-266 MHz DDR in Alpha 21264 & Athlon, 400-800 MHz DDR/QDR in Pentium-4, 800-1000 MHz DDR in PowerPC G5). An even better (but less flexible) approach is to integrate the memory controller directly onto the processor chip, which allows the 2 bus cycles to be converted into much faster processor cycles instead (eg: UltraSPARC-IIi/IIe & III, Opteron). Unfortunately, both DDR memory and faster frontside busses (or on-chip memory controllers) are only able to do so much – and memory latency continues to climb at an alarming rate as the clock speeds of processors continue to rise. This problem of the rapidly widening gap between the processor and memory is sometimes called the memory wall. It is perhaps the single most important problem facing hardware engineers and compiler writers today. The trend is clear – processors increase in clock speed by about 40-60% per year, while DRAM speeds increase by less than 10%. If this continues, it will soon make very little difference what type of processor is used – all that will matter is the memory system...
Modern processors try to solve this problem with caches6. A cache is a small but fast type of memory located on or near the processor chip. Its role is to keep copies of small pieces of main memory. When the processor asks for a particular piece of main memory, the cache can supply it much more quickly than main memory would be able to – if the data is in the cache.
It's a bit like working at a desk in a library7... You might have two or three books open on the desk itself. Accessing them is fast (you can just look), but you can't fit more than a couple on the desk at the same time – and even if you could, accessing 100 books laid out on a huge desk would take longer because you'd have to walk between them. Instead, in the corner of the desk you might have a pile of a dozen more books. Accessing them is slower, because you have to reach over, grab one and open it up. Each time you open a new one, you also have to put one of the books already on the desk back into the pile to make room. Finally, when you want a book that's not on the desk, and not in the pile, it's very slow to access because you have to get up and walk around the library looking for it. However the size of the library means you have access to thousands of books, far more than could ever fit on your desk.
Caches can achieve these seemingly amazing hit rates because of the way programs work. Most programs exhibit locality in both time and space – when a program accesses a piece of memory, there's a good chance it will need to re-access the same piece of memory in the near future (temporal locality), and there's also a good chance that it will need to access other nearby memory in the future as well (spatial locality). Temporal locality is exploited by merely keeping recently-accessed data in the cache. To take advantage of spatial locality, data is transferred from main memory up into the cache in blocks of a few dozen bytes at a time, called a cache block. From the hardware point of view, a cache works like a two column table – one column is the memory address and the other is the block of data values (remember that each cache line is a whole block of data, not just a single value). Of course, in reality the cache need only store the necessary higher-end part of the address, since lookups work by using the lower part of the address to index the cache. When the higher part, called the tag, matches the tag stored in the table, this is a hit and the appropriate piece of data can be sent to the CPU...
It is possible to use either the physical address or the virtual address to do the cache lookup. Each has pros and cons (like everything else in computing). Using the virtual address might cause problems because different programs use the same virtual addresses to map to different physical addresses – the cache might need to be flushed on every context switch. On the other hand, using the physical address means the virtual-to-physical mapping must be performed as part of the cache lookup, making every lookup slower. A common trick is to use virtual addresses for the cache indexing but physical addresses for the tags. The virtual-to-physical mapping (TLB lookup) can then be performed in parallel with the cache indexing so that it will be ready in time for the tag comparison. Such a scheme is called a virtually-indexed physically-tagged cache. The sizes and speeds of caches in modern processors differ quite a lot. The most important is the primary data cache. Some processors go for small data caches (Pentium-Pro/II/III, UltraSPARC & Itanium-I/II have 16k D-caches, Alpha 21164 & Pentium-4 are even smaller at just 8k – but they also have on-chip L2 caches). Others are a bit larger (32k for MIPS R10000, PowerPC G3/G4, G4e, G5 & Pentium-M, 64k for Alpha 21264, Athlon & UltraSPARC-III). For such caches, access latency is either 2 cycles (Alpha 21164, UltraSPARC, MIPS R10000, PowerPC G3/G4, UltraSPARC-III, Pentium-4, Itanium-I) or 3 cycles (Pentium-Pro/II/III/M, Alpha 21264, Athlon, PowerPC G4e & G5). Of particular note here is the Itanium-II, which achieves the remarkable (and ideal) feat of having a mere 1 cycle access latency for its D-cache! At the other extreme, some processors use much larger but slower L1 caches (HP's PA-8000 had a huge 1 MB L1 data cache placed off-chip as an array of expensive, ultra-high-performance SRAM's, greatly increasing the cost of the PA-8000). Many modern processors also have a second level of on-chip cache, between primary (L1) and external (E-cache). As a result, you should get out of the old habit of thinking of L2 as E-cache. Processors with on-chip L2 caches include the Alpha 21164 (but curiously not the 21264), the Pentium-4, PowerPC G4e & G5, Itanium-I, Pentium-M and Opteron. The Itanium-II includes both an on-chip L2 and an on-chip L3 cache! Relatively small L1 caches already take up to half of the chip area for many modern processors, so you can imagine how much area an L2 cache would take, yet this is still probably the best use for the high transistor budgets allowed by modern chip fabrication technology. Cache Conflicts & AssociativityIdeally, a cache should keep the data that is most likely to be needed in the future. Since caches aren't psychic, a good approximation of this is to keep the most recently used data. Unfortunately, keeping exactly the most recently used data would mean that data from any memory location could be placed into any cache line. The cache would thus contain exactly the most recently used n KB of data, which would be great for exploiting locality but unfortunately is not suitable for allowing fast access – accessing the cache would require checking every cache line for a possible match, which would be very slow for a modern cache with thousands of lines. Instead, a cache usually only allows data from any particular address in memory to occupy one, or at most a handful, of locations within the cache. Thus, only one or a handful of checks are required during access, so access can be kept fast (which is the whole point of having a cache in the first place). This approach does have a downside, however – it means the cache doesn't store the absolutely best set of recently accessed data, because several different locations in memory will all map to the same one location in the cache. When two such memory locations are wanted at the same time, such a scenario is called a cache conflict. Cache conflicts can cause "pathological" worst-case performance problems, because when a program repeatedly accesses two memory locations which happen to map to the same cache line, the cache must keep storing and loading from main memory and thus suffering the long main memory latency on each access. This type of situation is called thrashing, since the cache is not achieving anything and is simply getting in the way – despite obvious temporal locality and reuse of data, the cache is unable to exploit the locality offered by this particular access pattern due to limitations of its simplistic mapping between memory locations and cache lines. To address this problem, more sophisticated caches are able to place data in a small number of different places within the cache, rather than just a single place. The number of places a piece of data can be stored in a cache is called its associativity8. As described above, the simplest and fastest caches allow for only one place in the cache for each address in memory – each piece of data is simply mapped to address % size within the cache by simply looking at the lower bits of the address (as in the above diagram). This is called a direct mapped cache. Any two locations in memory whose addresses are the same for the lower address bits will map to the same cache line in a direct mapped cache, causing a cache conflict.
Each table, or way, may also have marker bits so that only the line of the least recently used way is evicted when a new line is brought in (or perhaps some faster approximation of that ideal). Usually, set-associative caches are able to avoid the misses that occasionally occur with direct mapped caches due to unfortunate cache conflicts. Adding even more ways allows even more conflicts to be avoided. Unfortunately, the more highly associative a cache is, the slower it is to access, because there are more comparisons to perform during each access. Even though the comparisons themselves are performed in parallel, additional levels of logic are required to select the appropriate hit, if any, and the cache may also need to update the marker bits appropriately within each way. More chip area is also required, because relatively more of the cache's data is consumed by tag information rather than data blocks, and extra datapaths are needed to access each individual way of the cache in parallel. Any and all of these factors may negatively affect access time. Thus, a 2-way set-associative cache is slower but smarter than a direct mapped cache, with 4-way and 8-way being slower and smarter again. In most modern processors, the primary on-chip instruction cache is usually set associative, but the data cache may go either way (direct mapped in Alpha 21164 & UltraSPARC; 2-way in MIPS R10000, Alpha 21264, Athlon & PowerPC G5; 4-way in Pentium-Pro/II/III, UltraSPARC-III, Pentium-4 & Itanium-I/II; 8-way in PowerPC G3/G4, G4e and Pentium-M). If present, any on-chip L2 cache is usually set associative, but the much larger external E-cache is normally direct mapped, though not always – the MIPS R10000, PowerPC G3/G4 and UltraSPARC-III have E-caches which are 2-way set associative, and the PowerPC G4e features an 8-way set-associative E-cache. The concept of caches also extends up into software systems. For example, main memory is used to cache the contents of the filesystem to speed up disk I/O, and web caches (also known as proxy-caches) cache the contents of remote web servers on a more local server. With respect to main memory and virtual memory (paging/swapping), it can be thought of as being a smart, fully associative cache, like the ideal cache mentioned initially (above). After all, the virtual memory system is managed by the (hopefully) intelligent software of the operating system kernel. Memory Bandwidth vs LatencySince memory is transferred in blocks, and since cache misses are an urgent "show stopper" type of event with the potential to halt the processor in its tracks (or at least severely hamper its progress), the speed of those block transfers from memory to the E-cache is critical. The transfer rate of a memory system is called its bandwidth. But how is that different from latency? A good analogy is a highway... Suppose you want to drive in to the city from 100 miles away. By doubling the number of lanes, the total number of cars that can travel per hour (the bandwidth) is doubled, but your own travel time (the latency) is not reduced. If all you want to do is increase cars-per-second, then adding more lanes (wider bus) is the answer, but if you want to reduce the time for a specific car to get from A to B then you need to do something else – usually either raise the speed limit (bus & DRAM speed), or reduce the distance, or perhaps build a regional mall so that people don't need to go to the city as often (a cache). When it comes to memory systems, there are often subtle trade-offs between latency and bandwidth. Lower latency designs will be better for pointer chasing code, such as compilers and database systems, whereas bandwidth-oriented systems have the advantage for programs with simple linear access patterns, such as image processing and scientific code. The two major memory technologies of today, standard SDRAM and Rambus RDRAM, differ slightly in this respect – for any given level of chip technology, SDRAM should have lower latency but RDRAM should have higher bandwidth. This is due to the "snake-like" structure of RDRAM memory systems, which reduce signal reflections by avoiding splitting the wires to go to each memory module in parallel and instead go "through" each module in sequence – allowing RDRAM systems to run at higher clock speeds but with a longer average length to the memory modules. Of course, it's reasonably easy to increase bandwidth – simply adding more memory banks and making the busses wider can easily double or quadruple bandwidth. In fact, many high-end systems do this to increase their performance, but it comes with downsides as well. In particular, wider busses mean a more expensive motherboard, restrictions on the way RAM can be added to a system (install in pairs or groups of 4) and a higher minimum RAM configuration. An example of the two extremes is the difference between Apple's iMac systems and Sun's high-end SunBlade workstations. The high-end SunBlade has a 512-bit datapath to RAM, which allows excellent memory bandwidth, but DIMM's must be installed in groups of 4 and the minimum RAM configuration is 512 MB (4x128). The iMac, on the other hand, only has a 64-bit wide bus, and consequently has lower memory bandwidth, however DIMM's can be installed one-at-a-time and the minimum configuration is a mere 64 MB. These two extremes stem from differences in the markets the two systems target – few high-end SunBlade users would want less than 512 MB of RAM, whereas a 512 MB minimum configuration might make the iMac too expensive for many of its target users. Unfortunately, latency is much harder to improve than bandwidth – as the saying goes: "you can't bribe god". Even so, there have been some good improvements in effective memory latency over the past few years, chiefly in the form of synchronously-clocked DRAM (SDRAM) which uses the same clock as the memory bus. The main benefit of SDRAM is that it allows pipelining of the memory system, because the internal timing aspects and interleaved structure of SDRAM chip operation are exposed to the system and can thus be taken advantage of. This reduces effective latency because it allows a new memory access to be started before the current one has completed, thereby eliminating the small amounts of waiting time found in asynchronous DRAM systems, which must wait for the current access to complete before starting the next (on average, an asynchronous memory system must wait for the transfer of half an E-cache block from the previous access before starting a new request, which is often several bus cycles). In addition to the reduction in effective latency, there is also a substantial increase in bandwidth, because in an SDRAM memory system multiple memory requests can be outstanding at any one time, all being processed in a highly efficient, fully pipelined fashion. Pipelining of the memory system can have dramatic effects for memory bandwidth – it is not uncommon for an SDRAM memory system to provide double or triple the sustained memory bandwidth of an asynchronous memory system, even though the latency of the SDRAM system is only slightly lower. Will further improvements in DRAM technology be able to hold off the memory wall for much longer? It will be interesting to watch... AcknowledgementsThe overall style of this article, particularly with respect to the style of the processor "instruction flow" and microarchitecture diagrams, is derived from the combination of a well-known 1989 ASPLOS research paper by Norman Jouppi and David Wall, the book POWER & PowerPC by Shlomo Weiss and James Smith, and the two very famous Hennessy/Patterson textbooks Computer Architecture: A Quantitative Approach and Computer Organization and Design. There have, of course, been many other presentations of this same material, and naturally they are all somewhat similar, however the above four are exceptionally good (in my opinion). To learn more about these topics, those books are an excellent place to start. More Detailed Information?If you want more detail on the specifics of recent processor designs – and something more insightful than the raw technical manuals – here are a few good articles...
And if you want to keep up with the latest news in the world of microprocessors... That should keep you busy! |
Home | Software | Articles | About | Search | Help |
| Copyright © 1991-2003 PattoSoft. All rights reserved. Terms of Use | Privacy Policy | Legal Information |


This page lists all the events that can be triggered for the
google.maps.Map
object and shows when that happens, by highlighting the names to the right.
This page lists all the events that can be triggered for the
google.maps.Map
object and shows when that happens, by highlighting the names to the right.


Me and 2 friends decided to try 5-meo-dipt for the first time about a week ago. We each had 10mg each home at my place.
After about 20-30 min my friends said they were starting to feel some effects and wanted to go outside.
We went outside for a walk. I strongly regreted I have had dinner so close before we took the drug cause I still didnt feel a thing and by this time my friends were already tripping. After about 1 hour or more we were up in a nice park and decided to lay on our back on a hill and just look at the sky.
By that time the effects slowly started to kick in for me. I started to see some buzzing lights moving around the sky and feeling somehow strange, the feeling is hard to explain but it started to grow and after laying on the hill for about 15 mins we went up and started walking again.
The park seemed much more interesting then before. The nature all around us looked different, it seemed very beautiful instead of just looking ordinary. We all sat around under a big tree. Oh this was really great- the sun, the trees, the grass and the sky, it was all so very beautiful. I lied down on my back and just looked up at the trees and the sky above us.
We started to walk again. Laughing at simple things like bushes and colorful leafves lying on the ground. Colors seemed much brighter too and everything we looked at was clearer in a strange way like we were watching it trough a lens. We then went to store and bought some snacks and stuff that we ate sitting on a bench. It also seemed to taste different, more intense and somehow better taste.
After that we went home to my place again. I lied down in the bed and listened to some music [my choice: Radiohead and Belle & Sebastian] Now music was fantastic !! I could hear each tone and every instrument in the songs clearer then ever before. The music also seemed to slow down wich felt amazing. I felt really really good :)
Some time later we decided to watch some episodes of The Simpsons. We were really laughing hysterically at the jokes. I usually find Simpsons to be very funny but this was extreme. We watched maybe 3 episodes then it started to get late and my friends were heading home.
It had been a great day!
I was feeling pretty tired but somehow I was unable to sleep :( I watched some Tv but I just couldnt sleep.... lied awake until the morning. Went out of bed and had breakfast, sat at my computer for a while and then finally I was able to get some sleep. I dont know why I was unable to sleep for so long, didnt feel any nauesa or anything.
Well it was a nice experience and I would surley do it again. My only regret was that I ate before I took the drug. Oh well I wont do that next time.
But this was fun stuff.
Experience Reports are the writings and opinions of the individual authors who submit them.
Some of the activities described are dangerous and/or illegal and none are recommended by Erowid.


- getDestinationNode() - Method in interface it.polito.dp2.NFFG.LinkReader
-
Gives access to the information associated to the destination node of this link.
- getDestinationNode() - Method in interface it.polito.dp2.NFFG.ReachabilityPolicyReader
-
Gives the destination node of this policy.
- getException() - Method in error it.polito.dp2.NFFG.FactoryConfigurationError
-
Returns the underlying cause of this exception, if any.
- getException() - Method in exception it.polito.dp2.NFFG.NffgVerifierException
-
Returns the underlying cause of this error, if any.
- getFuncType() - Method in interface it.polito.dp2.NFFG.NodeReader
-
Gives the functional type of the node.
- getLinks() - Method in interface it.polito.dp2.NFFG.NodeReader
-
Gives the known links that connect this node (i.e. have this node as the source node) in the network topology.
- getMessage() - Method in error it.polito.dp2.NFFG.FactoryConfigurationError
-
Returns the message for this error, if any.
- getMessage() - Method in exception it.polito.dp2.NFFG.NffgVerifierException
-
Returns the message for this exception, if any.
- getName() - Method in interface it.polito.dp2.NFFG.NamedEntityReader
-
Gives the name of this entity.
- getNffg(String) - Method in interface it.polito.dp2.NFFG.NffgVerifier
-
Gives access to a single NF-FG given its name.
- getNffg() - Method in interface it.polito.dp2.NFFG.PolicyReader
-
Gives the name of the NF-FG on which this policy has to be verified
(a string made of alphanumeric characters only, where the first character is alphabetic).
- getNffgs() - Method in interface it.polito.dp2.NFFG.NffgVerifier
-
Gives access to the set of known NF-FGs.
- getNode(String) - Method in interface it.polito.dp2.NFFG.NffgReader
-
Gives access to a single network node given its name (a node is uniquely identified in the NF-FG by its name).
- getNodes() - Method in interface it.polito.dp2.NFFG.NffgReader
-
Gives access to the set of network nodes of this NF-FG.
- getPolicies() - Method in interface it.polito.dp2.NFFG.NffgVerifier
-
Gives access to the set of known policies to be verified.
- getPolicies(String) - Method in interface it.polito.dp2.NFFG.NffgVerifier
-
Gives access to the set of known policies to be verified,
filtered by NF-FG's name.
- getPolicies(Calendar) - Method in interface it.polito.dp2.NFFG.NffgVerifier
-
Gives access to the set of known policies to be verified, filtered by
verification time and date.
- getPolicy() - Method in interface it.polito.dp2.NFFG.VerificationResultReader
-
Gives the policy to which this result is associated.
- getResult() - Method in interface it.polito.dp2.NFFG.PolicyReader
-
Gives the result of the verification of this policy.
- getSourceNode() - Method in interface it.polito.dp2.NFFG.LinkReader
-
Gives access to the information associated to the source node of this link.
- getSourceNode() - Method in interface it.polito.dp2.NFFG.ReachabilityPolicyReader
-
Gives the source node of this policy.
- getTraversedFuctionalTypes() - Method in interface it.polito.dp2.NFFG.TraversalPolicyReader
-
Gives the set of network functionalities that must be traversed for the property of this policy to hold.
- getUpdateTime() - Method in interface it.polito.dp2.NFFG.NffgReader
-
Gives the date and time of the last update of the NF-FG.
- getVerificationResult() - Method in interface it.polito.dp2.NFFG.VerificationResultReader
-
Gives the result of the verification.
- getVerificationResultMsg() - Method in interface it.polito.dp2.NFFG.VerificationResultReader
-
Gives a human-readable message associated with the result of the verification.
- getVerificationTime() - Method in interface it.polito.dp2.NFFG.VerificationResultReader
-
Gives the date when the policy has been verified.


God in Islam
Who is Allah?
It is a known fact that every language has one or more terms that are used in reference to God and sometimes to lesser deities. This is not the case with Allah. Allah is the personal name of the One true God. Nothing else can be called Allah. The term has no plural or gender. This shows its uniqueness when compared with the word god which can be made plural, gods, or feminine, goddess. It is interesting to notice that Allah is the personal name of God in Aramaic, the language of Jesus and a sister language of Arabic.
The One true God is a reflection of the unique concept that Islam associates with God. To a Muslim, Allah is the Almighty, Creator and Sustainer of the universe, Who is similar to nothing and nothing is comparable to Him. The Prophet Muhammad was asked by his contemporaries about Allah; the answer came directly from God Himself in the form of a short chapter of the Quran, which is considered the essence of the unity or the motto of monotheism. This is chapter 112 which reads:
"In the name of God, the Merciful, the Compassionate.
Say (O Muhammad) He is God the One God, the Everlasting Refuge, who has not begotten, nor has been begotten, and equal to Him is not anyone."
 He is not loving and kind. Nothing can be farther from truth than this allegation. It is enough to know that, with the exception of one, each of the 114 chapters of the Quran begins with the verse: "In the name of God, the Merciful, the Compassionate." In one of the sayings of Prophet Muhammad (PBUH) we are told that "God is more loving and kinder than a mother to her dear child."
But God is also Just. Hence evildoers and sinners must have their share of punishment and the virtuous, His bounties and favors. Actually God's attribute of Mercy has full manifestation in His attribute of Justice. People suffering throughout their lives for His sake and people oppressing and exploiting other people all their lives should not receive similar treatment from their Lord. Expecting similar treatment for them will amount to negating the very belief in the accountability of man in the Hereafter and thereby negating all the incentives for a moral and virtuous life in this world. The following Quranic verses are very clear and straightforward in this respect:
"Verily, for the Righteous are gardens of Delight, in the Presence of their Lord. Shall We then treat the people of Faith like the people of Sin? What is the matter with you? How judge you?" (68:34-36)
Islam rejects characterizing God in any human form or depicting Him as favoring certain individuals or nations on the basis of wealth, power or race. He created the human beings as equals. They may distinguish themselves and get His favor through virtue and piety only.
The concept that God rested in the seventh day of creation, that God wrestled with one of His soldiers, that God is an envious plotter against mankind, or that God is incarnate in any human being are considered blasphemy from the Islamic point of view.
The unique usage of Allah as a personal name of God is a reflection of Islam's emphasis on the purity of the belief in God which is the essence of the message of all God's messengers. Because of this, Islam considers associating any deity or personality with God as a deadly sin which God will never forgive, despite the fact He may forgive all other sins.
[Note that what is meant above applies ONLY to those people who die in a state wherein they are associating others with God.  - MSA of USC]
The Creator must be of a different nature from the things created because if he is of the same nature as they are, he will be temporal and will therefore need a maker. It follows that nothing is like Him. If the maker is not temporal, then he must be eternal. But if he is eternal, he cannot be caused, and if nothing outside him causes him to continue to exist, which means that he must be self-sufficient. And if the does not depend on anything for the continuance of his own existence, then this existence can have no end. The Creator is therefore eternal and everlasting: "He is the First and the Last."
He is Self-Sufficient or Self-Subsistent or, to use a Quranic term, Al-Qayyum. The Creator does not create only in the sense of bringing things into being, He also preserves them and takes them out of existence and is the ultimate cause of whatever happens to them.
"God is the Creator of everything. He is the guardian over everything. Unto Him belong the keys of the heavens and the earth." (39:62, 63)
"No creature is there crawling on the earth, but its provision rests on God. He knows its lodging place and it repository." (11:6)
God's Attributes
If the Creator is Eternal and Everlasting, then His attributes must also be eternal and everlasting. He should not lose any of His attributes nor acquire new ones. If this is so, then His attributes are absolute. Can there be more than one Creator with such absolute attributes? Can there be for example, two absolutely powerful Creators? A moment's thought shows that this is not feasible.
The Quran summarizes this argument in the following verses:
"God has not taken to Himself any son, nor is there any god with Him: For then each god would have taken of that which he created and some of them would have risen up over others." (23:91)
And Why, were there gods in earth and heaven other than God, they (heaven and earth) would surely go to ruin." (21:22)
The Oneness of God
The Quran reminds us of the falsity of all alleged gods. To the worshippers of man-made objects, it asks:
"Do you worship what you have carved yourself?" (37:95)
"Or have you taken unto you others beside Him to be your protectors, even such as have no power either for good or for harm to themselves?" (13:16)
To the worshippers of heavenly bodies it cites the story of Abraham:
"When night outspread over him he saw a star and said, 'This is my Lord.' But when it set he said, 'I love not the setters.' When he saw the moon rising, he said, 'This is my Lord.' But when it set he said, 'If my Lord does not guide me I shall surely be of the people gone astray.' When he saw the sun rising, he said, 'This is my Lord; this is greater.' But when it set he said, 'O my people, surely I quit that which you associate, I have turned my face to Him Who originated the heavens and the earth; a man of pure faith, I am not of the idolaters.'" (6:76-79)
The Believer's Attitude
In order to be a Muslim, i.e., to surrender oneself to God, it is necessary to believe in the oneness of God, in the sense of His being the only Creator, Preserver, Nourisher, etc. But this belief - later on called "Tawhid Ar-Rububiyyah" - is not enough. Many of the idolaters knew and believed that only the Supreme God could do all this, but that was not enough to make them Muslims. To tawhid ar-rububiyyah one must add tawhid al'uluhiyyah, i.e., one acknowledges the fact that is God alone Who deserves to be worshipped, and thus abstains from worshipping any other thing or being.
Having achieved this knowledge of the one true God, man should constantly have faith in Him, and should allow nothing to induce him to deny truth.
When faith enters a person's heart, it causes certain mental states which result in certain actions. Taken together these mental states and actions are the proof for the true faith. The Prophet said, "Faith is that which resides firmly in the heart and which is proved by deeds." Foremost among those mental states is the feeling of gratitude towards God which could be said to be the essence of 'ibada' (worship).
The feeling of gratitude is so important that a non-believer is called 'kafir' which means 'one who denies a truth' and also 'one who is ungrateful.'
A believer loves, and is grateful to God for the bounties He bestowed upon him, but being aware of the fact that his good deeds, whether mental or physical, are far from being commensurate with Divine favors, he is always anxious lest God should punish him, here or in the Hereafter. He, therefore, fears Him, surrenders himself to Him and serves Him with great humility. One cannot be in such a mental state without being almost all the time mindful of God. Remembering God is thus the life force of faith, without which it fades and withers away.
The Quran tries to promote this feeling of gratitude by repeating the attributes of God very frequently. We find most of these attributes mentioned together in the following verses of the Quran:
"He is God; there is no god but He, He is the Knower of the unseen and the visible; He is the All-Merciful, the All-Compassionate. He is God, there is no God but He. He is the King, the All-Holy, the All-Peace, the Guardian of Faith, the All-Preserver, the All-Mighty, the All-Compeller, the All-Sublime. Glory be to God, above that they associate! He is God the Creator, the Maker, the Shaper. To Him belong the Names Most Beautiful. All that is in the heavens and the earth magnifies Him; He is the All-Mighty, the All-Wise." (59:22-24)
"There is no god but He, the Living, the Everlasting. Slumber seizes Him not, neither sleep; to Him belongs all that is in the heavens and the earth. Who is there that shall intercede with Him save by His leave? He knows what lies before them and what is after them, and they comprehend not anything of His knowledge save such as He wills. His throne comprises the heavens and earth; the preserving of them oppresses Him not; He is the All-High, the All-Glorious." (2:255)
"People of the Book, go not beyond the bounds in your religion, and say not as to God but the truth. The Messiah, Jesus son of Mary, was only the Messenger of God, and His Word that He committed to Mary, and a Spirit from Him. So believe in God and His Messengers, and say not, 'Three.' Refrain; better is it for you. God is only one God. Glory be to Him - (He is) above having a son." (4:171)
This article is published by The Institute of Islamic Information and Education (III&E)


Raw organism counts
were normalized to the sample volume using the following formula:
\(organisms/mL = {CA_{c} \over VA_{f}F}\)
where \(C\) is the organism count, \(A_{c}\) is the area of the cell bottom (mm\(^2\)), \(A_{f}\)
is the area of each grid field (mm\(^2\)), \(F\) is the number of fields examined, and
\(V\) is the settled volume (mL). This simplifies to:
\(organisms/mL = {C \over cV}\)
where \(cV = {A_{c} \over VA_{f}F}\) and is equal to the counted volume.


Used to show what has occurred within any given entity. Should show a cursory overview of each item and allow users to dig into the details
35MB · PDF Document
35MB · PDF Document
35MB · PDF Document
Start of email begins here and then may wrap around and do some other stuff too - what do you think? of this idea? more text...


This global inspection reports any classes which are mutually or cyclically dependent on other classes. Such cyclic dependencies make for fragile code and high maintenance costs.
Powered by InspectionGadgets


Which programming language is right for me?
To give you a a better idea of which programming language and track you should take, please answer the following questions.
The programming language you should learn is C#
The programming language you should learn is CSS/React
The programming language you should learn is Ruby/rails


See your most recent campaign summary here
This is a section that gives a summarized text for the most recent campaign held from your account. It provides insight on whom the campaign was targeted to and to which groups. It also provides insights on the statistical insights for the campaign.
| Statistical Analysis | Hit Percentage |
| Total Reach | |
| Total clicks | |
| Usubscribes |
Check how many people have opted out of this portal
This is a section that gives a summary of recent unsubscribes for the last 60 days
| Serial No | Contact Name | Date | |||||
| No unsubscriptions found | |||||||
| {{data.serialNo}} | {{data.firstName}} {{data.lastName}} | {{data.email}} | {{data.unsubscribedOn}} |


What is it?
Glottologist is JavaScript framework that works in the browser (or on NodeJS). It will help you to provide a different version of your website, depending on the user language.
 The philosophy behind Glottologist (and my other projects), is that you should only add, not remove or refactorize the code. So, you'll be able to integrate Glottologist with each of your projects 😊.
Installation
Glottologist is simple to install and setup. Here is how you install the browser and node version.


As a member of the Open JS Foundation, Express has adopted the Contributor Covenant 2.0.
If an issue arises and you cannot resolve it directly with the parties involved, you can report it to the Express project TC through the following email<EMAIL_ADDRESS>In addition, the OpenJS Foundation maintains a Code of Conduct Panel (CoCP). This is a foundation-wide team established to manage escalation when a reporter believes that a report to a member project or the CPC has not been properly handled. In order to escalate to the CoCP send an email to<EMAIL_ADDRESS>We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
Examples of behavior that contributes to a positive environment for our community include:
Examples of unacceptable behavior include:
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at<EMAIL_ADDRESS>All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
Community Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
Consequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
Community Impact: A violation through a single incident or series of actions.
Consequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
Community Impact: A serious violation of community standards, including sustained inappropriate behavior.
Consequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
Community Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
Consequence: A permanent ban from any sort of public interaction within the project community.
This Code of Conduct is adapted from the Contributor Covenant, version 2.0.
Community Impact Guidelines were inspired by Mozilla’s code of conduct enforcement ladder.
For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations.


| HOME | DOWNLOAD | DOCUMENTATION | FORUM | FAQ |
> Home > Documentation
Complex Spherical Harmonics
"Geodesy" 4π-normalized complex spherical harmonics
Any complex square-integrable function can be expressed in spherical harmonics as
where flm is the complex spherical harmonic coefficient, Ylm is the complex spherical harmonic function, Ω represents position on the sphere in terms of co-latitude θ and longitude φ,and l and m are the spherical harmonic degree and order, respectively. The complex spherical harmonics are defined as
where the normalized associated Legendre functions, as used in geodesy and gravity applications (calculated by the routine PlmBar with cnorm=1), are given by
with the following definition for the unnormalized associated Legendre functions:
The normalized associated Legendre functions are orthogonal for a given value of m,
.The normalized Legendre functions (including the orthonormalized and Schmidt semi-normalized functions) are efficiently calculating using the algorithm of Holmes and Featherstone (2002, J. Geodesy, 76, 279-299) that is accurate to approximately degree 2800. Note that the above definition of the associated Legendre functions does not include the Condon-Shortley phase of (-1)m that is often employed in the physics and seismology communities (e.g., Varsalovich et al., 1988; Dahlen and Tromp, 1998). This can be included by specifying the optional argument CSPHASE=-1 in most of the Legendre and spherical harmonic routines. (Alternatively, the default phase convention can be modified during compilation of the SHTOOLS archive.)
The complex spherical harmonics possess the following symmetry relationship for positive and negative angular orders,
and satisfy the orthogonality relationship
.Using this relationship, it is straightforward to show that the spherical harmonic coefficients of a function can be calculated by the integral
.The cross-power of two functions f and g is equal to
,where the "cross-power spectrum" of the two functions is given by
.If the functions f and g have a zero mean, then Sfg is the contribution to the covariance as a function of spherical harmonic degree.
If a function defined on the sphere is entirely real, then the real and complex spherical harmonic coefficients are related by
Other complex normalization conventions
The above 4π normalization scheme is the default for all routines in SHTOOLS with the exception of MakeMagGrid2D, which uses the standard geomagnetism Schmidt semi-normalization. By specifying an optional parameter norm, the default normalization can be changed in most of the routines (e.g., PreCompute, MakeGridGLQ, SHExpandGLQ, SHExpandDH, MakeGrid2D, MakeGridPoint, SHMultiply and SHExpandLSQ). The following normalizations are permitted:
- norm=1: Geodesy 4π normalization (default, unless stated otherwise).
- norm=2: Schmidt semi-normalization.
- norm=3: Unnormalized.
- norm=4: Orthonormalized.
Explicit expressions for the Legendre functions and real spherical harmonic normalizations are the following:
Geodesy 4π
Schmidt semi-normalized
Unnormalized
Orthonormalized
Further information
Further information regarding spherical harmonic functions can be found at the following web sites:
> Home > Documentation
| Centre National de la Recherche Scientifique | Institut de Physique du Globe de Paris | © 2011 Mark Wieczorek |


| Have you ever had a DUI? | ||
| Primary: | ||
| Spouse: | ||
| Do you smoke Tobacco or Marijuana | ||
| Primary: | ||
| Spouse: | ||
| Primary: | ||
| Spouse: | ||
| Have you had any health issues in your lifetime | ||
| Primary: | ||
| Spouse: |
| Primary Insured | ||||
| First Name | Date of Birth / Age | Tobacco User | Gender | Children under 19 |
|
|
| |||
| Spouse | ||||
| First Name | Date of Birth / Age | Tobacco User | Gender | Children under 19 |
|
|
|
| You and your Spouses Hourly Wage |
|
|
| Life Insurance outside of work | ||
| Are you planning on being buried | ||
| Are you planning on being cremated | ||
| Do you plan on being buried in the VA National Cemetary | ||
| Have you purchused plots | ||
| Are you currently paying or have you prepaid any burial costs |
|
|
|
Daily Value:
|
Recommended:
Used:
Remaining:
|


LIFE AT SMART SCIENCE COLLEGE
ACADEMIC ACTIVITIES
Our Academic process is broken down into series by the Ministry of Education Nigeria, so that impacting of knowledge into students can followed to give our best. We make sure no student in the school is less than the average score so to maintain a nice learning environment, but if that happens based on a resonlble condition, the student is meant to take that class again or other wise.
BOARDING HOUSE
We do have a good boarding house with nice facilities. Parents are encourage to use this if they live outside calabar. There is a matron and a sit-in night teacher in the hostels.
SAFETY AND SECURITY
At Smart Science College, the security and well being of our students is of utmost improtance to us. For this reason we have well trained out-sourced security outfit that patrol the schools' perimeter both day and night.
EXTRA-CURRICULAR ACTIVITIES
At Smart Science College, we know that every student's forte lies in academics; nothing is wrong with that once it is recognized. The many icons today did not necessarily excel in academics e.g., David Branson, etc. So we therefore encourage students to join clubs and to participate in sports.
The extracurricular programme is also desingend to expose students to the real world. Some include;
FACILITIES
SCIENCE LABORATORY
We have nice and well equiped labratories for practical works during the learning session, the laboratories ranging from the physics lab, chemistry lab, biology lab and craft lab. At the laboratory the students safety is also our concern therefore all our scince staffs are areaapproprriately trained on safety and a wide range of safety equipment
COMPUTER LABORATORY
We have nice and well-maintained Computers with internet services.
MUSIC ROOM
The music room was built for those that have talents in music and are intrested in using that talents, although this is a science school there are some students that are intrested in music as an elective subject
LIBRARY
We have a well-stocked library (rated 'A' by the outreach Abuja Inspection Team). The comprehensive stock of books meets not only the academic needs of students, but also encourages their leisure reading.
TECHNICAL DRAWING STUDIO
Our Technical Drawing studio is a well-equipped one it was rated 'A' by T.D Inspection Team
ABOUT US
Smart science high college was established in the year 1956, we were established by Professor.Dr. Akak Favour Eyo with the aim of providing stable educational background for children intrested in science about cars providing good and interactive services to our vistors.
LINKS
GALLERY
©2018 - 2019 School. All Rights Reserved


{% extends 'ctflex/base/headered.template.html' %} {% load static from staticfiles %} {% load sponsor from ctflex_sponsors %} {% block title %} Sponsors - {{ block.super }} {% endblock %} {% block header %} Our Sponsors {% endblock %} {% block extra_css %} {{ block.super }} {% endblock %} {% block extra_header %}
For 180 years, J.P. Morgan Asset & Wealth Management has brought its fiduciary mindset to help clients navigate portfolios. Throughout this time, we have institutionalized our insights and passed on the cumulative wisdom and knowledge of those before us to incoming generations. As stewards of our clients’ wealth, our mission at J.P. Morgan is to help clients of all types get, and stay, properly invested. These clients can choose to work with any firm they wish. They turn to J.P. Morgan because they know we will be there for them when they need us most and that we will always put their interests first.
In today’s digital world where financial institutions and their clients and customers are increasingly targeted by sophisticated cyber-attacks, J.P. Morgan Asset & Wealth Management places Cybersecurity as a top priority. To help clients prevent and more effectively react to cybercrime, we are educating our employees and customers, proactively detecting internal violations, blocking millions of suspicious emails every day and meeting with clients to improve their awareness and provide fraud prevention training.
{% endsponsor %} {% endwith %}


Mary Sutherland is an author and researcher focusing her work
on consciousness studies, ancient history and unusual
phenomena.
For the last 5 years, she has been
exploring, mapping and documenting the ancient underwater
structures of Rock Lake – near Aztalan. For the last fourteen
years she has been documenting the ancient sites around
Burlington, WI.
She believes it is through
truth that we will break ourselves free of our present
entanglements in life.
These woods are filled with activity - gateways into other parallel worlds, time loops- strange lights- wood spirits - ghosts and the illusive 'big foot'. By day these woods are 'enchanted' - By night - even the bravest of hearts have fled - including both of your hosts for this show!'
click to know moreOur universe is operating on a certain wavelength. A phased shift, where we simply operate on a slightly different wavelength, may reveal whole new worlds that are beyond our sensory constructs. Interference between these wavelengths has been postulated as an answer for ghosts and other paranormal activity
click to know moreIn the situation where there are hyperdimensional overlappings of one or more 3D universes, these overlappings may be of finite area and last for some finite time. These overlappings could allow, on occasion, for materials and entities from the inter-involved 3D universes to "feed-through" into other 3D universes.
click to know moreWhen the veil drops people will be shocked to see what is all around them. Some so shocked they will lose their minds - others will commit suicide - and others that prepare now will accept what they will come to know has always been there.
click to know moreRoswell has been at the heart of the UFO scene since in July 1947 the military sensationally announced in a press release it had found the remains of a crashed flying saucer in the desert nearby. This event is considered as the birth of modern ufology and has led to numerous debates, theories and speculations about the existence of extraterrestrial life that many other cases totally unfounded , accepting the official version that mentions the collision of a meteorological balloon in a context of great secrecy like that of the Cold War.
click to know moreJourney into an incredible spiritual terrain unknown to all but the most adventurous seekers. The Mirrored World - a hyperdimensional reality that exists beyond the constraints of time, space, and intellect.  A reality which, once you enter it, will reveal the astonishing power of true, authentic spirituality.
click to know more