Advanced CPU Designs: Crash Course Computer Science #9

Summary

This CrashCourse Computer Science episode explores the evolution of computer processors from mechanical devices to Gigahertz-speed CPUs. It delves into techniques like instruction pipelining, cache usage, and multi-core processors that enhance performance. The video also touches on challenges like data bottlenecks and the importance of efficient programming to harness the immense processing power available today.

Takeaways

• 🚀 Computers have evolved from mechanical devices to processors running at Gigahertz speeds, executing billions of instructions per second.
• 🔍 Early processors increased speed by improving transistor switching times, but this approach has limitations, leading to the development of various performance-boosting techniques.
• 🛠️ Modern processors include specialized circuits for complex tasks like graphics operations, video decoding, and file encryption, which are known as MMX, 3DNow!, or SSE.
• 📈 The instruction set of processors has grown over time, with modern processors having thousands of instructions for enhanced capabilities and backward compatibility.
• 🔄 High clock speeds create a data bottleneck with RAM, which is addressed by using caches to store frequently accessed data closer to the CPU, reducing access time.
• 💡 Caching improves efficiency through cache hits and cache misses, utilizing a 'dirty bit' to manage data synchronization between cache and RAM.
• 🔄 Instruction pipelining allows multiple instructions to be processed simultaneously in different stages of the CPU, increasing throughput.
• 🔄 Out-of-order execution in high-end processors dynamically reorders instructions to minimize pipeline stalls and improve efficiency.
• 🤖 Speculative execution and branch prediction are techniques used to deal with conditional jumps, guessing the flow of execution to reduce delays.
• 🔢 Superscalar processors can execute multiple instructions per clock cycle by utilizing idle areas of the CPU or adding duplicate circuitry for popular instructions.
• 💻 Multi-core processors allow for multiple independent processing units within a single CPU chip, sharing resources and improving performance on shared computations.
• 🌐 Supercomputers, like the Sunway TaihuLight, utilize millions of cores to perform an immense number of calculations, showcasing the pinnacle of computational power.

Q & A

How have computers evolved from their early days to the present?

Computers have evolved from mechanical devices capable of one calculation per second to CPUs running at Gigahertz speeds, executing billions of instructions every second.

What was one of the early methods to make processors faster?

One of the early methods to make processors faster was by improving the switching time of the transistors inside the chip, which make up all the logic gates, ALUs, and other components.

Why are additional circuits added to modern computer processors?

Additional circuits are added to modern computer processors to perform more sophisticated operations and to execute instructions that would take many clock cycles with standard operations, such as graphics operations, video decoding, and file encryption.

What is the significance of MMX, 3DNow!, and SSE in processors?

MMX, 3DNow!, and SSE are extensions to the instruction set that allow processors to execute additional instructions for specific tasks like gaming and encryption, enhancing performance for these operations.

Why did the Intel 4004, the first integrated CPU, only have 46 instructions?

The Intel 4004 had 46 instructions because that was enough to build a fully functional computer at the time. As technology advanced, more instructions were needed to perform a wider variety of tasks.

What is the role of a cache in a CPU?

A cache is a small piece of RAM located on the CPU that stores data to speed up access times. It helps to alleviate the bottleneck caused by the slower speed of RAM compared to the CPU.

What is a cache hit and a cache miss?

A cache hit occurs when the data requested from RAM is already stored in the cache, allowing for faster access. A cache miss happens when the data is not in the cache, requiring a slower access from the main RAM.

What is instruction pipelining and how does it improve CPU performance?

Instruction pipelining is a technique where different stages of instruction processing (fetch, decode, execute) are overlapped, allowing for continuous operation and higher throughput, effectively executing one instruction per clock cycle.

What is the purpose of speculative execution in CPUs?

Speculative execution is a technique used by advanced CPUs to guess the outcome of a conditional jump instruction and start filling the pipeline with instructions based on that guess, reducing delays when the jump is resolved.

How do superscalar processors differ from regular pipelined processors?

Superscalar processors can execute more than one instruction per clock cycle by fetching and decoding multiple instructions at once and executing instructions that require different parts of the CPU simultaneously.

What is the advantage of multi-core processors over single-core processors?

Multi-core processors have multiple independent processing units within a single CPU chip, allowing for parallel processing of multiple instruction streams and improved performance for multi-threaded applications.

Why are supercomputers necessary for certain types of calculations?

Supercomputers are necessary for performing extremely large and complex calculations, such as simulating the formation of the universe, which require a massive amount of processing power beyond what is available in standard desktop or server CPUs.

Outlines

00:00

🚀 Evolution of Computer Processors and Performance Enhancement Techniques

Carrie Anne introduces the progress in computer processors, from early mechanical devices to modern CPUs operating at Gigahertz speeds. The video discusses the limitations of increasing transistor speed and the development of various techniques to improve performance, such as specialized circuits for graphics, video decoding, and encryption. It also covers the concept of instruction set extensions like MMX, 3DNow!, and SSE, and the continuous growth of these sets for backward compatibility. The issue of data transfer speed between CPU and RAM is highlighted, along with the introduction of caches to mitigate this bottleneck, explaining how caches work, the concept of cache hits and misses, and the use of dirty bits for synchronization.

05:04

🔄 Advanced CPU Techniques: Pipelining, Caching, and Multi-Core Processing

This paragraph delves into instruction pipelining as a method to increase CPU performance, comparing it to washing laundry to illustrate the concept of parallelizing tasks. The explanation includes the benefits of pipelining, such as increased throughput, and the challenges it presents, like handling data dependencies and jump instructions. The paragraph also touches on advanced techniques like out-of-order execution and speculative execution with branch prediction to minimize pipeline stalls. The discussion then shifts to superscalar processors capable of executing multiple instructions per clock cycle and the concept of multi-core processors, which allow for parallel processing of multiple instruction streams, concluding with the mention of multi-CPU systems in high-performance computers.

10:06

🌐 Scaling Up: From Multi-Core Processors to Supercomputers

The final paragraph discusses the escalation from multi-core processors to the construction of supercomputers for massive computational tasks. It explains the necessity of supercomputers for complex calculations like simulating the universe's formation and introduces the Sunway TaihuLight, a supercomputer with over ten million cores capable of processing 93 quadrillion floating-point operations per second. The paragraph emphasizes the sophistication and speed of modern processors and sets the stage for the next episode, which will focus on programming and utilizing this computational power.

Keywords

💡Gigahertz

Gigahertz refers to a unit of frequency equal to one billion cycles per second. In the context of the video, it is used to describe the speed at which modern CPUs operate, executing billions of instructions every second. The script mentions that the device viewers are using is likely running at Gigahertz speeds, emphasizing the significant advancement in computing power over time.

💡Transistors

Transistors are semiconductor devices that act as switches or amplifiers and are fundamental components of modern electronic devices. The video script discusses how improving the switching time of transistors within a chip was one method of increasing processor speed in the early days of electronic computing, highlighting their importance in the evolution of computer processors.

💡ALU (Arithmetic Logic Unit)

An Arithmetic Logic Unit is a part of the CPU that performs arithmetic and logical operations. The script explains that modern processors have the division operation performed by the ALU in hardware, which is a more efficient approach than the earlier method of using many subtractions to achieve division, as demonstrated in the video's example.

💡Instruction Set

An instruction set is the collection of basic instructions that a processor can understand and execute. The video script notes the growth of instruction sets over time, with modern processors having thousands of instructions, as opposed to the 46 instructions of the Intel 4004, to perform a wide variety of complex operations.

💡Cache

A cache is a small, fast memory storage used to temporarily store frequently accessed data to reduce access time from main memory (RAM). The script describes the use of a cache in CPUs to speed up data retrieval, explaining how a cache hit is more efficient than a cache miss, which requires data to be fetched from RAM.

💡Pipelining

Pipelining is a method of improving performance by allowing multiple instructions to be processed at different stages of the execution pipeline simultaneously. The video script uses the analogy of washing sheets to explain how pipelining can increase throughput, allowing an instruction to be executed every clock cycle rather than every few.

💡Superscalar Processors

Superscalar processors are CPUs capable of executing more than one instruction per clock cycle. The script mentions superscalar processors as a further advancement in CPU design, allowing for even greater efficiency by utilizing different parts of the CPU for parallel execution of instructions.

💡Multi-core Processors

Multi-core processors contain multiple independent processing units (cores) within a single CPU. The video script discusses how multi-core processors can handle several streams of instructions at once, providing an example of dual-core or quad-core processors and their ability to share resources like cache.

💡Supercomputer

A supercomputer is a computer with a high level of performance compared to a general-purpose computer. The script refers to supercomputers as machines designed for massive computational tasks, such as simulating the universe's formation, and mentions the Sunway TaihuLight as an example of a supercomputer with over ten million cores.

💡FLOPS (Floating Point Operations Per Second)

FLOPS is a measure of a computer's performance, indicating the number of floating-point operations it can execute in one second. The video script uses FLOPS to quantify the immense computational power of the Sunway TaihuLight supercomputer, which can process 93 quadrillion FLOPS.

💡Branch Prediction

Branch prediction is a technique used in pipelined processors to guess the direction a program will take at a conditional branch instruction. The script describes how advanced CPUs use branch prediction to minimize pipeline stalls, often guessing with over 90% accuracy, which helps maintain high execution efficiency.

Highlights

Computers have evolved from mechanical devices to CPUs running at Gigahertz speeds, executing billions of instructions per second.

Processors were traditionally made faster by improving the switching time of transistors.

Processor designers have developed techniques to boost performance beyond transistor efficiency.

Modern CPUs include hardware support for complex operations like division to reduce clock cycles.

ALUs have become more complex to perform additional operations like graphics and encryption.

Instruction sets have grown larger over time, retaining old opcodes for backward compatibility.

High-speed CPUs face bottlenecks with RAM due to data transmission delays.

Caching is used to mitigate RAM bottlenecks by storing data closer to the CPU.

Caches work by transmitting blocks of data, reducing the need for repeated RAM access.

Cache hits and misses are key concepts in CPU performance optimization.

The dirty bit in caches helps manage synchronization between cache and RAM.

Instruction pipelining allows for overlapping stages of instruction processing, increasing throughput.

Pipeline hazards, such as data dependencies, can be mitigated with advanced processor techniques.

Out-of-order execution and speculative execution are methods used to minimize pipeline stalls.

Branch prediction improves the efficiency of handling conditional jump instructions.

Superscalar processors can execute multiple instructions per clock cycle, further increasing performance.

Multi-core processors allow for multiple independent streams of instructions to run simultaneously.

Supercomputers, like the Sunway TaihuLight, utilize millions of cores for massive computational tasks.

Programming harnesses the power of sophisticated processors to perform useful computations.

Transcripts

Hi, I’m Carrie Anne and welcome to CrashCourse Computer Science!

As we’ve discussed throughout the series, computers have come a long way from mechanical

devices capable of maybe one calculation per second, to CPUs running at kilohertz and megahertz speeds.

The device you’re watching this video on right now is almost certainly running at Gigahertz

speeds - that’s billions of instructions executed every second.

Which, trust me, is a lot of computation!

In the early days of electronic computing, processors were typically made faster by improving

the switching time of the transistors inside the chip - the ones that make up all the logic

gates, ALUs and other stuff we’ve talked about over the past few episodes.

But just making transistors faster and more efficient only went so far, so processor designers

have developed various techniques to boost performance allowing not only simple instructions

to run fast, but also performing much more sophisticated operations.

INTRO

Last episode, we created a small program for our CPU that allowed us to divide two numbers.

We did this by doing many subtractions in a row... so, for example, 16 divided by 4

could be broken down into the smaller problem of 16 minus 4, minus 4, minus 4, minus 4.

When we hit zero, or a negative number, we knew that we we’re done.

But this approach gobbles up a lot of clock cycles, and isn’t particularly efficient.

So most computer processors today have divide as one of the instructions that the ALU can

perform in hardware.

Of course, this extra circuitry makes the ALU bigger and more complicated to design,

but also more capable - a complexity-for-speed tradeoff that has been made many times in

computing history.

For instance, modern computer processors now have special circuits for things like graphics

operations, decoding compressed video, and encrypting files - all of which are operations

that would take many many many clock cycles to perform with standard operations.

You may have even heard of processors with MMX, 3DNow!, or SSE.

These are processors with additional, fancy circuits that allow them to execute additional,

fancy instructions - for things like gaming and encryption.

These extensions to the instruction set have grown, and grown over time, and once people

have written programs to take advantage of them, it’s hard to remove them.

So instruction sets tend to keep getting larger and larger keeping all the old opcodes around

for backwards compatibility.

The Intel 4004, the first truly integrated CPU, had 46 instructions - which was enough

to build a fully functional computer.

But a modern computer processor has thousands of different instructions, which utilize all

sorts of clever and complex internal circuitry.

Now, high clock speeds and fancy instruction sets lead to another problem - getting data

in and out of the CPU quickly enough.

It’s like having a powerful steam locomotive, but no way to shovel in coal fast enough.

In this case, the bottleneck is RAM.

RAM is typically a memory module that lies outside the CPU.

This means that data has to be transmitted to and from RAM along sets of data wires,

called a bus.

This bus might only be a few centimeters long, and remember those electrical signals are

traveling near the speed of light, but when you are operating at gigahertz speeds – that’s

billionths of a second – even this small delay starts to become problematic.

It also takes time for RAM itself to lookup the address, retrieve the data, and configure

itself for output.

So a “load from RAM” instruction might take dozens of clock cycles to complete, and during

this time the processor is just sitting there idly waiting for the data.

One solution is to put a little piece of RAM right on the CPU -- called a cache.

There isn’t a lot of space on a processor’s chip, so most caches are just kilobytes or

maybe megabytes in size, where RAM is usually gigabytes.

Having a cache speeds things up in a clever way.

When the CPU requests a memory location from RAM, the RAM can transmit not just one single

value, but a whole block of data.

This takes only a little bit more time than transmitting a single value, but it allows

this data block to be saved into the cache.

This tends to be really useful because computer data is often arranged and processed sequentially.

For example, let say the processor is totalling up daily sales for a restaurant.

It starts by fetching the first transaction from RAM at memory location 100.

The RAM, instead of sending back just that one value, sends a block of data, from memory

location 100 through 200, which are then all copied into the cache.

Now, when the processor requests the next transaction to add to its running total, the

value at address 101, the cache will say “Oh, I’ve already got that value right here,

so I can give it to you right away!”

And there’s no need to go all the way to RAM.

Because the cache is so close to the processor, it can typically provide the data in a single

clock cycle -- no waiting required.

This speeds things up tremendously over having to go back and forth to RAM every single time.

When data requested in RAM is already stored in the cache like this it’s called a cache

hit,

and if the data requested isn’t in the cache, so you have to go to RAM, it’s a called

a cache miss.

The cache can also be used like a scratch space, storing intermediate values when performing

a longer, or more complicated calculation.

Continuing our restaurant example, let’s say the processor has finished totalling up

all of the sales for the day, and wants to store the result in memory address 150.

Like before, instead of going back all the way to RAM to save that value, it can be stored

in cached copy, which is faster to save to, and also faster to access later if more calculations

are needed.

But this introduces an interesting problem -- the cache’s copy of the data is now different

to the real version stored in RAM.

This mismatch has to be recorded, so that at some point everything can get synced up.

For this purpose, the cache has a special flag for each block of memory it stores, called

the dirty bit -- which might just be the best term computer scientists have ever invented.

Most often this synchronization happens when the cache is full, but a new block of memory

is being requested by the processor.

Before the cache erases the old block to free up space, it checks its dirty bit, and if

it’s dirty, the old block of data is written back to RAM before loading in the new block.

Another trick to boost cpu performance is called instruction pipelining.

Imagine you have to wash an entire hotel’s worth of sheets, but you’ve only got one

washing machine and one dryer.

One option is to do it all sequentially: put a batch of sheets in the washer and wait 30

minutes for it to finish.

Then take the wet sheets out and put them in the dryer and wait another 30 minutes for

that to finish.

This allows you to do one batch of sheets every hour.

Side note: if you have a dryer that can dry a load of laundry in 30 minutes, please tell

me the brand and model in the comments, because I’m living with 90 minute dry times, minimum.

But, even with this magic clothes dryer, you can speed things up even more if you parallelize

your operation.

As before, you start off putting one batch of sheets in the washer.

You wait 30 minutes for it to finish.

Then you take the wet sheets out and put them in the dryer.

But this time, instead of just waiting 30 minutes for the dryer to finish, you simultaneously

start another load in the washing machine.

Now you’ve got both machines going at once.

Wait 30 minutes, and one batch is now done, one batch is half done, and another is ready

to go in.

This effectively doubles your throughput.

Processor designs can apply the same idea.

In episode 7, our example processor performed the fetch-decode-execute cycle sequentially

and in a continuous loop: Fetch-decode-execute, fetch-decode-execute, fetch-decode-execute,

and so on.

This meant our design required three clock cycles to execute one instruction.

But each of these stages uses a different part of the CPU, meaning there is an opportunity

to parallelize!

While one instruction is getting executed, the next instruction could be getting decoded,

and the instruction beyond that fetched from memory.

All of these separate processes can overlap so that all parts of the CPU are active at

any given time.

In this pipelined design, an instruction is executed every single clock cycle which triples

the throughput.

But just like with caching this can lead to some tricky problems.

A big hazard is a dependency in the instructions.

For example, you might fetch something that the currently executing instruction is just

about to modify, which means you’ll end up with the old value in the pipeline.

To compensate for this, pipelined processors have to look ahead for data dependencies,

and if necessary, stall their pipelines to avoid problems.

High end processors, like those found in laptops and smartphones, go one step further and can

dynamically reorder instructions with dependencies in order to minimize stalls and keep the pipeline

moving, which is called out-of-order execution.

As you might imagine, the circuits that figure this all out are incredibly complicated.

Nonetheless, pipelining is tremendously effective and almost all processors implement it today.

Another big hazard are conditional jump instructions -- we talked about one example, a JUMP NEGATIVE,

last episode.

These instructions can change the execution flow of a program depending on a value.

A simple pipelined processor will perform a long stall when it sees a jump instruction,

waiting for the value to be finalized.

Only once the jump outcome is known, does the processor start refilling its pipeline.

But, this can produce long delays, so high-end processors have some tricks to deal with this

problem too.

Imagine an upcoming jump instruction as a fork in a road - a branch.

Advanced CPUs guess which way they are going to go, and start filling their pipeline with

instructions based off that guess – a technique called speculative execution.

When the jump instruction is finally resolved, if the CPU guessed correctly, then the pipeline

is already full of the correct instructions and it can motor along without delay.

However, if the CPU guessed wrong, it has to discard all its speculative results and

perform a pipeline flush - sort of like when you miss a turn and have to do a u-turn to

get back on route, and stop your GPS’s insistent shouting.

To minimize the effects of these flushes, CPU manufacturers have developed sophisticated

ways to guess which way branches will go, called branch prediction.

Instead of being a 50/50 guess, today’s processors can often guess with over 90% accuracy!

In an ideal case, pipelining lets you complete one instruction every single clock cycle,

but then superscalar processors came along which can execute more than one instruction

per clock cycle.

During the execute phase even in a pipelined design, whole areas of the processor might

be totally idle.

For example, while executing an instruction that fetches a value from memory, the ALU

is just going to be sitting there, not doing a thing.

So why not fetch-and-decode several instructions at once, and whenever possible, execute instructions

that require different parts of the CPU all at the same time!?

But we can take this one step further and add duplicate circuitry

for popular instructions.

For example, many processors will have four, eight or more identical ALUs, so they can

execute many mathematical instructions all in parallel!

Ok, the techniques we’ve discussed so far primarily optimize the execution throughput

of a single stream of instructions, but another way to increase performance is to run several

streams of instructions at once with multi-core processors.

You might have heard of dual core or quad core processors.

This means there are multiple independent processing units inside of a single CPU chip.

In many ways, this is very much like having multiple separate CPUs, but because they’re

tightly integrated, they can share some resources, like cache, allowing the cores to work together

on shared computations.

But, when more cores just isn’t enough, you can build computers with multiple independent

CPUs!

High end computers, like the servers streaming this video from YouTube’s datacenter, often

need the extra horsepower to keep it silky smooth for the hundreds of people watching

simultaneously.

Two- and four-processor configuration are the most common right now, but every now and

again even that much processing power isn’t enough.

So we humans get extra ambitious and build ourselves a supercomputer!

If you’re looking to do some really monster calculations – like simulating the formation

of the universe - you’ll need some pretty serious compute power.

A few extra processors in a desktop computer just isn’t going to cut it.

You’re going to need a lot of processors.

No.. no... even more than that.

A lot more!

When this video was made, the world’s fastest computer was located in The National Supercomputing

Center in Wuxi, China.

The Sunway TaihuLight contains a brain-melting 40,960 CPUs, each with 256 cores!

Thats over ten million cores in total... and each one of those cores runs at 1.45 gigahertz.

In total, this machine can process 93 Quadrillion -- that’s 93 million-billions -- floating

point math operations per second, knows as FLOPS.

And trust me, that’s a lot of FLOPS!!

No word on whether it can run Crysis at max settings, but I suspect it might.

So long story short, not only have computer processors gotten a lot faster over the years,

but also a lot more sophisticated, employing all sorts of clever tricks to squeeze out

more and more computation per clock cycle.

Our job is to wield that incredible processing power to do cool and useful things.

That’s the essence of programming, which we’ll start discussing next episode.

See you next week.