Multi-core Programming
Multi-core CPUs have become a norm. While dealing with HPC, Deep Learning and high frequency data, we continuously use libraries which help us in unlocking the power of these chips. To give you a picture, AMD Ryzen™ Threadripper™ 3990X Processor has 64 CPU cores!
The following are my notes from Multi-core Programming lecture offered by MIT OCW
With multi-core processors, each core can have dedicated L1 and L2 cache, but they can share an L3 cache.
But for a very long time we were having single core processors the why semi conductor vendors started to provide us with multi-core chips?
The reason is that Moore’s Law and the end of the scaling of clock frequency.
As we can see that although we are able to pack more Transistors cheaply on a chip but clock speed has consolidated, it is upper bounded at 4 GHz. This is why we needed more cores if we wanted to perform more complicated tasks in less time.
Cache Coherence —
Getting the value from other processor’s cache is sometimes cheaper than getting the value from main memory. However, Let’s say one processor updates the value of x in its cache. Now How do we make sure that the cache remains coherent across during updates? How do we make sure that all the processors have same value of x in their cache during updates?
MSI Protocol —
Each cache line is labelled with a state:
- M — Cache has been modified. No other cache contains this block in M or S states.
- S — Other caches may be sharing this block.
- I — Cache block is invalid (same as not there).
Before a cache modifies a location, the hardware first invalidates all other copies.
Concurrency Platforms —
Programming directly on processor cores is painful and error prone. A Concurrency platform abstracts processor cores, handles synchronisation and communication protocols, and performs load balancing. A few examples are OpenMP, Intel Threading Building Blocks(ITBB), Numba etc.
Read my other works —
- https://vaibhaw-vipul.medium.com/matrix-multiplication-optimizing-the-code-from-6-hours-to-1-sec-70889d33dcfa
- https://medium.com/@vaibhaw.vipul/writing-synthetic-data-generator-using-opencv-e2658a63f195
- https://medium.com/@vaibhaw.vipul/building-autoencoder-in-pytorch-34052d1d280c
I hope that you liked the article! Keep Learning and sharing knowledge. Follow me on github, linkedin or twitter.