Home Topic 7 Last Next

--- Distributed systems ---


Compare a centrally controlled system with a distributed system.


Teaching Note:

Technical hardware details are not expected.


Sample Question:


Marks Scheme

From Sample Paper 1 - 2014:

(Sample paper, so mark schemes:)


JSR Notes:

(Make sure to see just above, from the marking scheme of the IB sample paper 1 an "advantages/disadvantages" list for centralized vs. distributed control from , once you have a good idea of the difference between the two.)

Centralized" vs. "Distributed"

In Topic 7, note that 7.1.1 - 7.1.6 was under the title "Centralized Control Systems".
And 7.1.7 - 7.1.8 are under the title "Distributed Systems".

So now, in 7.1.7, and 7.1.8 we consider systems which are controlled in a distributed way - so we're still talking "systems", just systems whose component parts have more "self-control".

Definition from WikiAudio :

"A distributed control system (DCS) is a control system for a process or plant, wherein control elements are distributed throughout the system. This is in contrast to non-distributed, centralized systems, which use a single controller/server/mainframe at a central location. In a DCS, a hierarchy of controllers is connected by communications networks for command and monitoring."

DCS vs CCS & Autonomous Agents

The key here, compared to centralized and autonomous control systems is the hierarchical system. If there's no hierarchical system, then there's no distribution - either because one processor does all the control (CCS), or the agents do everything themselves (autonomous agents). So when it comes to DCS, you can ask yourself where are the processors and what do they do, and their should be an answer.

Example scenarios where a DCS might be used include:

A large scale Distributed Control System (DCS) will often consist of:
- Custom designed
hardware and software for both controllers and the interconnections/buses between them.
- The various controllers will receive input from instruments including a variety of bespoke (i.e. custom made) or standard analogue an digital sensors,
- which send their output to likewise standard or customized "output instruments".
- All of these control systems will be supervised and controlled by other controllers further on up the DCS hierarchy (see diagram).

A good example to keep in mind to envision an example of this is an automotive assembly line.

The Control Hierarchy

With most DCS systems, you can identify tasks which are fully centrally controlled, and others which seem to be more distributed; but those, ultimately, are indeed some way controlled via the heirarchy, all the way up to the highest level.

In the example of a modern automotive assembly line, (see this video of the Kia assembly line) most of the spot welding will be done by robots most controlled by the individual controllers of the individual robots (distributed control). But those controllers are, in turn being supervised/controlled, so that - to take one key example - if there is a major problem, the whole assembly line can be halted (centralized control). Another aspect which would have to be controlled one level above the individual robot controller is communication about when the cars are all ready to be moved on to the next step in the assembly line.

Wikimedia.org Functional_levels_of_a_Distributed_Control_System.svg.png

- Wikimedia Commons Image

Distributed Control Systems Examples

Through each of theses thorough examples, at the very least zoom in on the key points, which will be bolded and italicized.

Public Utility Examples.:

With the following examples, I won't necessarily identify separately all the centralized and distributed parts, but in all cases you would be able to find several examples of both.

Electrical power grids and electrical generation plants
- The "power grid" of a city or state or nation is the network of wires and relay stations which carry electricity to its destination. And if you stop to think about it, this is an absolutely awesome feat in modern developed cities/nations. There are literally millions and millions of kilometers of wires, which, under normal circumstances, "light" the world. The technology responsible for the generation, distribution, and "book-keeping" of electrical usage charges has become so reliable and invisible that we just take it for granted.

But there is a huge amount of control necessary to manage such a system. And though there will be certain centralized systems within the greater system, much will need to be distributed to individual generating plants, and relay stations etc. And even within those plants/stations themselves, certain control will be distributed further to specific parts and sub-systems. So sophisticated and large a system is this, that it simply isn't possible to have complete centralized control.

Here is a great video showing an ---> electrical grid control center <---- there is an impressive amount of central control from it, but mention is also made of "embedded" parts of the system, and "self-healing" (i.e. autonomous "agents") parts of the system.

Centralized Part: ...... From the video linked above, with a modern facility the more centralized control the better, and so almost all of the system is indeed controlled.

Distributed Part: ......In older systems, more of the control would be local to the individual relay station.




Water management systems
- Along with flipping the switch and expecting the lights to come on, we turn on the tap and expect water to come out. Yet this too, with water systems both supplying fresh water and properly treating waste water, can support millions of people and billions of water a day. Such a system, theoretically could be controlled by a mainframe for a city/region. But in reality, it makes sense to distribute some of the specific process control to IT systems local to individual water management plants.

Higher up the DCS hierarchy of control: A prime example would be control of the overall reservoir levels in each sub district, so that even in times of relative water shortage, the water resources are shared equally among districts. Re-direction from those with surplus to those with defecit would be an ongoing job of control systems fairly high up the heirarchy.

Direct control level tasks: In middle tiers of the water management system, there would be equivalent control of water amounts to individual streets and even down to individual buildings and houses. At the lower level of "field" control, control systems could manage water pressure at each exit point into private property.

Military Sensor networks
- A good example of this is NORAD, the North American Aerospace Defense Command (in spite of the antiquated acronym), which distributes radar detection facilities across North America; it was originally organized during the Cold War to warn against incoming Soviet missiles. Naturally there is centralized control by way of communication of warnings. But actual control of the sophisticated radars at each station is distributed to the IT infrastructure of those radar facilities.

Higher up the DCS hierarchy of control: ......

Direct control level tasks: ......

Factory Examples

Factory Assembly Line - A mainframe computer will have master control over various systems, but those systems themselves will have control distributed to them. So if you think of a car factory, each station can have its own robot, to which control has been distributed. And though each robot station requires the partially finished car to be passed onto them for work, you could imagine it possible for the robot to do its own job without being connected to the mainframe control computer at all. This is not the case, for the obvious reasons of monitoring the overall system, but what makes this a primarily distributed system is that the mainframe does not do the processing work of each robot, that work is distributed to them.

Higher up the DCS hierarchy of control: ......

Direct control level tasks: ......

Oil refining plants
- Even more complex an operation than a typical assembly line factory, there are many very sophisticated, particular processes going on inside an oil refining plant, and central control of all of those, though theoretically possible, would be a burden for even the most advanced super computer. Therefore, as in the assembly line, specific tasks and processes are controlled by servers/processes.

Higher up the DCS hierarchy of control: ......

Direct control level tasks: ......


Air Traffic Control - usually hybrid as well, with ultimate control residing with each pilot and each plane, but a centralized monitoring system, and ability to control the landing of the aircraft.

Higher up the DCS hierarchy of control: Managing and determination of flight paths/altitudes for each plane.

Direct control level task: Releasing control of the plane to have the pilots themselves land the plane, rather than have the computer land the plane in conjunction with the centralized system.

Chemical plants
- The same idea as above, with potentially even more specific processes being handled by even more specific IT.

Pharmaceutical manufacturing
- Same.


IT Examples of "Distributed Computing"

Distributed computing is not what is thought of classically as distributed control systems, but in fact they do work in a similar fashion. So this is a good place in the syllabus to take a look at them. And you'll note that several of the other IB CS websites out there certainly do focus in on them here.

BitCoin - and other digital currency systems (see the bottom of this page for my explanation of how Bitcoin works if you would like) distribute both the "mining" of the currency and the tracking of transaction on thousands of computers around the world, connected to the Internet.

So BitCoin mining is distributed, yet not autonomous, in that each computer works for itself, but eventually (if their hashes are successful!) reports back to the node on the BitCoin network to which it is connected a newly mined block to be added to the bit-chain.


, the Search for Extraterrestrial Life project in which home computers, during idle time, do the crunching of data analysis for a certain section of space, and send results back to the astronomers running the project. The idea with this, and other "cluster" computers (see next item) is that super computers are expensive, and not everyone has one in their possession. Even well healed research facilities and universities may not posses a super computer. So the processing that would be done by one super computer is distributed around many computers.

What makes this example more in tune with the hardware systems above is that there actually is sensing - a great deal of sensing, as in scanning the entire universe! - involved.



Rosetta@home - This, directly from the site:

What is Rosetta@home?

Rosetta@home needs your help to determine the 3-dimensional shapes of proteins in research that may ultimately lead to finding cures for some major human diseases. By running the Rosetta program on your computer while you don't need it you will help us speed up and extend our research in ways we couldn't possibly attempt without your help. You will also be helping our efforts at designing new proteins to fight diseases such as HIV, Malaria, Cancer, and Alzheimer's. Please join us in our efforts! Rosetta@home is not for profit.

We believe that we are getting closer to accurately predicting and designing protein structures and protein complexes, one of the holy grails of computational biology. But in order to prove this, we require an enormous amount of computing resources, an amount greater than the world's largest super computers. This is only achievable through a collective effort from volunteers like you.

For more information, click on the following links:

Follow us on Twitter: @rosettaathome



Big and Ugly Rendering Project (BURP) - This directly from Wikipedia.org: BURP "is a non-commercial distributed computing project using the BOINC framework. It is currently[when?] under development to work as a publicly distributed system for the rendering of 3D graphics.

The BURP idea

The idea of BURP is to use spare CPU cycles on participating computers around the world to render 3D images and animations submitted by the users of the BURP network - in other words to build a large shared render farm that can be freely used by those who also contribute computing power to it. The potential processing power of a system like this is enormous—theoretically the rendering speed would only be limited by available network bandwidth.

The fundamental goal of BURP is to give users access to computing power to render animations that would take an impossibly long time on a single computer. By dividing the work among hundreds of computers, an animation that takes possibly months to render in CPU time could be completed in only a few days. BURP hopes to make animations and images public as soon as they are finished so that all participants will be able to see the outcome."



Cluster Computer - For example, a Beowulf Cluster is an open source (free) way to do "cluster" computers; i.e. to connect many together to act as one big, fast computer. In fact we would like to do it with the older Mac Pro computers we have in the lab. They would be all hooked up together via Ethernet cables, attached to a switch, and through the Beowulf software be able to spread a processing intensive job out among the 16 computers.

Jobs that could be done with cluster computers could include hacking passwords (not), and BitCoin mining. But another use would be to take simulation situations like Yusuke's dossier where we want to run thousands or millions of iterations of a certain "best situation" scenario, such as scheduling mock exams, or students Week Without Walls choices, to find the one most optimal combination we can.



And one more for good measure:

BotNets that contribute to DDoS attacks (Distributed Denial of Service attacks.). With DDoS attacks, botNets
are spread out into unknowing computers, which, unknown to their users, send multiple requests to a targeted
server. If enough of these botNet send enough requests to the particular server, it will freeze or shut down.  
The botNets act as autonomous agents because they are not controlled once released, and they serve the entity
that released them by eventually shutting down the targeted server.


Pros and Cons of Distributed Computing

Pros Cons
Cost - The processing is shared out among many cheap computers, or indeed among computers belonging to other people (for ex. SETI) Performance Capabilities - Depending on the system and the requirements, might not be as powerful as the mainframe/super computer alternative, or even capable of doing the job
Individual computers can fail or go off-line and not significantly affect the overall project Security - Definitely an issue that has to be carefully considered and controlled, given shared nature of the project
Scalability - the number of computers, and so the power to be harnessed is easily increased Bandwidth Requirements - this is not an issue at all with centralized processes



"Small-scale" Examples - & NOTE THAT THESE ARE ALMOST ALL BETTER FOR THEIR CENTRALIZED (LOCAL) CONTROL ASPECTS; never-the-less they all do have a distributed aspect to them:

With all these, you would ask yourself, what part is controlled in a centralized way, and what control is distributed to the various parts?

Heating Systems - though centrally in a building like our school, where each room has no control, in many heating systems, you can have sensors/meters/dials/controllers in each room/apartment for this control to be distributed, which in many buildings it is.

Centralized: Determination of when the main boiler is on, and to what temperature the water is heated.

Distributed: Individual rooms sense the temperature and regulate the amount of heat being released.

Taxi Meters - The meter inside a "freelancer" taxi can be completely centrally controlled. Yet, with a modern taxi dispatch system there is digital, central control - AAA Taxi of Prague, for example, uses such a system which can track taxis and communicate with the passengers regarding estimated pick up time and taxi number.

Centralized Part: Tracking of cars and assigning jobs.

Distributed Part: The individual taxi calculations of charges.

Multi-Elevator System in a Large Building - Usually centrally controlled by their own sensors only.
But possibly centrally controlled for reasons of security and safety. Shut down with doors open in fire/flood situation. But individual elevators are not controlled.

Centralized Part: Intelligent determination of what elevator goes to what level is chosen and gone to by each elevator.

Distributed Part: Overload indication, doors closing on people re-opening.

Traffic Lights - Can be either centralized or distributed or bot. Synchronized lights, like on Evropska, would be an example of centralized control. but often there are camera and sensors which work per individual junction, in Prague looking at both car and tram traffic, with trams trumping cars. Even the traffic officer working the Evropska / Horomerika lights control is working in a "distributed" way.

Centralized: If "zelena vlna" ("green wave"), i.e. synchronized lights, and if camera control.

Distributed: Walker buttons, or police control of intersection.




And do refer to the St. Julians page for a good selection of examples also; mainly software examples.





BitCoin as Explained by JSR: (You don't have to know this; I just put it here for your reference.)

Hi Mark and Perry,

I finally figured out what the calculation is that you do in mining BitCoin. Taken one step at a time, it's not that hard to understand, and is quite ingenious.

The one line summary is: you perform a very sophisticated calculation on a specific string of letters trying to yield a hexadecimal number result which begins with 13 zeros - the first computer in the world to achieve this with each new string receives 50 BitCoin, and this happens several times per hour.

Fleshed out a bit, you take a "message", which is based on, among other things, the transactions that occurred with the last block's mining (so some combination of public keys of sellers and buyers, combined with dates, in all likelihood - so, a simplistic example might be PB1234-MF5678-02-14-13-etc) and add a random number (in the example, it could become PB1234-MF5678-02-14-13-etc-9876) and apply the SHA256 hashing algorithm to it (hashing is where you do any specific calculation to something which can be repeated to confirm receipt of it (and for other reasons) - I pasted the pseudocode of SHA256 below).

If you come up with a hexadecimal "hash result" which (as of February 2013) starts with 13 zeros (for example 00000000000003da6b507b256cde91e063eed9a13c8b93b57bc70a3260b3f4b0), then Bingo!, you've come up a new block mining result, and you receive (as of Feb 2013) 50 bitcoins (today 1 BTC is equal to 25.64 USD, so that's 1282 USD). The average number of times it would take to come up with that result is 1/2 * 16 to the power of 13 (since the odds of getting a 0 out of the 16 digits of hexadecimal (0-F) is 1/16, and so to get two 0s in a row is 1/16*16, and so on.)

And the other thing to note is that not only does your computer have to do, on average, an enormous number of calculations to get a result, the calculation itself is enormously time consuming compared to most calculations computers do (again, see the pseudocode below). 

Never-the-less, through the network of bitcoin mining nodes around the world, the "Bingo"s happen several times per hour, based on, currently, tens to hundreds of transactions per mining block. What people like us do to hedge their bets is pool their processing with others to increase the likelihood of "scoring".

Interestingly, and importantly, the integrity of the system - i.e. what prevents it from being counterfeited - comes from the fact that one of the things used in each new block's calculation is the last hash result in the chain!  221,089 blocks have so far been mined since the start on January 1st, 2009, and enough blocks will be mined until there are exactly, and no more than, 21,000,000 bitcoins in existence. But that's moving into economics, and out of my area of competence!  So, on to you, Perry!! ;)

And here's the link to the Wikipedia page that includes the algorithm.


And for good measure:

How Bitcoin Works under the Hood (22 minute YouTube video (I found this the most useful))

Bitcoin - Overview (11 minute (but everything before 1:42 is useless) YouTube video by Khan Academy)