Fundamental design principles

Intuition

Nobody has found a way to design software that systematically excludes every security flaw. What the field has instead is a set of principles, tested across decades of systems that failed and systems that held, for how to build protection mechanisms that fail in smaller, safer ways when they do fail. None of them guarantee security on their own — each closes off one specific way things tend to go wrong.

Mechanism

Week 1 lists thirteen of these principles. Each is stated below with the lecture’s own example.

1. Economy of mechanism

Security measures in hardware and software should be as simple and small as possible. A simpler, smaller design is easier to test and verify thoroughly; a complex one gives an adversary more opportunities to find subtle weaknesses that are hard to spot ahead of time.

Example: UNIX was designed modularly with a small codebase, and that design philosophy carried forward into Linux and macOS. Windows Vista, by contrast, shipped with many more complex features and a much larger codebase than its predecessors — the larger attack surface gave adversaries more places to look for exploitable vulnerabilities.

2. Fail-safe defaults

Access decisions should be based on permission rather than exclusion: the default is no access, and the protection scheme states explicitly the conditions under which access is granted. This fails more safely than the alternative, where the default is to permit access.

Example: Operating system file permissions default to denying access, requiring explicit grants — so an error in the rules denies rather than leaks access. The POODLE attack is the counter-example: it exploited a fallback mechanism in SSL/TLS that downgraded a connection to the older, weaker SSL 3.0 whenever negotiation for a stronger protocol failed. Fail-safe defaults says that failed negotiation should mean no connection, not a weaker one.

Threat

POODLE forces an SSL/TLS connection to downgrade to SSL 3.0 by making negotiation for a stronger protocol fail, then exploits known weaknesses in that older protocol.

Control

Fail-safe defaults says the correct response to failed negotiation is to refuse the connection, not silently accept a weaker one. A protocol that fails open — permitting a downgrade by default — is the opposite of this principle.

3. Complete mediation

Every access must be checked against the access control mechanism — not served from a cached decision. In a system that runs continuously, this means considering carefully how a change in authority propagates into any local memory of past decisions. Ordinary file systems do not fully satisfy this: once a file is open, most systems do not recheck permissions on every subsequent read. Fully implementing complete mediation, checking access on every single field or record, is resource-intensive and rarely done in practice.

Example: Zero Trust Networks validate every access request against policy and access controls regardless of the user’s prior authentication status — no trust is granted purely because of network location or an earlier login.

4. Open design

A security mechanism’s design should be open to scrutiny rather than kept secret. Encryption keys must stay secret, but encryption algorithms should be open, so experts can review them and users can have confidence in that review. This is the philosophy behind NIST’s standardisation of encryption and hash algorithms.

Example: The Content Scramble System used on DVDs kept its algorithm secret. That secrecy prevented expert review, and in November 1999 Frank A. Stevenson published exploits that made the cipher practically ineffective; a separate group then reverse-engineered CSS into the DeCSS program. Secrecy did not make the design more secure — it only delayed discovery of how weak it already was.

Aside

Kerckhoffs’s principle is the cryptographic version of the same idea: a cryptographic system should remain secure even if everything about it except the key is public knowledge. Security should rest on the secrecy of keys, not on the obscurity of the algorithm.

5. Separation of privilege

Achieving access to a restricted resource should require multiple independent privilege attributes, not one. Multi-factor authentication — a password and a smart card, say — is the common example; the term also covers splitting a program so that a high-privilege operation runs in its own process, separate from the lower-privilege process handling day-to-day interaction.

Worked example

AnswerMeeting either condition alone is not enough — both are required, together.

1. On Berkeley-derived UNIX systems, switching to the root account requires two separate conditions.
2. Condition one: the user must know the root password.
3. Condition two: the user must be a member of the wheel group (the group with GID 0).
4. Knowing the password without wheel membership is not sufficient to gain root access.
5. Being in the wheel group without the password is equally insufficient.
6. Only satisfying both conditions together grants root access — that is separation of privilege in practice.

Pitfall

Separation of privilege and least privilege (below) get confused because both restrict access. Separation of privilege is about how many independent things are needed before an action can happen at all — the wheel group example needs two. Least privilege is about how much any single actor holds once it is authorised. A system can enforce one without the other.

6. Least privilege

Every process and every user should operate with the least set of privileges necessary to perform its task. Role-based access control is the standard mechanism: each role gets only the permissions its function needs, and nothing is granted beyond an explicit permission. There is a temporal side to this too — administrators should hold elevated privileges only while they need them, not permanently, since leaving them in place invites accidents even without malice.

Example: Apache’s web server runs as the www-data account, which can read files from the web directory and write to specific log files, and nothing more — no administrative rights, no access to the rest of the operating system. If the web server process is compromised, the damage is contained to what www-data can reach.

7. Least common mechanism

Minimise the functions shared between different users, so the amount of shared hardware and software everyone depends on stays small. Fewer shared paths means fewer unintended communication channels between users, and it becomes easier to check what remains for undesirable security implications.

Example: Passwords must not be reused between different systems or services — the one habit the lecture gives as its example of violating this principle when broken.

8. Psychological acceptability

Security mechanisms should not interfere unduly with users’ work while still meeting the needs of whoever authorises access. Mechanisms that are not transparent, or that force a user to translate their own mental model of protection into something unfamiliar, tend to get worked around or turned off.

Example: Fingerprint or facial recognition for unlocking a device gives strong security with almost no friction, matching how users already expect to identify themselves.

9. Isolation

Public-access systems should be isolated from critical resources, physically or logically, to prevent disclosure or tampering. The same idea applies to individual users on the same system: their processes, memory and files should stay separate from one another except where sharing is explicitly wanted, and every modern operating system provides this separation.

Example: Apple’s Secure Enclave is a coprocessor, separate from the main processor, that handles encryption, decryption and key management on its own. Even if the main operating system is compromised, keys and cryptographic operations inside the Secure Enclave — including the biometric data behind Face ID and Touch ID — stay isolated from that compromise.

10. Encapsulation

A specific form of isolation built on object-oriented structure: a collection of procedures and data is enclosed in its own domain, so the internal structure of a data object is reachable only through the procedures of that protected subsystem, entered only at designated points.

Example: A healthcare database restricts direct table access. A stored procedure GetPatientRecord returns records matching specific criteria, and a view PublicPatientInfo exposes only non-sensitive fields like name and date of birth. Every interaction with sensitive data goes through one of these defined entry points, which keeps access control, auditing and data integrity enforceable.

11. Modularity

Security functions should be built as separate, protected modules, and the wider architecture should itself be modular, so any one part can migrate to new technology or gain new features without redesigning the whole system.

Example: AWS Key Management Service is a single, centrally managed cryptographic module used by many other AWS services — S3, EBS, RDS, Lambda — instead of each service implementing its own encryption. AWS can upgrade KMS’s cryptographic capability without touching the services that depend on it.

12. Layering

Multiple, overlapping protection approaches covering the people, technology and operational sides of a system together, so the failure or bypass of any single approach does not leave the system unprotected. This is also called defence in depth.

Example: A company might require multi-factor authentication to log in, enforce role-based access control, keep the OS patched, run antivirus software, filter traffic with network and host firewalls, encrypt sensitive data, and train staff to recognise phishing. If a phishing attack steals a password, the attacker still has to get past MFA, access control and antivirus before the OS itself is compromised — no single layer failing is enough on its own.

Aside

The slide deck’s own definition of this principle did not survive extraction from the source PDF — the text is interleaved and garbled where it should read cleanly. It is not reconstructed here. The definition and example above come from the lecture notes, which give this principle in full elsewhere in the same week’s material.

13. Least astonishment

A program or interface should always respond in the way least likely to astonish the user — the mechanism for authorisation should be transparent enough that the user has a good intuitive sense of how the security goal maps to what the system actually does.

Example: A standard library integer-parsing function typically defaults to base 10, only interpreting a string as binary or octal when told to explicitly. JavaScript originally broke this convention: a string beginning with "0" was parsed as base 8 by default, which caused real developer confusion and bugs. The behaviour was discouraged in ECMAScript 3 and dropped by ECMAScript 5.

Exam detail

When a question describes a scenario and asks which principle it illustrates, the fastest check is what specifically failed: one actor with too much unilateral power points at separation of privilege; a process or account holding more access than its task needs points at least privilege; a system defaulting to open access points at fail-safe defaults; and a design that was kept secret and then broken once examined points at open design.

Recall

A company's backup admin keeps full production database access permanently, even though they only need it during the monthly backup window. Which principle is being violated, and how would you fix it?

Least privilege — specifically its temporal aspect. The fix is withdrawing that access outside the backup window rather than leaving it in place at all times: privileges held longer than the task requires them create risk even without any malicious intent behind the access itself.