CSEC3616Cybersecurity Engineering

    Human error and cognitive bias

    The three categories of human error the lecture borrows from safety research, and every named cognitive bias that pulls people away from rational decisions about security.

    • Name the three categories of human error the lecture borrows from safety research, each with its own example.
    • Identify which named cognitive bias a given scam or scenario is exploiting.
    • Distinguish rational-choice and expected-utility reasoning from the biases that override it in practice.

    18 min read

    Intuition

    People are not naturally good statisticians, and an attacker does not need to break a cryptographic system when a predictable slip in judgement gets the same result for free. The lecture opens with a figure — 80 to 90% of security breaches caused by human error — that varies a great deal by source and methodology but sits in the right neighbourhood of every industry estimate checked against it. Two things drive that number: predictable failures in the mechanics of doing a task, and predictable biases in how people weigh risk, reward and effort.

    Mechanism

    Human error, in any equipment-operating task, falls into three broad categories the lecture borrows from safety research:

    • Manual skill fails. A skill performed often enough becomes automatic, and automatic skills slip under distraction or fatigue. Typosquatters exploit exactly this — registering a domain one keystroke from the real one and waiting for a typing slip to send the victim there instead: github.comglthub.com, google.comgougle.com, amazon.comamozon.com, netflix.comnetfllix.com, linkedin.comlinkedin.co.
    • Following the wrong rule. A rule that is normally safe gets applied somewhere it should not be. “Trust HTTPS because it is secure” is the security version — a padlock icon proves the connection is encrypted, not that the site behind it is trustworthy, and attackers can obtain certificates too.
    • Cognitive reasons. The person does not understand the problem, pretends to, or understands it and ignores it to get the job done — an employee who finds new update procedures cumbersome and keeps using the old ones, or postpones a patch because installing it now interrupts their work.

    These are failures at three different levels — of skill, of rule, and of understanding or intent — and an attacker designs around whichever one a target is most likely to make.

    Exam detail

    Rational choice theory says people select whichever option maximises their own satisfaction — a shopper comparing cereal brands on price, taste and nutrition. Expected utility theory narrows this to decisions made under risk and uncertainty, where the outcome is not guaranteed: an investor weighing several assets considers both the potential return and the probability of getting it, and picks whichever combination maximises expected utility. Both theories assume people compute this correctly. Kahneman and Tversky’s prospect theory showed they mostly do not.

    Exam detail

    Loss aversion. Losses are felt more intensely than an equivalent gain. A message reading “Your PayPal account has been frozen, and you need to click here to unlock it” exploits exactly this — the threatened loss of account access pulls harder than any equivalent framing of a gain would.

    Exam detail

    Framing effect. The same offer, presented as avoiding a loss rather than securing a gain, produces a stronger reaction. “Don’t miss out, sale ends soon!” outperforms “Get it now on sale!” for an identical deal — only the frame changed.

    Exam detail

    Anchoring effect. A judgement is built on the first number seen and adjusted only slightly from there. A car marked $30,000 looks expensive on its own; the same car marked “was $60,000, now $30,000” gets bought, because the buyer anchors on the size of the discount rather than the final price. Fake antivirus pop-ups use the same trick — “$1,000 per year, we are giving it away for $500” — and so do lottery scams, where the size of the promised jackpot makes a small “processing fee” look negligible by comparison.

    Exam detail

    Availability heuristic. Information that comes to mind easily gets treated as the best guide to what is likely, regardless of the real statistics. Someone who watched a shark-attack documentary the night before hesitates to swim the next morning — the vivid recollection outweighs the actual, very low, risk. “Spray and pray” SMS scams around tax time exploit the same effect, arriving exactly when a recent tax return is fresh in the recipient’s memory so an ATO-themed message feels immediately plausible.

    Exam detail

    Present bias and hyperbolic discounting. People weight an immediate cost far more heavily than a larger, merely delayed one. Postponing a software update because installing it now is inconvenient, even though leaving it unpatched carries a bigger long-term risk, is present bias in action. Hyperbolic discounting is the model decision scientists use to quantify it: the discount applied to future costs grows so steep that avoiding the present interruption dominates the choice, whatever the deferred cost turns out to be.

    The privacy paradox is the same effect applied to privacy specifically: people say they care about it, then click “Accept” on a cookie banner, skip the privacy policy, and leave default sharing settings alone, because the immediate cost of reading and deciding outweighs a data-exposure risk that is diffuse and delayed.

    Compare

    Seeing a pattern in data that is genuinely random. The “hot hands” belief that a basketball player who just scored is due to score again, and the gambler’s fallacy that a coin “owes” tails after several heads, are both patterns the mind supplies where none exists. A phishing email’s fake logo and familiar layout register as “looks right” the same way — the mind is pattern-matching, not verifying.

    Filtering new evidence to fit a belief already held, and dismissing what contradicts it. Once a reader has decided a PayPal email is genuine, small discrepancies in the URL or the logo get overlooked — not because they pattern-match as correct, but because the reader has already stopped checking.

    Exam detail

    Zero-risk bias. People prefer eliminating one risk completely over reducing total risk by a larger amount, because certainty feels better than a bigger but merely probabilistic improvement. Asked to rank hazard-site cleanup plans, participants preferred a plan that cleaned one site perfectly while leaving others untouched over a plan that reduced risk more overall across every site. The same bias shows up in security engineering: insisting on an information-theoretically perfect protocol for every channel, including low-risk internal ones, when a “very good but not perfect” alternative would cover far more ground for far less cost.

    Threat

    Anchoring, loss aversion and the availability heuristic are not abstract — attackers build scams around them directly. A lottery scam anchors the victim on a large jackpot so a small “processing fee” looks negligible. A fake account-freeze notice exploits loss aversion by threatening to take something away rather than offering a gain. An SMS scam timed to tax season exploits the availability heuristic by arriving exactly when a real tax return is fresh in memory.

    Control

    None of these biases are unique to careless people — they are how human judgement works under uncertainty, for everyone. The lecture’s response is not “train people to stop being biased.” It is to design the system so the secure choice does not depend on overriding the bias in the moment — the subject of 02-03.

    Pitfall

    Section 4’s biases and Section 5’s persuasion techniques (02-03) share examples and even some names — “respect to authority” reads like a bias, but the lecture places it among Cialdini’s six techniques, not this list. If a question asks for one of the ten items above, an answer drawn from Cialdini’s six is off by one section, even though the lecture itself says the two lists are closely related.

    Recall

    A worker slips back into their old, familiar routine despite having been trained on a new, safer procedure, because the new one feels cumbersome. Which of the three categories of human error is this?

    Cognitive reasons. The worker understands the new procedure and chooses to ignore it, rather than making a skill slip or following a rule that was wrong for the situation.