HTTP Security Headers Explained: CSP, Cookies, and Robots

Modern web applications are only as secure as the HTTP response headers they send. A handful of well-configured headers can prevent entire classes of attacks — XSS, clickjacking, CSRF, and unintentional indexing — without changing a single line of application code.

Content-Security-Policy (CSP)

Content-Security-Policy is the most powerful security header available. It instructs the browser to only execute scripts, load images, or apply stylesheets from origins you explicitly allow:

Content-Security-Policy: default-src 'self'; script-src 'self' 'nonce-abc123'; object-src 'none'; frame-ancestors 'none'

Key directives:

Directive	What it controls
`default-src`	Fallback for any fetch directive not explicitly set
`script-src`	JavaScript sources — use `'nonce-<random>'` for inline scripts
`style-src`	CSS sources
`object-src 'none'`	Disables Flash/plugins (always set this)
`frame-ancestors`	Who can embed your page in an `<iframe>`

Start with Report-Only mode. Set Content-Security-Policy-Report-Only first and point report-uri at a logging endpoint. You'll see violations without breaking anything. Tighten the policy until violations stop, then switch to Content-Security-Policy.

Use the CSP Generator to build your policy directive by directive.

Every session cookie should carry three flags:

Set-Cookie: session=abc123; Path=/; HttpOnly; Secure; SameSite=Lax

Flag	Effect
`HttpOnly`	Prevents JavaScript (`document.cookie`) from reading the cookie — mitigates XSS session hijacking
`Secure`	Only sent over HTTPS — prevents cookie theft on insecure networks
`SameSite=Lax`	Sent on top-level navigation GET requests but not cross-site AJAX — prevents most CSRF
`SameSite=Strict`	Never sent cross-site — use for sensitive admin cookies
`SameSite=None; Secure`	Allows cross-site sending (required for third-party embeds)

Warning: SameSite=None requires Secure — browsers silently drop cookies that set None without Secure.

Use the HTTP Cookie Builder to configure and copy the correct header.

X-Robots-Tag and meta robots

The robots meta tag and its HTTP equivalent X-Robots-Tag tell crawlers how to handle a page. They differ from robots.txt in an important way: robots.txt controls whether a page is crawled, while robots meta tags control whether it is indexed after crawling.

<!-- Prevent indexing and caching of a page -->
<meta name="robots" content="noindex, noarchive">

Equivalent HTTP header (useful for non-HTML responses like PDFs):

X-Robots-Tag: noindex, noarchive

Common values: noindex removes the page from search results; nofollow prevents link equity passing to outbound links; nosnippet suppresses the snippet in search results.

Use the Robots Meta Tag Generator to build tags for specific crawlers.

Other important headers

Header	Recommended value	Prevents
`X-Content-Type-Options`	`nosniff`	MIME-type sniffing
`X-Frame-Options`	`DENY` (superseded by `frame-ancestors`)	Clickjacking
`Strict-Transport-Security`	`max-age=63072000; includeSubDomains`	Protocol downgrade attacks
`Referrer-Policy`	`strict-origin-when-cross-origin`	Referrer leakage
`Permissions-Policy`	`camera=(), microphone=(), geolocation=()`	Feature abuse

Use the HTTP Header Generator to generate a full security header bundle.

Quick checklist

Content-Security-Policy — start in Report-Only, tighten, then enforce
Set-Cookie — all session cookies have HttpOnly; Secure; SameSite=Lax
Strict-Transport-Security — enforce HTTPS for at least 2 years
X-Content-Type-Options: nosniff
X-Frame-Options: DENY or frame-ancestors 'none' in CSP
Referrer-Policy: strict-origin-when-cross-origin
No pages with sensitive data are missing noindex

Testing your headers

After deploying, test your headers with:

securityheaders.com — graded report
Google Search Console — shows indexing status for robots rules
Browser DevTools → Network tab → Response Headers — live inspection

A few well-configured headers are worth more than a firewall rule written to compensate for a missing HttpOnly flag.