Templates
Templates You Can Use Today
Observability
OpenTelemetry Collector to ClickHouse
Receive OTLP, protect the Collector with a memory limiter, keep every error and slow trace with tail sampling, and write to ClickHouse with a 14-day TTL. Requires the Collector Contrib distribution; check key names against your Collector version.
# OpenTelemetry Collector (contrib) -> ClickHouse
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
tail_sampling:
decision_wait: 10s
policies:
- name: keep-errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: keep-slow
type: latency
latency: { threshold_ms: 1000 }
- name: sample-the-rest
type: probabilistic
probabilistic: { sampling_percentage: 10 }
batch:
send_batch_size: 10000
timeout: 5s
exporters:
clickhouse:
endpoint: tcp://clickhouse:9000?dial_timeout=10s
database: otel
ttl: 336h # 14 days
traces_table_name: otel_traces
logs_table_name: otel_logs
create_schema: true
timeout: 10s
retry_on_failure:
enabled: true
sending_queue:
queue_size: 1000
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [clickhouse]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [clickhouse]
Observability
ClickHouse Logs Table With Tiered Retention
Raw logs kept for 14 days, plus a per-minute error rollup kept for 13 months through a materialized view. Long-term trends cost a fraction of raw retention.
-- Raw logs: 14 days
CREATE TABLE logs
(
Timestamp DateTime64(9) CODEC(Delta, ZSTD(1)),
ServiceName LowCardinality(String),
SeverityText LowCardinality(String),
TraceId String CODEC(ZSTD(1)),
Body String CODEC(ZSTD(1)),
Attributes Map(LowCardinality(String), String) CODEC(ZSTD(1)),
INDEX idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SeverityText, toDateTime(Timestamp))
TTL toDateTime(Timestamp) + INTERVAL 14 DAY;
-- Per-minute rollup: 13 months
CREATE TABLE logs_errors_per_minute
(
Minute DateTime,
ServiceName LowCardinality(String),
Errors UInt64,
Total UInt64
)
ENGINE = SummingMergeTree
ORDER BY (ServiceName, Minute)
TTL Minute + INTERVAL 13 MONTH;
CREATE MATERIALIZED VIEW logs_errors_per_minute_mv
TO logs_errors_per_minute AS
SELECT
toStartOfMinute(Timestamp) AS Minute,
ServiceName,
countIf(SeverityText IN ('ERROR', 'FATAL')) AS Errors,
count() AS Total
FROM logs
GROUP BY Minute, ServiceName;
Security
Non-Human Identity Inventory Sheet
The minimum fields to record for every service account, token, and CI credential. Import it into a spreadsheet to start an inventory, then retire anything without an owner.
identity,type,system,owner_team,purpose,permissions,created,last_used,rotation_method,last_rotated,stored_in,status
deploy-bot,aws_access_key,aws:prod,platform,CI deploys to ECS,ecs:UpdateService;ecr:Push,2024-03-11,2026-10-24,manual,2026-01-15,github-actions-secret,replace-with-oidc
ci-runner@proj.iam.gserviceaccount.com,gcp_service_account_key,gcp:analytics,data,nightly export,bigquery.dataViewer,2025-07-02,2026-10-26,none,never,gitlab-ci-variable,review
Compliance
Quarterly Access Review Checklist
A repeatable access review that satisfies SOC 2 CC6, ISO 27001 A.5.18, and HIPAA 164.312(a), including machine identities that human-only reviews miss.
# Quarterly Access Review
Period: ____ Reviewer: ____ Approver: ____
## 1. Scope
- [ ] List in-scope systems (cloud accounts, identity provider, source control, production databases, key SaaS).
- [ ] Export current users, roles, and groups from each system (save the export as evidence).
- [ ] Export non-human identities: service accounts, API tokens, CI/CD credentials, agent tokens.
## 2. Review
- [ ] Every account maps to a current employee, contractor, or owned service.
- [ ] Leavers since the last review have no remaining access.
- [ ] Privileged roles (admin, owner, production write) are still justified.
- [ ] Machine identities unused for 90+ days are flagged for removal.
- [ ] Shared accounts are documented with an owner and a reason.
## 3. Act
- [ ] Remove or reduce access for every flagged account; record ticket links.
- [ ] Rotate credentials for any identity whose owner changed.
## 4. Record
- [ ] Reviewer and approver sign-off with date.
- [ ] Exports, decisions, and tickets stored with the review.
Operations
Blameless Incident Postmortem
A short postmortem format focused on detection, impact by customer, and follow-up actions with owners. Pairs well with trace and log queries for the timeline.
# Incident Postmortem: <title>
Date: ____ Severity: ____ Author: ____ Status: draft | reviewed
## Summary
Two or three sentences: what happened, who was affected, how it was resolved.
## Impact
- Duration (start to full recovery):
- Customers or tenants affected (query your traces by tenant ID):
- SLOs affected and error budget consumed:
## Timeline (UTC)
| Time | Event |
| ---- | ----- |
| | First signal (alert, forecast, or customer report) |
| | Incident declared |
| | Mitigation applied |
| | Recovery confirmed |
## Detection
How did we find out? How could we have found out sooner?
## Root cause and contributing factors
Describe the system conditions, not individual mistakes.
## What went well / what was hard
## Action items
| Action | Owner | Due | Ticket |
| ------ | ----- | --- | ------ |
Security
security.txt (RFC 9116)
Publish at /.well-known/security.txt so researchers know how to report vulnerabilities. Expires is required; set a reminder to renew it before it lapses.
Contact: mailto:security@example.com
Expires: 2027-10-31T00:00:00.000Z
Preferred-Languages: en
Canonical: https://example.com/.well-known/security.txt
Policy: https://example.com/security#disclosure
Talk to an Architect
See the Four Products Running in Your Environment.
A technical session with a Lionhouse architect covering your stack, your deployment model, and a pilot plan.