Product changelog
Recent API and platform changes (excerpt from the engineering changelog).
# Changelog
Owlpane publishes breaking API changes here before they ship. Patch releases that only add fields are not breaking.
## Policy
- HTTP APIs under `/v1/` stay compatible for at least 90 days after a breaking change is announced.
- A breaking change gets a dated entry below and a response header `X-Owlpane-Deprecation` on the old behavior when both can run together.
- Helm chart `owlpane-agent` uses semver. Minor chart bumps may add values; major bumps may rename them (`UPGRADING.md`).
## Unreleased
- **Lifecycle emails (operators, off by default):** owners of a new workspace can get a welcome, two nudges while no data has arrived, a note when data first arrives, and warnings 3 days and 1 day before the Team trial ends. Tips carry a one-click unsubscribe; the trial warnings do not. To use it: set `OWLPANE_LIFECYCLE_EMAILS=on`, `OWLPANE_UNSUBSCRIBE_SECRET` (32+ characters; nothing is sent without it) and `OWLPANE_EMAIL_FROM_LIFECYCLE`, and add SPF, DKIM and DMARC for the sender. New public route `GET`/`POST /v1/email/unsubscribe`. See `docs/ops/lifecycle-emails.md`. Migration `137_lifecycle_emails`; additive.
- **Alerts on a query:** new metrics `log_sql` (a Logs SQL aggregate over the rule's window) and `promql` (a PromQL expression read when the rule is evaluated) with a `ruleQuery` field (1–1,000 characters); the comparator and threshold judge the one number. A SQL rule that does not give one number, or a PromQL rule using `rate()` or `increase()`, is refused with `400`. New field and metrics, migration `093_alert_query_rules`; additive.
- **Metric usage:** `GET /v1/dashboards/usage` also returns `alerts`, the PromQL rules that read the metric. Additive.
- **PromQL:** instant queries with an aggregate (`sum`, `avg`, `max`, `min`, `count`, with or without `by`) and `histogram_quantile` no longer fail on ClickHouse versions that read an alias in `WHERE` as the aggregate it names.
- **Search jobs:** `/v1/search-jobs` runs a Logs SQL query over a big range in the background (up to 100,000 rows, kept 7 days) and keeps its rows to download as CSV or NDJSON; `/v1/search-jobs/schedules` runs one every 15 minutes, hour, 6 hours or day and keeps the last 5 results. New routes and migration `092_search_jobs`; additive. `OWLPANE_SEARCH_JOB_WORKER=off` keeps an API process from running jobs. See `docs/reference/search-jobs.md`.
- **Log settings:** `GET /v1/logs/settings` returns the log stream's retention, size, attributes and the rules that hide or rewrite lines. New route; additive.
- **Logs SQL mode:** `POST /v1/logs/sql` runs a query in a safe subset of SQL over the caller's own lines; `POST /v1/logs/sql/check` says whether a query is valid and where it is wrong; saved functions at `/v1/logs/functions` (list, create, change, delete, and `functions/test`). New routes; additive. Read-only tokens may run queries. See [Logs: SQL mode and functions](reference/logs-sql.md).
- **Dashboards:** a `logs` widget's `query` may carry `chart` (`area` or `lines`; bars is the default). Saved views of the Logs page may keep `ctype`. Additive.
- **Shared dashboards:** `POST /v1/public/dashboards/:token/logs` draws one `logs` widget of a shared dashboard (its own stored search, up to a day). New route; additive.
- **Telemetry store:** queries now ask ClickHouse to stop when the caller's connection closes (`cancel_http_readonly_queries_on_client_close`), so a cancelled search stops on the server too. Set `OWLPANE_CLICKHOUSE_CANCEL_ON_CLOSE=0` to turn it off; if the server refuses the setting the API drops it by itself.
- **LLM:** `GET /v1/llm/sessions` lists LLM conversations (by conversation, session or thread id) with calls, tokens, cache hits, cost and busy time, and with `ids` returns chosen ones to compare. New route; additive.
- **Profiler:** `GET /v1/profiler/span` returns the CPU profile windows of a service that overlapped a span (`frames`, `windows`, `mode: "overlapping-windows"`). New route; additive.
- **Service map:** `GET /v1/service-hosts` lists the pods or hosts a service's requests ran on, with requests, 5xx and p95. New route; additive.
- **Search:** `POST /v1/search/traces` finds traces by their spans (`traces`, `found`, `atLeast`); saved views and history of the Search page may keep `results=traces`. Read-only tokens may call it. Additive.
- **Saved views:** `kind` may now be `requests`, `spans` or `errors` (the Traces pages). `GET /v1/views` adds `favourite` (the caller's own star) and lists the caller's favourites after pinned views; `PUT /v1/views/:id/favourite` stars or unstars. Additive.
- **Metrics explorer:** `POST /v1/explorer/query` takes `agg: "increase"` on a sum (counter) metric and `p50`, `p95`, `p99` on a histogram metric (estimated from the bucket counts; not groupable). Additive. A capped answer now carries `partialResult` with `partialThroughMs` and `direction`.
- **Partial results:** `partialResult` on `/v1/logs`, `/v1/search/*`, `/v1/requests` and `/v1/spans` now also says `direction: "back"` (newest-first lists) and `partialThroughMs` is the oldest row shown. Additive.
- **Logs:** `GET /v1/logs/patterns` adds `rangeCount` and `rangeShare` to each pattern and `rangeTotal` to the response: counts over every line of the range, not only the 5,000-line sample. Additive.
- **Errors:** `GET /v1/errors` takes `from` and `to` (epoch milliseconds) for a custom window. Additive.
- **Metrics explorer:** `POST /v1/explorer/query`, `/buckets` and `/exemplars` accept `fromMs` and `toMs` (both or neither) for a custom window instead of the last `minutes`. Additive. The console's Metrics explorer, Search and Dashboards take a custom range, and Requests, the explorer and Dashboards have keyboard shortcuts (listed in Settings > Keyboard shortcuts).
- **Console (no API change):** the custom range on Logs, Requests and Spans takes expressions such as `now-2h` or `now-3h to now-1h`.
- **Console (no API change):** a request step with no log lines of its own shows the request's nearest lines; each service-map node has a bar for its calls against the busiest node.
- **Metrics explorer:** `POST /v1/explorer/exemplars` returns the readings a client kept with a trace id next to a metric (`t`, `value`, `traceId`, `spanId`), largest first, for the console's Example traces dots. Additive.
- **Metrics explorer:** `POST /v1/explorer/buckets` returns a histogram metric's bucket counts per time bucket and estimated p50, p95 and p99 (each with the range of the bucket it falls in, `mode: "estimated"`), for the console's Distribution card. Additive.
- **Console shortcuts:** Ctrl+Enter (Cmd+Enter) runs the search from inside a field on Logs and Search; the Search page gets its own letter and number keys. No API change.
- **Metrics explorer (console only):** links and chart clicks into Search now carry the metric's filters and, on a split chart, the clicked group. No API change.
- **PromQL:** the `summary` of `POST /v1/promql/check` now also lists `labelMatchers` (`key`, `op`, `value`, in the order written); `GET /v1/promql/names?label=__name__|service` lists metric or service names for the editor's pickers. Additive.
- **PromQL:** `POST /v1/promql/check` checks a query against the supported subset and says where it is wrong (`message`, `from`, `to`, `line`, `column`) or what it is, without running it; `GET /v1/promql/query` and `/v1/promql/query_range` run the same instant and range queries as `/api/v1/query` and `/api/v1/query_range`, under `/v1` for the console's PromQL editor. Additive.
- **Logs:** `GET /v1/logs/values` lists the commonest values of one attribute over the whole range with exact counts (`field`, `values` 1 to 50, and the filters of `/v1/logs`), for "show more values" beside the sampled field list. Additive.
- **Logs:** the group names of `GET /v1/logs/histogram?groupBy=` (and of a dashboard logs widget) are now shown redacted, as the log list shows attribute values, so a field holding an email or a token no longer names a group by it. Values that read the same once redacted are one group. Before, they were the raw stored values.
- **Dashboards:** a `logs` widget's `query` may carry `groupBy` (one attribute name): the chart then splits its bars by that attribute, as `groupBy` on `GET /v1/logs/histogram` does. A redacted attribute answers `400`; empty is dropped. Widgets without it are stored and returned exactly as before. Additive.
- **Logs:** `GET /v1/logs/histogram` takes `groupBy` (an attribute name). The answer then adds `groupBy`, `groups` (the 5 commonest values, `(no value)`, `(other)`) and `by` on each bucket. Exact counts. A redacted attribute answers `400`. Without it nothing changes. A Logs view may also keep `grp` (the attribute the chart is grouped by). Additive.
- **Views:** a Logs view may now also keep `sort`, `sdir`, `tf` (time format) and `msg` (message column hidden). Before, these were refused with a `400`. Additive.
- **Search:** `POST /v1/search/compare` takes `baseline` (`rest`, the default, or `before`). `before` compares the matches with the spans or log lines that match the same filters in the range of equal length just before this one; the answer adds `baselineMode` and `baselineWindow`. `400` for another value. Additive.
- **Short links:** `POST /v1/short-links` turns a console page (Logs, the metrics explorer, Search, Requests, Spans or Errors) and its query string into a 10-character code, and `GET /v1/short-links/:code` gives it back. A link lasts 30 days, is for one workspace only (another workspace sees a 404), and carries no access of its own. New table `obs_short_links` (migration `089_short_links`). Additive.
- **Traces:** each span in `GET /v1/trace/:traceId` now carries `events` (its events other than exceptions: `name`, `t` in epoch milliseconds, redacted `attributes`; earliest first, up to 50) and `links` (`traceId` and `spanId` of the spans it links to, up to 20). Additive.
- **Logs:** `GET /v1/logs/facets` takes `values` (1 to 50, default 8): how many values each field lists, most common first. Counts still describe the sampled lines. `400` for another value. Additive.
- **Logs:** repeating `attr` for the same key on the log routes (`/v1/logs`, `/patterns`, `/histogram`, `/facets` and `/scan`) now means any of those values, not all of them. Two values of one key could never both hold on a line, so no request that matched lines before changes. Other keys, and `xattr`, behave as before. Additive.
- **Metrics:** `GET /v1/explorer/metrics` accepts `minutes` (15, 60, 360, 1440 or 10080) and then lists only metrics that reported in that window; without it, the 7-day list as before (still the allow-list for queries). `400` for another value. Additive.
- **Metrics:** `POST /v1/explorer/query` (and the public dashboard query) accepts `topK`, 1 to 10: with a `groupBy`, draw only the busiest that many groups (by number of readings) instead of the 10 it draws by default. `400` for a value outside 1 to 10 or without a `groupBy`. Additive.
- **Traces:** `GET /v1/trace/:traceId` returns at most 2,000 spans, as before, but now says how many the trace has (`totalSpans`) and whether more follow (`more`), and takes `offset` (0 to 100,000) to read the next page. Spans are ordered by time and then span id so pages do not overlap. Log lines come with the first page only. Additive: a client that sends no `offset` gets the first page with the same fields it always had.
- **Dashboards:** canonical import paths `POST /v1/dashboards/import/prometheus-panels/*` and `.../tabbed-grid/*` replace product names in code and API surface. Legacy `.../import/grafana/*` and `.../import/openobserve/*` remain as aliases with `X-Owlpane-Deprecation`. Stored `import_key` prefixes are `prometheus_panels:` and `tabbed_grid:`; re-apply still finds rows keyed `grafana:` or `openobserve:`. Migration guides renamed to [migrate-prometheus-panel-dashboards.md](migrations/migrate-prometheus-panel-dashboards.md) and [migrate-tabbed-grid-dashboards.md](migrations/migrate-tabbed-grid-dashboards.md).
- **Dashboards (row 4, slice 1):** wider PromQL for import and `/api/v1/query*`: `max`/`min`/`count`, `=~`/`!~` matchers (length-capped), `group by` with up to two labels (widgets group by the first). **Text** widgets (`type: text`, `body` up to 4000 characters); tabbed-grid `markdown`/`html` panels import as text. Metric `chart` adds `area` and `stacked`; console charts render them. SQL/table/pie/heatmap panels remain unsupported.
- **Dashboards:** tabs and a **192-column grid layout** (OpenObserve-compatible). Bodies take `tabs` (up to 8, 24 widgets per tab, each widget has `layout: { x, y, w, h }`) or legacy `widgets` (one Overview tab). OpenObserve import keeps one file as one dashboard with tabs and layout; `importKey` is `openobserve:<dashboardId>`. Starter library regenerated with `tabs`. Console: tab bar, Arrange mode with drag/resize (GridStack), shared links view-only. `test/dashboard-layout.test.ts`.
- **Redaction:** custom redaction patterns are now **applied**, on read: log lines, error messages, SQL and attribute values shown or exported for a workspace have each match replaced by `[redacted]`. Stored telemetry is not rewritten. New `DELETE /v1/orgs/:orgId/redaction-rules/:ruleId`; `GET` lists `applied` and `reason` per rule and `appliesTo`. `POST` now also refuses patterns that match an empty value, use lookarounds or backreferences, or repeat a group holding a repeat or an alternative, and allows 20 rules per workspace; a rule stored earlier that fails these checks is listed as not applied and is not run. Saving the same pattern twice returns the existing rule.
- **Marketplace:** starter dashboards converted from the Apache-2.0 OpenObserve dashboards library (33 multi-tab dashboards after row 3; was 85 when each tab was a separate dashboard) (AWS, databases, Kafka, Nginx, Kubernetes and more) are listed with a new `category` field and install with `POST /v1/dashboard-templates/:slug/install`, including their variables. They expect the metric names of the exporter they were written for; charts for metrics you do not send stay empty. The template gallery still lists only Owlpane's own templates. Additive. Attribution in `THIRD_PARTY_NOTICES.md`.
- **Dashboards:** `POST /v1/dashboards/import/grafana` and `POST /v1/dashboards/import/openobserve` are new additive endpoints: they create the dashboards the matching `.../preview` proposes. Running one again with the same file changes nothing; a dashboard made by an earlier import is updated when the file changed and left alone when someone edited it. Exact translations only unless `includeApproximate: true`. Needs write access. Migration `083_dashboard_import_key` adds `import_key` and `import_hash` to `obs_dashboards`.
- **Dashboards:** dashboards take optional `variables` (up to 10): a fixed list (`custom`) or the values of one metric label (`metric_label`, optionally narrowed by earlier variables through `dependsOn`). A widget filter whose value is exactly `$name` takes the chosen value; `$service` and `$env` stay built in. `PUT /v1/dashboards/:id` without `variables` leaves the stored ones, so older clients do not wipe them. A `$name` filter on a variable that is not declared is now `400`. `GET /v1/explorer/attribute-values` takes an optional `where` to narrow values by other labels. The OpenObserve import keeps variables on a metric label and custom lists. Migration `082_dashboard_variables` adds `variables` to `obs_dashboards`. Additive; existing dashboards have none.
- **Logs:** `GET /v1/logs/patterns` patterns carry `examples` (up to three different lines, newest first) and `variables` (what each kind of masked part held: the commonest values with counts, and how many different values there were). Additive; `example` is unchanged.
- **Search:** `POST /v1/search/scan` is a new additive endpoint: how many spans or log lines a window holds, counted up to 5,000,000 as `{ rows, atLeast, cap }`, so the Search page can hold a very large search until it is confirmed.
- **Logs:** `GET /v1/logs/scan` is a new additive endpoint (`minutes` or `from` and `to`, `env`, `service`): how many log lines the range holds, counted up to 5,000,000, as `{ rows, atLeast, cap }`; `atLeast` is true when the count stopped at the cap. It does not take the search text or filters, because the query has to scan the range whatever they are. The Logs page uses it to hold a very large search until the person confirms.
- **Applications:** `GET /v1/applications` rows carry `avgMs` and `maxMs` (mean and slowest duration of the same spans as `p95Ms`; null when there are none). Additive.
- **Dashboards:** a new widget type, `logs` (`POST` and `PUT /v1/dashboards`): a chart of how many log lines match a search over time, with the Logs page's search (`query: { q, level, attrs, xattrs }`). The same limits as the log routes apply and an invalid query is a `400`. Additive: dashboards without such a widget are unchanged. A console from before this change cannot draw the widget (it expects `metric` or `builtin`), so a dashboard that has one needs the current console.
- **Search:** `POST /v1/search/operation` is a new additive endpoint: how long one operation (a service's span name) usually takes, with an exact count and estimated duration percentiles over the last hour, 6 hours, day or week.
- **Metrics:** new additive endpoints `GET` and `PUT /v1/favourites/metrics`: the metrics a signed-in person starred in the explorer, kept on the server per person and workspace (before, the console kept stars in the browser). `PUT` replaces the whole list and is safe to repeat.
- **PromQL API and dashboard imports:** label matchers may use `=~` and `!~` (RE2, matched against the whole value, as in Prometheus) as well as `=` and `!=`. A pattern the database cannot run is a `400` with the reason. The Grafana and OpenObserve dashboard imports keep such a matcher as a pattern filter on the widget instead of refusing the query.
- **Dashboards:** `GET /v1/dashboards/usage?metric=` is a new additive endpoint: the dashboards of the workspace that chart a metric, with how many of their metric widgets do.
- **Metrics:** explorer filters (`POST /v1/explorer/query` and the public dashboard query) accept two new operators, `regex` and `nregex`: an RE2 pattern that has to match the whole attribute value (PromQL's `=~` and `!~`), up to 200 characters, without lookahead, lookbehind or back-references. An unusable pattern is a `400` with the reason. Before, any operator other than `eq`, `neq` and `contains` was a `400`. Additive.
- **Search history:** `kind` also accepts `search`, for the console's Search page: the filters of a spans or log-lines search with its scope and range (`kind`, `f`, `env`, `service`, `minutes`, `fromMs`, `toMs`; at least one filter is needed). Saved views are unchanged (`logs` and `metrics`); the Search page keeps named searches of its own. Additive.
- **Search:** `POST /v1/search/heatmap` is a new additive endpoint: spans counted by time bucket and by duration (fixed bins from under 1 ms to 30 s and up), for the same environment, service, window and filters as the span search. Counts are exact, not sampled.
- **Saved views and search history:** `kind` also accepts `metrics`, for the metrics explorer: a view keeps the chart (`metric`, `type`, `agg`, `group`, `f`, `env`, `service`, `minutes`, `cmp`, `fc`), and history needs a `metric`. Logs and metrics lists are separate; the logs behaviour is unchanged. Migration `086` widens the `kind` checks. Additive.
- **Metrics:** `POST /v1/explorer/query` (and the public dashboard query) takes an optional `offsetMinutes`: the same window drawn from an earlier period. It must be a whole number of chart buckets and window plus shift at most 14 days (20160 minutes); the returned times are the real, earlier ones. Without it nothing changes. The metrics explorer's Compare with uses it.
- **Search history:** new additive endpoints `GET`, `POST`, `PATCH` and `DELETE /v1/history`: the searches a signed-in person ran on the Logs page (search text, level, filters, service, range), private to that person, newest 100 plus any they starred. Nothing existing changes; the console's Logs page will use it.
- **Saved views:** new additive endpoints `GET`, `POST`, `PATCH` and `DELETE /v1/views`: a named set of settings of a console page (so far the Logs page: search text, level, filters, range, columns and how lines are shown), shared with the workspace, with an optional pin. Nothing existing changes; the console's Logs page will use it.
- **Logs:** `GET /v1/logs` takes `offset` (0 to 10,000) for paging and always returns the `offset` it applied. Lines are ordered by time with a fixed tie-break, and the lines the log pipeline hides (health checks) are left out in the query rather than after it, so a page is no longer short because of them. Additive; the Logs page uses it for page numbers.
- **Logs:** `GET /v1/logs`, `/v1/logs/patterns`, `/v1/logs/facets` and `/v1/logs/histogram` take `from` and `to` (epoch milliseconds) for an absolute time window within the last 7 days, instead of `minutes`. Additive; the Logs page uses it for a custom range and for zooming into a histogram bar.
- **Logs:** `GET /v1/logs`, `/v1/logs/patterns`, `/v1/logs/facets` and `/v1/logs/histogram` take a repeatable `xattr=key~value`, the opposite of `attr`: it leaves out the lines whose attribute `key` is exactly `value`. Additive; the Logs page uses it for "exclude this value".
Owlpane publishes breaking API changes here before they ship. Patch releases that only add fields are not breaking.
## Policy
- HTTP APIs under `/v1/` stay compatible for at least 90 days after a breaking change is announced.
- A breaking change gets a dated entry below and a response header `X-Owlpane-Deprecation` on the old behavior when both can run together.
- Helm chart `owlpane-agent` uses semver. Minor chart bumps may add values; major bumps may rename them (`UPGRADING.md`).
## Unreleased
- **Lifecycle emails (operators, off by default):** owners of a new workspace can get a welcome, two nudges while no data has arrived, a note when data first arrives, and warnings 3 days and 1 day before the Team trial ends. Tips carry a one-click unsubscribe; the trial warnings do not. To use it: set `OWLPANE_LIFECYCLE_EMAILS=on`, `OWLPANE_UNSUBSCRIBE_SECRET` (32+ characters; nothing is sent without it) and `OWLPANE_EMAIL_FROM_LIFECYCLE`, and add SPF, DKIM and DMARC for the sender. New public route `GET`/`POST /v1/email/unsubscribe`. See `docs/ops/lifecycle-emails.md`. Migration `137_lifecycle_emails`; additive.
- **Alerts on a query:** new metrics `log_sql` (a Logs SQL aggregate over the rule's window) and `promql` (a PromQL expression read when the rule is evaluated) with a `ruleQuery` field (1–1,000 characters); the comparator and threshold judge the one number. A SQL rule that does not give one number, or a PromQL rule using `rate()` or `increase()`, is refused with `400`. New field and metrics, migration `093_alert_query_rules`; additive.
- **Metric usage:** `GET /v1/dashboards/usage` also returns `alerts`, the PromQL rules that read the metric. Additive.
- **PromQL:** instant queries with an aggregate (`sum`, `avg`, `max`, `min`, `count`, with or without `by`) and `histogram_quantile` no longer fail on ClickHouse versions that read an alias in `WHERE` as the aggregate it names.
- **Search jobs:** `/v1/search-jobs` runs a Logs SQL query over a big range in the background (up to 100,000 rows, kept 7 days) and keeps its rows to download as CSV or NDJSON; `/v1/search-jobs/schedules` runs one every 15 minutes, hour, 6 hours or day and keeps the last 5 results. New routes and migration `092_search_jobs`; additive. `OWLPANE_SEARCH_JOB_WORKER=off` keeps an API process from running jobs. See `docs/reference/search-jobs.md`.
- **Log settings:** `GET /v1/logs/settings` returns the log stream's retention, size, attributes and the rules that hide or rewrite lines. New route; additive.
- **Logs SQL mode:** `POST /v1/logs/sql` runs a query in a safe subset of SQL over the caller's own lines; `POST /v1/logs/sql/check` says whether a query is valid and where it is wrong; saved functions at `/v1/logs/functions` (list, create, change, delete, and `functions/test`). New routes; additive. Read-only tokens may run queries. See [Logs: SQL mode and functions](reference/logs-sql.md).
- **Dashboards:** a `logs` widget's `query` may carry `chart` (`area` or `lines`; bars is the default). Saved views of the Logs page may keep `ctype`. Additive.
- **Shared dashboards:** `POST /v1/public/dashboards/:token/logs` draws one `logs` widget of a shared dashboard (its own stored search, up to a day). New route; additive.
- **Telemetry store:** queries now ask ClickHouse to stop when the caller's connection closes (`cancel_http_readonly_queries_on_client_close`), so a cancelled search stops on the server too. Set `OWLPANE_CLICKHOUSE_CANCEL_ON_CLOSE=0` to turn it off; if the server refuses the setting the API drops it by itself.
- **LLM:** `GET /v1/llm/sessions` lists LLM conversations (by conversation, session or thread id) with calls, tokens, cache hits, cost and busy time, and with `ids` returns chosen ones to compare. New route; additive.
- **Profiler:** `GET /v1/profiler/span` returns the CPU profile windows of a service that overlapped a span (`frames`, `windows`, `mode: "overlapping-windows"`). New route; additive.
- **Service map:** `GET /v1/service-hosts` lists the pods or hosts a service's requests ran on, with requests, 5xx and p95. New route; additive.
- **Search:** `POST /v1/search/traces` finds traces by their spans (`traces`, `found`, `atLeast`); saved views and history of the Search page may keep `results=traces`. Read-only tokens may call it. Additive.
- **Saved views:** `kind` may now be `requests`, `spans` or `errors` (the Traces pages). `GET /v1/views` adds `favourite` (the caller's own star) and lists the caller's favourites after pinned views; `PUT /v1/views/:id/favourite` stars or unstars. Additive.
- **Metrics explorer:** `POST /v1/explorer/query` takes `agg: "increase"` on a sum (counter) metric and `p50`, `p95`, `p99` on a histogram metric (estimated from the bucket counts; not groupable). Additive. A capped answer now carries `partialResult` with `partialThroughMs` and `direction`.
- **Partial results:** `partialResult` on `/v1/logs`, `/v1/search/*`, `/v1/requests` and `/v1/spans` now also says `direction: "back"` (newest-first lists) and `partialThroughMs` is the oldest row shown. Additive.
- **Logs:** `GET /v1/logs/patterns` adds `rangeCount` and `rangeShare` to each pattern and `rangeTotal` to the response: counts over every line of the range, not only the 5,000-line sample. Additive.
- **Errors:** `GET /v1/errors` takes `from` and `to` (epoch milliseconds) for a custom window. Additive.
- **Metrics explorer:** `POST /v1/explorer/query`, `/buckets` and `/exemplars` accept `fromMs` and `toMs` (both or neither) for a custom window instead of the last `minutes`. Additive. The console's Metrics explorer, Search and Dashboards take a custom range, and Requests, the explorer and Dashboards have keyboard shortcuts (listed in Settings > Keyboard shortcuts).
- **Console (no API change):** the custom range on Logs, Requests and Spans takes expressions such as `now-2h` or `now-3h to now-1h`.
- **Console (no API change):** a request step with no log lines of its own shows the request's nearest lines; each service-map node has a bar for its calls against the busiest node.
- **Metrics explorer:** `POST /v1/explorer/exemplars` returns the readings a client kept with a trace id next to a metric (`t`, `value`, `traceId`, `spanId`), largest first, for the console's Example traces dots. Additive.
- **Metrics explorer:** `POST /v1/explorer/buckets` returns a histogram metric's bucket counts per time bucket and estimated p50, p95 and p99 (each with the range of the bucket it falls in, `mode: "estimated"`), for the console's Distribution card. Additive.
- **Console shortcuts:** Ctrl+Enter (Cmd+Enter) runs the search from inside a field on Logs and Search; the Search page gets its own letter and number keys. No API change.
- **Metrics explorer (console only):** links and chart clicks into Search now carry the metric's filters and, on a split chart, the clicked group. No API change.
- **PromQL:** the `summary` of `POST /v1/promql/check` now also lists `labelMatchers` (`key`, `op`, `value`, in the order written); `GET /v1/promql/names?label=__name__|service` lists metric or service names for the editor's pickers. Additive.
- **PromQL:** `POST /v1/promql/check` checks a query against the supported subset and says where it is wrong (`message`, `from`, `to`, `line`, `column`) or what it is, without running it; `GET /v1/promql/query` and `/v1/promql/query_range` run the same instant and range queries as `/api/v1/query` and `/api/v1/query_range`, under `/v1` for the console's PromQL editor. Additive.
- **Logs:** `GET /v1/logs/values` lists the commonest values of one attribute over the whole range with exact counts (`field`, `values` 1 to 50, and the filters of `/v1/logs`), for "show more values" beside the sampled field list. Additive.
- **Logs:** the group names of `GET /v1/logs/histogram?groupBy=` (and of a dashboard logs widget) are now shown redacted, as the log list shows attribute values, so a field holding an email or a token no longer names a group by it. Values that read the same once redacted are one group. Before, they were the raw stored values.
- **Dashboards:** a `logs` widget's `query` may carry `groupBy` (one attribute name): the chart then splits its bars by that attribute, as `groupBy` on `GET /v1/logs/histogram` does. A redacted attribute answers `400`; empty is dropped. Widgets without it are stored and returned exactly as before. Additive.
- **Logs:** `GET /v1/logs/histogram` takes `groupBy` (an attribute name). The answer then adds `groupBy`, `groups` (the 5 commonest values, `(no value)`, `(other)`) and `by` on each bucket. Exact counts. A redacted attribute answers `400`. Without it nothing changes. A Logs view may also keep `grp` (the attribute the chart is grouped by). Additive.
- **Views:** a Logs view may now also keep `sort`, `sdir`, `tf` (time format) and `msg` (message column hidden). Before, these were refused with a `400`. Additive.
- **Search:** `POST /v1/search/compare` takes `baseline` (`rest`, the default, or `before`). `before` compares the matches with the spans or log lines that match the same filters in the range of equal length just before this one; the answer adds `baselineMode` and `baselineWindow`. `400` for another value. Additive.
- **Short links:** `POST /v1/short-links` turns a console page (Logs, the metrics explorer, Search, Requests, Spans or Errors) and its query string into a 10-character code, and `GET /v1/short-links/:code` gives it back. A link lasts 30 days, is for one workspace only (another workspace sees a 404), and carries no access of its own. New table `obs_short_links` (migration `089_short_links`). Additive.
- **Traces:** each span in `GET /v1/trace/:traceId` now carries `events` (its events other than exceptions: `name`, `t` in epoch milliseconds, redacted `attributes`; earliest first, up to 50) and `links` (`traceId` and `spanId` of the spans it links to, up to 20). Additive.
- **Logs:** `GET /v1/logs/facets` takes `values` (1 to 50, default 8): how many values each field lists, most common first. Counts still describe the sampled lines. `400` for another value. Additive.
- **Logs:** repeating `attr` for the same key on the log routes (`/v1/logs`, `/patterns`, `/histogram`, `/facets` and `/scan`) now means any of those values, not all of them. Two values of one key could never both hold on a line, so no request that matched lines before changes. Other keys, and `xattr`, behave as before. Additive.
- **Metrics:** `GET /v1/explorer/metrics` accepts `minutes` (15, 60, 360, 1440 or 10080) and then lists only metrics that reported in that window; without it, the 7-day list as before (still the allow-list for queries). `400` for another value. Additive.
- **Metrics:** `POST /v1/explorer/query` (and the public dashboard query) accepts `topK`, 1 to 10: with a `groupBy`, draw only the busiest that many groups (by number of readings) instead of the 10 it draws by default. `400` for a value outside 1 to 10 or without a `groupBy`. Additive.
- **Traces:** `GET /v1/trace/:traceId` returns at most 2,000 spans, as before, but now says how many the trace has (`totalSpans`) and whether more follow (`more`), and takes `offset` (0 to 100,000) to read the next page. Spans are ordered by time and then span id so pages do not overlap. Log lines come with the first page only. Additive: a client that sends no `offset` gets the first page with the same fields it always had.
- **Dashboards:** canonical import paths `POST /v1/dashboards/import/prometheus-panels/*` and `.../tabbed-grid/*` replace product names in code and API surface. Legacy `.../import/grafana/*` and `.../import/openobserve/*` remain as aliases with `X-Owlpane-Deprecation`. Stored `import_key` prefixes are `prometheus_panels:` and `tabbed_grid:`; re-apply still finds rows keyed `grafana:` or `openobserve:`. Migration guides renamed to [migrate-prometheus-panel-dashboards.md](migrations/migrate-prometheus-panel-dashboards.md) and [migrate-tabbed-grid-dashboards.md](migrations/migrate-tabbed-grid-dashboards.md).
- **Dashboards (row 4, slice 1):** wider PromQL for import and `/api/v1/query*`: `max`/`min`/`count`, `=~`/`!~` matchers (length-capped), `group by` with up to two labels (widgets group by the first). **Text** widgets (`type: text`, `body` up to 4000 characters); tabbed-grid `markdown`/`html` panels import as text. Metric `chart` adds `area` and `stacked`; console charts render them. SQL/table/pie/heatmap panels remain unsupported.
- **Dashboards:** tabs and a **192-column grid layout** (OpenObserve-compatible). Bodies take `tabs` (up to 8, 24 widgets per tab, each widget has `layout: { x, y, w, h }`) or legacy `widgets` (one Overview tab). OpenObserve import keeps one file as one dashboard with tabs and layout; `importKey` is `openobserve:<dashboardId>`. Starter library regenerated with `tabs`. Console: tab bar, Arrange mode with drag/resize (GridStack), shared links view-only. `test/dashboard-layout.test.ts`.
- **Redaction:** custom redaction patterns are now **applied**, on read: log lines, error messages, SQL and attribute values shown or exported for a workspace have each match replaced by `[redacted]`. Stored telemetry is not rewritten. New `DELETE /v1/orgs/:orgId/redaction-rules/:ruleId`; `GET` lists `applied` and `reason` per rule and `appliesTo`. `POST` now also refuses patterns that match an empty value, use lookarounds or backreferences, or repeat a group holding a repeat or an alternative, and allows 20 rules per workspace; a rule stored earlier that fails these checks is listed as not applied and is not run. Saving the same pattern twice returns the existing rule.
- **Marketplace:** starter dashboards converted from the Apache-2.0 OpenObserve dashboards library (33 multi-tab dashboards after row 3; was 85 when each tab was a separate dashboard) (AWS, databases, Kafka, Nginx, Kubernetes and more) are listed with a new `category` field and install with `POST /v1/dashboard-templates/:slug/install`, including their variables. They expect the metric names of the exporter they were written for; charts for metrics you do not send stay empty. The template gallery still lists only Owlpane's own templates. Additive. Attribution in `THIRD_PARTY_NOTICES.md`.
- **Dashboards:** `POST /v1/dashboards/import/grafana` and `POST /v1/dashboards/import/openobserve` are new additive endpoints: they create the dashboards the matching `.../preview` proposes. Running one again with the same file changes nothing; a dashboard made by an earlier import is updated when the file changed and left alone when someone edited it. Exact translations only unless `includeApproximate: true`. Needs write access. Migration `083_dashboard_import_key` adds `import_key` and `import_hash` to `obs_dashboards`.
- **Dashboards:** dashboards take optional `variables` (up to 10): a fixed list (`custom`) or the values of one metric label (`metric_label`, optionally narrowed by earlier variables through `dependsOn`). A widget filter whose value is exactly `$name` takes the chosen value; `$service` and `$env` stay built in. `PUT /v1/dashboards/:id` without `variables` leaves the stored ones, so older clients do not wipe them. A `$name` filter on a variable that is not declared is now `400`. `GET /v1/explorer/attribute-values` takes an optional `where` to narrow values by other labels. The OpenObserve import keeps variables on a metric label and custom lists. Migration `082_dashboard_variables` adds `variables` to `obs_dashboards`. Additive; existing dashboards have none.
- **Logs:** `GET /v1/logs/patterns` patterns carry `examples` (up to three different lines, newest first) and `variables` (what each kind of masked part held: the commonest values with counts, and how many different values there were). Additive; `example` is unchanged.
- **Search:** `POST /v1/search/scan` is a new additive endpoint: how many spans or log lines a window holds, counted up to 5,000,000 as `{ rows, atLeast, cap }`, so the Search page can hold a very large search until it is confirmed.
- **Logs:** `GET /v1/logs/scan` is a new additive endpoint (`minutes` or `from` and `to`, `env`, `service`): how many log lines the range holds, counted up to 5,000,000, as `{ rows, atLeast, cap }`; `atLeast` is true when the count stopped at the cap. It does not take the search text or filters, because the query has to scan the range whatever they are. The Logs page uses it to hold a very large search until the person confirms.
- **Applications:** `GET /v1/applications` rows carry `avgMs` and `maxMs` (mean and slowest duration of the same spans as `p95Ms`; null when there are none). Additive.
- **Dashboards:** a new widget type, `logs` (`POST` and `PUT /v1/dashboards`): a chart of how many log lines match a search over time, with the Logs page's search (`query: { q, level, attrs, xattrs }`). The same limits as the log routes apply and an invalid query is a `400`. Additive: dashboards without such a widget are unchanged. A console from before this change cannot draw the widget (it expects `metric` or `builtin`), so a dashboard that has one needs the current console.
- **Search:** `POST /v1/search/operation` is a new additive endpoint: how long one operation (a service's span name) usually takes, with an exact count and estimated duration percentiles over the last hour, 6 hours, day or week.
- **Metrics:** new additive endpoints `GET` and `PUT /v1/favourites/metrics`: the metrics a signed-in person starred in the explorer, kept on the server per person and workspace (before, the console kept stars in the browser). `PUT` replaces the whole list and is safe to repeat.
- **PromQL API and dashboard imports:** label matchers may use `=~` and `!~` (RE2, matched against the whole value, as in Prometheus) as well as `=` and `!=`. A pattern the database cannot run is a `400` with the reason. The Grafana and OpenObserve dashboard imports keep such a matcher as a pattern filter on the widget instead of refusing the query.
- **Dashboards:** `GET /v1/dashboards/usage?metric=` is a new additive endpoint: the dashboards of the workspace that chart a metric, with how many of their metric widgets do.
- **Metrics:** explorer filters (`POST /v1/explorer/query` and the public dashboard query) accept two new operators, `regex` and `nregex`: an RE2 pattern that has to match the whole attribute value (PromQL's `=~` and `!~`), up to 200 characters, without lookahead, lookbehind or back-references. An unusable pattern is a `400` with the reason. Before, any operator other than `eq`, `neq` and `contains` was a `400`. Additive.
- **Search history:** `kind` also accepts `search`, for the console's Search page: the filters of a spans or log-lines search with its scope and range (`kind`, `f`, `env`, `service`, `minutes`, `fromMs`, `toMs`; at least one filter is needed). Saved views are unchanged (`logs` and `metrics`); the Search page keeps named searches of its own. Additive.
- **Search:** `POST /v1/search/heatmap` is a new additive endpoint: spans counted by time bucket and by duration (fixed bins from under 1 ms to 30 s and up), for the same environment, service, window and filters as the span search. Counts are exact, not sampled.
- **Saved views and search history:** `kind` also accepts `metrics`, for the metrics explorer: a view keeps the chart (`metric`, `type`, `agg`, `group`, `f`, `env`, `service`, `minutes`, `cmp`, `fc`), and history needs a `metric`. Logs and metrics lists are separate; the logs behaviour is unchanged. Migration `086` widens the `kind` checks. Additive.
- **Metrics:** `POST /v1/explorer/query` (and the public dashboard query) takes an optional `offsetMinutes`: the same window drawn from an earlier period. It must be a whole number of chart buckets and window plus shift at most 14 days (20160 minutes); the returned times are the real, earlier ones. Without it nothing changes. The metrics explorer's Compare with uses it.
- **Search history:** new additive endpoints `GET`, `POST`, `PATCH` and `DELETE /v1/history`: the searches a signed-in person ran on the Logs page (search text, level, filters, service, range), private to that person, newest 100 plus any they starred. Nothing existing changes; the console's Logs page will use it.
- **Saved views:** new additive endpoints `GET`, `POST`, `PATCH` and `DELETE /v1/views`: a named set of settings of a console page (so far the Logs page: search text, level, filters, range, columns and how lines are shown), shared with the workspace, with an optional pin. Nothing existing changes; the console's Logs page will use it.
- **Logs:** `GET /v1/logs` takes `offset` (0 to 10,000) for paging and always returns the `offset` it applied. Lines are ordered by time with a fixed tie-break, and the lines the log pipeline hides (health checks) are left out in the query rather than after it, so a page is no longer short because of them. Additive; the Logs page uses it for page numbers.
- **Logs:** `GET /v1/logs`, `/v1/logs/patterns`, `/v1/logs/facets` and `/v1/logs/histogram` take `from` and `to` (epoch milliseconds) for an absolute time window within the last 7 days, instead of `minutes`. Additive; the Logs page uses it for a custom range and for zooming into a histogram bar.
- **Logs:** `GET /v1/logs`, `/v1/logs/patterns`, `/v1/logs/facets` and `/v1/logs/histogram` take a repeatable `xattr=key~value`, the opposite of `attr`: it leaves out the lines whose attribute `key` is exactly `value`. Additive; the Logs page uses it for "exclude this value".