Elasticsearch DSL Debugger

Start here

What it is, what it is not, and the first three minutes.

Three minutes

1. Open the ES Debugger tool window, on the bottom strip.
2. Choose a source: a bundled dataset, or one of your saved clusters and an index.
3. Paste a search body, or pick one of the 68 examples in the sidebar.
4. Press Debug query. The run stops at stage one.
5. Press Run all stages, then read the badges on the clauses.

The tool window

One line at the top always says what to do next, with the button that does it beside it. The rest is four areas, and any of them can be hidden: Focus the stages hides all but the stages, and Reset layout puts everything back.

The readiness line under the connection row is where the order of things is written. With a cluster selected it reads Step 1 of 2 — name the index this query runs against, then Step 2 of 2 — press Refresh fields to load the mapping of products, then Ready — products, 57 fields loaded. With a bundled dataset it reads Bundled dataset — nothing to load, and the index box shows that dataset’s index and is not editable.

Connecting a cluster

Settings › Tools › Elasticsearch DSL Debugger holds the clusters this project can query, as a list with the add, edit, remove and test buttons every other list in Settings has. Double-click edits, the buttons grey out when nothing is selected, and changes are saved as you make them. Remove asks first, because it takes the credentials with it. The tool window’s Settings button opens the same screen.

The Add Cluster dialog shows only the fields the chosen authentication kind uses. It validates in place: the first thing missing is named and its field outlined, and OK waits until it is fixed. Its own Test Connection button tries what has been typed before it is kept, and the answer lands under the form.

Secrets never reach the settings file. Passwords, API keys, bearer tokens and AWS secrets go to the IDE password safe, keyed by the connection’s id, so renaming a cluster or moving it to a new host keeps its credentials. Switching a connection’s authentication kind clears the secrets the new kind does not use. The settings file itself, which teams commit, holds only names, URLs and choices.

Run each feature

The short version: what to press, in order. Every feature reads the query in the editor; only the stages need it to have run. With the bundled dataset there is nothing to set up at all.

The funnel

Where the documents went, clause by clause.

Elasticsearch runs a search as a query phase on every shard and a fetch phase for the returned page, and does not stop between them. The debugger breaks that into ten stages you can stop at, each reporting how many documents are still alive.

1 Parse · 2 kNN · 3 Rewrite · 4 Match · 5 Score · 6 Aggregate · 7 Sort & collect · 8 Rescore · 9 Reduce & paginate · 10 Fetch

Step with Next stage and Previous stage, or jump to the end with Run all stages; Restart discards the results and starts again from stage one, and reads Re-run once the query has been edited. The position is spelled out beside them, Stage 4 of 10 — Match. Each chip carries the count of documents still in play, ↓N for how many the stage removed compared with the one before, and a count of the findings the stage raised in the worst severity’s colour. A skipped stage says why in two words, no knn, no aggs, and a failed one names the exception. Once the strip has focus, the Left and Right keys step through the stages.

Reading the badges

Step through it

Starts where you would: a query in the editor and nothing run yet. Press Next to press Debug query, then walk the funnel the way the debugger builds it. Every count comes from the plugin’s own engine run against the bundled E-commerce catalogue, 48 documents, with the query shown.

Switching clauses off

Right-click any clause in the graph.

Your query in the editor is never edited. A switched-off clause stays on screen, dashed, and only the body handed to the engine changes: in a must or filter it becomes match_all, inside a must_not it becomes match_none, so that disabling can widen the result and never narrow it. Results from a reduced query are labelled everywhere they appear, the guidance line counts the switched-off clauses, and the substitutions are listed in the Request tab.

Navigating a large query

Scroll to zoom, drag to pan, Shift+scroll to pan sideways. Find a clause jumps to a field or type. Collapse all, Expand all and Fit handle the shape; Indented tree rearranges the same graph as one clause per row for queries wider than the screen; Legend hides and shows the legend at the foot of the canvas. Arrow keys move the selection the way they move through a file tree, Enter opens the clause’s explanation and Space folds and unfolds it.

Where the numbers come from

Which parts are Elasticsearch’s own output, which are arithmetic on real queries, and which are ours.

Inside a bool, the result is the intersection of the clauses. Two quantities follow from that and are measured directly, one search each:

• What a clause matches alone — that clause run by itself against the index.
• The running intersection — the first n clauses run together. removed is the difference between one of these and the next.

Against a cluster, every number in the funnel is the count of a query that was really run. The _msearch carrying them is in the Request tab, and any line of it pasted into Kibana returns the same number.

Two things worth knowing about them

removed depends on the order the clauses are walked, and that order is ours: measured counts, cheapest first, which approximates what Lucene does rather than reproducing it. N docs does not depend on it. A clause that removes zero placed last removes zero in any order, which is why that finding is the one to act on.

The funnel checks itself. The last step is the whole conjunction, so its count must equal the count the query itself returned: two independent requests. If they disagree, the panel says so and names both numbers.

Source of each number

That describes a run against a cluster. Offline the counts come from the plugin’s own engine, timings are absent rather than invented, and Text analysis answers with the bundled analyzer and says so above the table. See the offline simulator.

The five views

Debug execution steps through the stages. The other four each ask their own question of the query in the editor, and need no run.

Diagnose

38 rules for the mistakes that never raise an error: the ones that return a result rather than a failure, so nothing tells you they happened. A term on a text field is the familiar example; most of the others are less obvious. Press Analyze query; it needs no run.

The view is a list and a detail. The list is filtered by severity and shows, per finding, the rule, the stage, how sure the rule is and how many examples it offers. Every finding states its certainty separately from what it proposes: Certain follows from the mapping, the analyzer, the settings or the version; Likely was inferred from a sample or a rule of thumb; Depends on what you meant says what the query does so you can decide.

A finding carries examples to try rather than a fix: several whole requests, each with a sentence saying when it is the right reading, its own risk level and, when a data source is available, its own measured outcome. Try it on the data runs the example and your query through every stage without touching the editor and shows what moved in the Inspector’s Compare tab. Show what changes shows the edit before it is applied. The apply button says what it is doing: Apply enhancement for an additive change, Apply suggestion… for one that changes what matches or how it ranks, which asks first, and Copy suggested query where the fix needs a mapping change or a value only you have. From the list, 1–9 pick an example, ← and → cycle them and Enter shows the change.

Performance

Where the milliseconds went, from Elasticsearch’s own profiler. Measure performance sends one profiled search: the query exactly as written, with profile: true as the only addition. It needs a cluster; without one the view dims.

It reads verdict first, evidence after: one sentence saying which phase dominated, one stacked bar of the phases with time and share in its legend, then sections that open and close. Why first, with the cause and a suggestion; then every clause by its own time, one row per clause as written, named in your DSL rather than in Lucene’s toString(), with the Lucene text on hover; aggregations when there are any; fetch closed until asked for; and By shard, opening itself only when one shard set the response time. Clicking a clause row selects that clause in the graph. Collector time is shown on its own line and kept out of the shares, as Elasticsearch documents it. A clause that a 31-shard index describes 31 times is still one row, with the shards’ time summed onto it.

Measure more than once. The first sample pays for cold caches. The view counts the runs of the same query against the same target, keeps the earlier totals in view, and says when the last two agree, when the numbers are still falling, and when a run came out slower than the one before.

Why not?

The problem. The funnel says how many documents each clause removed. A bug report is about one document: order 4471 should be in these results and it isn’t. Elasticsearch’s _explain gives a readable breakdown when the document matches, a terse nested “no matching term” when it does not, and nothing about paging either way.

How to use it. Have the query in the editor — it does not need to have run. Name the document by its _id, or by a field and value you know, such as sku = SKU-0000003, and press Find. Every clause is then asked about that one document separately, one _msearch entry per clause, so a document failing three clauses reports all three rather than the first one Lucene reached.

What you get. One row per clause and what it did — matches, rejects it, excludes it for a must_not that matched — and, on every row that decided against the document, the value the document holds in that field, read from _source: range price ≥ 50 — rejects it — price is 42. When every clause accepts it, and post_filter and min_score let it through, its position in the hits is measured against the page the request returns: at position 137 of 480, sorted by price desc, and the request returns positions 1–10 — paging is why it is missing. Offline the position is exact; against a cluster it comes from two counts, so it is a range when other documents tie with it. A text clause that rejected the document has a button into Text analysis, with the field, the text and the document filled in.

“No document with that id” is its own answer, an indexing problem rather than a query problem; against an alias holding the same id in several indices the row names the index; when several documents match your key they stay in a dropdown you can move between without re-running. The Data tab’s right-click, Why this document?, opens this view on a row that did match.

Text analysis

Why a match finds nothing when the words look right.

A search succeeds when the tokens a field stored and the tokens your query produces agree. This shows both. Against a cluster it asks _analyze twice with explain: true: by field for the index side, by analyzer name for the search side, resolved through the whole chain the cluster uses — the clause’s own analyzer, search_quote_analyzer for phrases, search_analyzer, the field’s analyzer, the index’s default_search, then default and standard — and it names which one ran and what chose it. A custom analyzer shows every tokenizer and filter as its own row with the tokens it added or removed; a built-in one shows its tokens, because _analyze does not expose its steps.

From query… fills field, text and analyzer from a clause in the editor. Document names the one you expected to match: its value is read from _source, analysed the way the index analysed it, and the clause is put to the cluster against that one document, so the verdict is Elasticsearch’s. Then it says what the difference means, one sentence per finding: a term on a text field is not analysed, a stemmer on one side only, a stop word that still counts a position, a keyword normalizer, a synonym_graph in the index chain.

Offline, the view answers with the bundled analyzer and says so above the table. The chain’s steps are not shown, and a custom analyzer, a synonym set or a stemmer can differ; pick a cluster for the real answer.

The Inspector

Beside the graph in Debug execution: eight tabs, each about the run, dimmed until it has what they read.

Compare

The rest of the Inspector answers what is happening. This tab answers did my edit work. It sits in the Inspector rather than in the view row because it reads what the stages produced: a comparison exists only once they have run twice, and the tab dims until then.

Every run that reaches Match is remembered, the last six, in memory. Change something, run again, and this shows what moved: the total, which documents on the returned page, which clauses’ counts, which buckets, and any stage whose status changed. The verdict leads with whatever would be missed most: a stage that started failing, then documents, then a total that moved while the page stayed the same, then clauses, then buckets.

The finding it exists for is the one a count cannot give you. A query going from 47 hits to 47 hits looks unchanged; if three of those documents are different ones, it is not.

By default a run is measured against the one before it. Several edits into chasing a regression that compares two broken states, so Pin as baseline holds the run that was last right, and everything is measured against it until you unpin. Warnings appear above the table only when the pair is not a fair comparison: a different index or source, a changed size, clauses switched off in one run only, a run stopped before Fetch, a reordering in a query with no sort, and a page that is a slice of the matches.

How far you run decides what you can compare

The sidebar

Examples, the mapping, the query’s outline, and everything you have run. Each tab title says how much it holds.

The offline simulator

How the plugin runs a query with no cluster, and where that stops being the same thing.

Pick a bundled dataset instead of a connection and the whole tool still works: ten stages, the funnel, per-clause counts, aggregations, sorting, scoring. There is no Elasticsearch behind it. The plugin carries its own query engine and runs your query against the documents inside the jar.

It is there so the tool can be learned, and a query understood, without a connection. Because the sample data is static, the simulator pins now to just after the newest sample document, so date-math examples keep selecting data however old the build is.

What it does itself

The plugin’s coverage file, generated from the code by the test suite, counts three catalogues separately.

Approximated means evaluated with a stated caveat carried on the result itself: kNN scans every vector rather than walking an HNSW graph, scripts run a Painless subset. BM25 is computed as Lucene computes it, with the lead clause chosen by a Lucene-style cost estimate; absolute scores differ from a real index because the term statistics do, and the arithmetic is the same.

Where it stops

The line matters more than the coverage, because a simulator that guesses is worse than no simulator. Where it cannot reproduce Elasticsearch it reports that it cannot, rather than producing a number:

• A clause or aggregation it does not evaluate stops the simulation of that stage and is reported as not simulated. No counts and no scores are invented, and the live engine is offered instead.
• An unknown clause name is reported as the parsing_exception it would cause; an unsimulated real clause as a limit of the tool. The two are kept apart.
• Text analysis answers with the bundled analyzer and says so. The built-in analyzers are approximated; a custom one falls back to standard and the note names it.
• Timings are absent rather than invented. There is no shard, so there is nothing to time, and the Performance view dims.

Limitations

What the plugin caps, approximates, or cannot know. Each one is stated in the interface at the moment it applies; they are gathered here so none of them is a surprise.

The funnel

Results and documents

• The last two stages cap the transfer at 100 documents even when the query asks for ten thousand. Totals, aggregations and ordering are decided on the shard before documents are sent, so nothing computed changes; the stage says so when the cap applies.
• Why not? takes an _id or up to three fields, combined with and: enough to identify a document, not a query builder.
• The per-shard listing stops at six shards. It costs one search per shard, so on a 100-shard index it would mean a hundred extra searches. Performance timings, its clause table and its shard-skew finding come from the single profiled search and cover every shard.

Text analysis

• One field and one text at a time. A multi_match over five fields is five checks.
• Built-in analyzers show tokens, not steps, because _analyze does not expose them. A custom analyzer shows every step.
• Offline is an approximation, labelled as one above the table.

Compare

• Six runs, this session only. A baseline from last week would be measured against an index that has changed underneath it.
• The document rows compare a page unless size covers every match, and the warning says how many of how many.
• Nothing is recorded before Match, and documents only exist from Fetch.

Performance timings

• Elasticsearch’s numbers, not ours. Every figure is profile’s time_in_nanos; the plugin times nothing itself.
• Per-clause time has its children subtracted, so a bool is not credited with the work of every clause inside it.
• Shard work is not wall clock. Shards run in parallel, so the summed work exceeds took and is meant to. The subtitle states both.
• Profiling changes what it measures, which is Elasticsearch’s own caveat. Read the proportions, not the milliseconds, and measure more than once.

By design, and not planned

• Elasticsearch 8 and 9. Older majors are flagged at Check cluster. OpenSearch is named correctly and its version numbers are not modelled, so the version-aware rules do not apply to it.
• The Query DSL only, not ES|QL, EQL, SQL or KQL.
• Nothing re-runs on its own. A debugger that re-ran on every keystroke would reach your cluster on every keystroke.
• No completion in the query editor, deliberately: a suggestion that is right most of the time is worse here than none. Field names are still offered where the question is unambiguous, in the index box and the field boxes of Why not? and Text analysis.
• AWS SigV4 is implemented and tested against the algorithm, not yet confirmed against a real Amazon domain. If it matters to you, write and it will be confirmed or fixed.

Reference

Every request the plugin makes, and where it makes it.

What each stage sends

Outside the stage loop: Performance sends one search with profile: true, Text analysis calls _analyze, Why not? calls _search once per clause for the document you named, Refresh fields reads _mapping, _settings and _field_caps, and the per-shard listing at Reduce re-runs the query with preference=_shards:n.

Keyboard

Next stage, Previous stage, Run all stages and Restart are in Find Action and bindable under Settings › Keymap. Six tips appear once each, chosen by what just happened, and Show Tips Again in Find Action brings them back.

Safety and privacy

What it can do to your cluster, and what leaves your machine.

It reads and it cannot write. Not by policy but by construction: every request is checked against an allow-list of read-only endpoints before a socket is opened, and everything else is refused, including anything unfamiliar. No _bulk, no _doc, no _update_by_query, no _delete_by_query, no _reindex, no PUT, no DELETE.

Endpoints that would leave server-side state, _pit, _scroll and _async_search, are refused too. A point in time may be suggested to you; the plugin never opens one.

Common questions