Tag: Superset Upgrade

Superset Upgrade Playbook: Moving a Production Instance from 3.x to 4.x Without Downtime

Posted on by Mandakinee
A Superset upgrade has one property that shapes everything else: the metadata database migration runs forward and there is no supported migration backward. Once the schema is at 4.x, a 3.x image cannot start against it. Every other risk in an upgrade is recoverable by redeploying the old image. This one is recoverable only by restoring the database. The playbook below is built around that fact.It is written for the 3.x to 4.x jump on a self-hosted deployment (Docker, Kubernetes or VMs) and applies with small changes to any major version. The specifics of what changed in 4.0 are from the release’s UPDATING.md; read it for your exact source and target versions rather than relying on this summary.

Phase 1: Read

Before any environment is touched.

`UPDATING.md`. The Superset repository keeps a running file of breaking changes by version. Next, read every entry between your current version and your target. Then copy each one into a checklist with a column for “affects us” and “action”. For 3.x to 4.0 the entries most deployments hit:

  • Filter box removed. The legacy filter box chart type is gone. Dashboards that used it need native filters. Superset provides a migration, but it is a data change, so it is rehearsed like everything else.
  • Legacy chart migrations completed. Several older chart types (the legacy area, bar, line and others) were migrated to their ECharts replacements and the originals removed. Saved charts of those types must be migrated with the superset viz-migrations command.
  • Talisman on by default. Flask-Talisman and its content security policy are enabled. Anything that loads external resources (map tiles, fonts, scripts) or embeds Superset in an iframe needs TALISMAN_CONFIG entries. If you use embedded dashboards, frame-ancestors is now mandatory.
  • Python version. 4.x requires Python 3.9 or later (3.10 or 3.11 recommended at the time of 4.0). A custom image on an older base needs a new base.
  • Feature flag defaults. Several flags flipped, notably around generic chart axes, cross-filters and native filters becoming default-on. Diff superset/config.py between the two tags for the DEFAULT_FEATURE_FLAGS dictionary and note every change.
  • Removed configuration keys. Deprecated settings are removed at major versions. Superset logs a warning for unknown keys at startup; your rehearsal will surface them.

What your own environment adds

Next, your own inventory. From the metadata database: count of charts by viz_type, dashboards using filter boxes, custom plugins and their peer dependency pins, custom security manager or other Python extensions, CSS templates (they target internal class names), and alerts and reports (the report worker’s browser dependency may change).

Finally, dependencies. Database drivers pinned in your image, the Celery and Redis versions, the reverse proxy configuration for websocket support if you use it.

At the end of Phase 1 you have a checklist of actions and an estimate. Therefore, if the checklist has surprises, this is the cheap moment to find them.

Phase 2: Rehearse

Never run the first upgrade against production. Instead, rehearse against a copy of the production metadata database, because the metadata is where the risk is.

  • Snapshot production metadata. pg_dump the metadata database (or your provider’s snapshot). Restore it into a staging database.
  • Stand up staging on the current version against that copy, with the same superset_config.py and the same Redis topology (a fresh Redis is fine). Confirm it works: log in, open the top ten dashboards.
  • Build the target image. Same Dockerfile, new base tag, plugins rebuilt against the new @superset-ui versions (see the plugin compatibility guide), updated superset_config.py with the Phase 1 actions applied.
  • Run the schema migration against the staging copy:
bash
superset db upgrade
superset init

Timing the rehearsal

Next, time it. On a metadata database with years of logs the logs table can make migrations slow; if it is over a few minutes, plan to prune old logs (reports.prune_log in Celery, or a direct delete of logs rows older than your retention period) before the production run.

  • Run the visualisation migrations for the chart types listed in Phase 1:
bash
superset viz-migrations upgrade --viz_type area
superset viz-migrations upgrade --viz_type bar
# repeat per type named in UPDATING.md for your versions

Each rewrites saved charts’ form data to the new chart type and keeps a record so it can be downgraded with the matching downgrade command if needed before cutover.

  • Migrate filter boxes to native filters using the provided migration, then open every affected dashboard and check the filters do what they did.
  • Start the target image against the migrated staging database. Read the startup log for unknown configuration keys and deprecation warnings. Fix them.
  • Verify using the checklist in Phase 4. Fix what fails, rebuild, and repeat from step 1 with a fresh copy until the rehearsal is clean end to end.

The rehearsal produces the exact command sequence and timing for production, and a list of dashboards and charts that needed attention. So, keep both.

Rehearsed it — now who signs off on cutover?

The rehearsal catches most breakage; the cutover window is where custom plugins and RBAC config quietly slip through. A Superset Architecture Review reviews your specific instance before that window opens.

Book a Superset Architecture Review

Phase 3: Cut over

The migration constraint means true zero-downtime blue-green, where old and new run side by side against the same database, is not safe across a major version: the old version may fail or corrupt data on the new schema. What is achievable is a short read-only window measured in minutes, with a tested rollback.

The night before

  • Announce the window. Ten to twenty minutes is realistic for a rehearsed upgrade.
  • Confirm backups are running and a restore has been tested recently.
  • Pre-pull the target image on every node so the deploy is a restart, not a download.
  • Prune old logs rows if the rehearsal showed the migration was slow.

The sequence

  • Freeze writes. Put the web tier into maintenance (a static page at the load balancer, or scale web replicas to zero). Stop Celery beat so no scheduled reports or warm-ups run mid-migration. Let in-flight Celery tasks finish, then stop workers.
  • Snapshot the metadata database. This is the rollback point. Do not skip it because you have nightly backups; you want the state from thirty seconds ago.
  • Run the migration with the rehearsed commands from a one-off container on the target image: superset db upgrade, superset init, the viz-migrations commands, the filter box migration. Compare the timing to rehearsal.
  • Flush the caches you configured: metadata cache certainly, data cache preferably. Cached metadata from the old version can render incorrectly on the new one.
  • Deploy the target image to workers first, then beat, then web. Health check /health on each web replica.
  • Smoke test while still in maintenance: log in, open two dashboards, run one SQL Lab query, load one embedded dashboard if you have them.
  • Open traffic.

Overall, elapsed time for a rehearsed upgrade on a moderate metadata database: five to fifteen minutes, dominated by the migration.

Rollback

However, if the smoke test fails and the cause is not obvious within a few minutes:

  • Stop the target web, beat and workers.
  • Restore the metadata database from the step 2 snapshot.
  • Redeploy the previous image to workers, beat and web.
  • Flush caches, smoke test, open traffic.

Also, rehearse the rollback at least once in staging. A rollback you have never run is a plan, not a procedure.

Phase 4: Verify

Do this in staging during rehearsal and again in production after cutover. Also, automate as much as you can, because you will do it again next year.

  • Authentication. SSO login, role assignment, an admin action, a viewer’s restricted view. Custom security managers are a common casualty.
  • Top dashboards. Every dashboard in the top twenty by views over the last month, opened and visually checked. A script that requests each dashboard’s chart data endpoints and asserts 200 with rows catches most breakage without eyes.
  • Migrated charts. Every chart touched by viz-migrations, opened in Explore. Colours, stacking and axis formatting are what typically drift.
  • Filters. Every dashboard that had a filter box, with each filter exercised.
  • Custom plugins. Every saved chart using each plugin, plus one new chart created from scratch.
  • SQL Lab. A query on each database connection, one async.
  • Alerts and reports. Trigger one report manually and check the screenshot renders with the new Talisman policy.
  • Embedded dashboards. Load one from the embedding application; the Talisman frame-ancestors change is the number one post-upgrade embedding failure.
  • Performance. Compare the dashboard load times to the numbers you noted before. New defaults occasionally disable a cache you relied on.
  • Logs. Tail the web and worker logs for the first hour for tracebacks and deprecation warnings.

Between major versions

Minor and patch upgrades (4.0 to 4.1, 4.1.1 to 4.1.2) use the same playbook with less reading. The migration step is still one-way, so the snapshot before migration is still mandatory, but the rehearsal can be lighter and the window shorter. So, do them regularly; the longer the gap between upgrades, the more entries in UPDATING.md and the more painful the jump. Overall, quarterly is a good cadence, with security patches applied as they land.

Common failures, from experience

Symptom Cause Prevention
Migration takes an hour Huge logs table Prune before, set a retention policy after
Embedded dashboards blank after upgrade Talisman enabled, no frame-ancestors Configure TALISMAN_CONFIG in Phase 1
Maps show no tiles CSP blocks the tile server Add tile and API hosts to connect-src and img-src
Charts show “Unknown viz type” Legacy chart types removed, migration skipped Run viz-migrations for every listed type
Dashboards lost their filters Filter box removed Run the filter box migration and review each dashboard
Custom plugin missing from picker Not rebuilt against new @superset-ui versions Rebuild in Phase 2
Startup warns about unknown config keys Deprecated settings removed Fix in rehearsal; do not ignore warnings
Login page broken Custom security manager uses a changed FAB API Test auth first in rehearsal
Everything works, dashboards slow A feature flag default changed caching or async behaviour Diff DEFAULT_FEATURE_FLAGS in Phase 1; compare load times in Phase 4

Before the next one

Two habits make the next upgrade cheaper. First, keep your superset_config.py free of anything not deliberately set, so the diff against defaults stays small. Second, keep a UPGRADE.md in your deployment repository with the exact commands, timings and gotchas from this run; the person doing the next upgrade will thank you, and it is often you.

Frequently Asked Questions

What breaks when upgrading Apache Superset from 3.x to 4.x?

In short, the entries most deployments hit are the removal of the legacy filter box, the completed legacy chart migrations (older area, bar and line types must be migrated with superset viz-migrations), Flask-Talisman and its content security policy being on by default, the Python 3.9-or-later requirement, several feature-flag defaults flipping, and removed configuration keys. Read every UPDATING.md entry between your current version and your target, and turn each into a checklist row with “affects us” and “action” columns.

Can a Superset major-version upgrade really be done without downtime?

For read traffic, yes, with a rehearsal against a restored copy of production, a blue-green or rolling cut-over, and a tested rollback. However, the part that is not free is the database migration: schema changes on a large metadata database take time, which is why the rehearsal exists, to measure that time before the change window rather than during it.

What is the most common post-upgrade failure?

Embedded dashboards going blank, because Talisman is on by default in 4.x and frame-ancestors is now mandatory for any origin that frames Superset. Similarly, custom security managers are the other frequent casualty, so SSO login and role assignment are the first things on the verification list rather than the last.

What should be verified after the cut-over?

Authentication end to end, every dashboard in the top twenty by views over the last month, every chart touched by viz-migrations, every dashboard that previously used a filter box, every saved chart from each custom plugin plus one new one, a query per database connection including an async one, one manually triggered report, one embedded dashboard loaded from the embedding application, and a load-time comparison against the numbers you recorded beforehand. Then tail the web and worker logs for the first hour.

Build It With Andolasoft

Upgrade readiness is one of the six areas in our Superset Architecture Review: the version gap, the UPDATING.md entries that apply to your deployment, the plugins and extensions that need rebuilding, and the state of your metadata database. In addition, it is a fixed-scope five-day review with a written report you can run the upgrade from. Under Managed Support we run the upgrades on a schedule, including the rehearsal, so the playbook above is something you read once rather than execute. The production configuration the upgrade has to preserve is covered in Superset in production, and the plugin side in what breaks between 3.x and 4.x for plugins.

If you have an upgrade on the calendar and a deployment people depend on, talk to us. We will tell you what in your specific instance breaks, how long the migration takes on your metadata database, and what the rollback looks like.

Superset Plugin Compatibility: What Breaks Between 3.x and 4.x and How to Migrate a Plugin

Posted on by Anurag Pattnaik
A custom chart plugin is compiled into the Superset frontend bundle. That is what makes it fast and first-class, and it is also why an upgrade can break it: the plugin is built against one version of @superset-ui/core and @superset-ui/chart-controls, and after the upgrade it runs against another. Minor releases rarely matter. Major ones, and the 3.x to 4.x jump in particular, changed enough that most plugins need at least a rebuild and some need real changes.This post is the checklist we work through when we carry a plugin across that line. It assumes you have a plugin structured the way the build guide describes: index.ts, buildQuery.ts, controlPanel.ts, transformProps.ts and a React component, registered in MainPreset.js.

One caveat up front. The exact list of breaking changes depends on the two versions you are moving between. The authoritative source is UPDATING.md in the Superset repository, read for every release between your current tag and your target. What follows is the set that has affected plugins we maintain.

Why plugins break at all

Three kinds of coupling exist between a plugin and the Superset it runs in:

  • Package coupling. The plugin imports from @superset-ui/core, @superset-ui/chart-controls and often @superset-ui/plugin-chart-echarts. These packages live inside the Superset monorepo and are versioned with it. A type that was exported in one version may be renamed or removed in the next.
  • Behavioural coupling. The plugin relies on how Superset builds queries, passes form data, applies filters and handles cross-filter events. Feature flags that flip their default between versions change this behaviour without changing any API.
  • Toolchain coupling. The plugin is compiled by Superset’s webpack, TypeScript and Babel configuration, on the Node version the Superset frontend requires. A stricter TypeScript setting or a newer Node can fail a build that has not changed.

Most upgrade pain comes from the second and third, because they do not show up as type errors.

What changed between 3.x and 4.x that plugin authors notice

Node and build tooling

The 4.x frontend requires Node 18. If your plugin’s CI still runs Node 16, the first symptom is a build failure with no obvious connection to your code. Align the Node version in your plugin repository, your Dockerfile and your CI with superset-frontend/package.json at the target tag.

TypeScript configuration also tightened across the 3.x line. Code that compiled with implicit any or loose null checks can start failing. Fix the types rather than loosening the config; the Superset build uses its own settings, not yours.

Peer dependency pins

Every plugin declares @superset-ui/core and @superset-ui/chart-controls as peer dependencies pinned to specific versions. Those versions moved through the 3.x and 4.x releases. A mismatch produces one of two failures:

  • The package manager resolves two copies of @superset-ui/core into the bundle. Registries and theme contexts are singletons, so the plugin’s copy cannot see the charts, colour schemes or theme registered by Superset’s copy. The symptom is an error along the lines of “theme is undefined” or a chart that never appears in the picker.
  • A type or helper the plugin imports no longer exists at the new version, which is a compile error and at least easy to find.

Update the pins to match the target Superset before doing anything else.

Time range and axis controls

The GENERIC_CHART_AXES feature flag, which lets any column be the x-axis and moves the time range into ad-hoc filters, became the default behaviour in the 3.x line and is the only behaviour in 4.x. Plugins built for 2.x or early 3.x often still use the legacy time section:

typescript
controlPanelSections: [
  sections.legacyRegularTime,
  // ...
]

This still compiles, and for regular (non-time-series) charts it still works. For time-series plugins it produces a control panel that does not match the rest of the product: users expect a time column and a time grain as controls, and the time range as a filter. Move to the generic axis pattern used by the ECharts time-series plugins: an x_axis control, a time_grain_sqla control, and no legacy time section. Your buildQuery then reads formData.x_axis and includes it in columns rather than assuming a __timestamp column.

Filter box removal

Superset 4.0 removed the legacy filter box chart. Plugins are rarely affected directly, but dashboards that used filter boxes to drive your chart now drive it through native filters, which arrive in extra_form_data on the query object. If your buildQuery manipulated extras or filters by hand, check that native filter values still reach the query. The buildQueryContext helper handles this correctly; hand-rolled query construction sometimes does not.

Cross-filter behaviours

Cross-filtering matured across 3.x and is on by default in 4.x. A plugin that declares Behavior.InteractiveChart is expected to emit and respond to cross-filters properly. Two things to check:

  • Emitting. The plugin should call setDataMask with both extraFormData.filters and filterState.value when the user selects an element, and clear both on deselect. Plugins that set only one of these leave the dashboard in an inconsistent state.
  • Receiving. When another chart filters this one, the filter arrives through the normal query path. Verify the chart re-queries rather than showing stale data.

If the plugin cannot participate in cross-filtering, remove Behavior.InteractiveChart from its metadata so the dashboard does not offer the option.

Theme tokens and styling

Superset’s theme object gained and reorganised tokens through 3.x. Plugins that reach into theme.colors.* or theme.gridUnit should be checked against the theme type exported by the target @superset-ui/core. Hard-coded colours keep working but look wrong in a customised deployment, which is the moment stakeholders notice.

Note for readers on the 5.x line: Superset 5 replaced the theme system with Ant Design 5 design tokens and a JSON theme format. That is a larger change than anything in 3.x to 4.x and deserves its own migration pass.

Content Security Policy

4.0 turned on Flask-Talisman and its Content Security Policy by default. Plugins that load scripts, fonts or images from external domains, for example a map tile server or a CDN-hosted library, are blocked unless TALISMAN_CONFIG allows those origins. This is a deployment change rather than a plugin change, but the plugin author is usually the one who has to explain the blank map.

Legacy visualisation migrations

4.0 completed the migration of several legacy charts to their ECharts replacements and removed the old implementations. This matters to plugin authors in one specific case: if your plugin extended or copied code from a legacy chart, the shared code it borrowed may be gone. Search your plugin for imports from legacy-plugin-chart-* packages.

Got more than one custom plugin to check?

Auditing every plugin against this list scales badly past two or three. A Plugin Scoping Call gets you a migration estimate across your whole plugin set, not just one at a time.

Book a Plugin Scoping Call

How to find out which of these affect you

Do this before touching any code.

  • Read `UPDATING.md` from your current tag to the target tag. Note every entry that mentions the frontend, feature flags, @superset-ui, or chart behaviour.
  • Diff the feature flag defaults between the two versions in superset/config.py. Any flag that flipped and touches charts or dashboards is a behavioural change your plugin will see.
  • Check the two package versions of @superset-ui/core and @superset-ui/chart-controls in superset-frontend/package.json at both tags, and read the changelogs for those packages.
  • Grep your plugin for the things listed above: legacyRegularTime, __timestamp, setDataMask, theme.colors, hard-coded colours, external URLs, and any import from a legacy- package.

You now have a list. Usually it is short.

Migration sequence

Work in this order; each step gives you a stable checkpoint.

  • Branch the plugin and name the branch for the target Superset version.
  • Bump the peer dependencies and dev dependencies to the target versions. Set Node to the target version.
  • Build the plugin alone (npm run build). Fix compile and type errors. This clears the package coupling.
  • Link it into a Superset checkout at the target tag and run the dev server. Open the chart in Explore. This surfaces registry and theme problems immediately.
  • Work through the behavioural list: time controls, filters, cross-filters, CSP. Test each in a dashboard, not only in Explore, because filters and cross-filters only exist on dashboards.
  • Restore or re-record thumbnails if the chart’s appearance changed.
  • Run the tests and update fixtures for any changed ChartProps shape.
  • Build the production image from the target tag with the plugin included, deploy to staging, and load every saved chart that uses the plugin. A saved chart carries the form data it was created with; charts saved under the old control panel are where migration bugs hide.

Budget half a day for a minor version and one to two days for a major one, assuming a single plugin of moderate complexity. Multiply for plugins that render maps or use WebGL.

Tests that make the next upgrade cheaper

  • Golden `ChartProps` fixtures. Save a real chartProps object from the running product for each chart configuration you support and assert transformProps output against it. When the shape changes, the diff tells you exactly what moved.
  • Saved-chart smoke test. A script that lists every saved chart with your viz_type and loads each one’s data endpoint. Run it against staging after every upgrade.
  • Control panel snapshot. A test that renders the control panel config and snapshots the control names. A rename in sharedControls shows up here instead of in production.

Keeping the fork thin

The lines you maintain inside the Superset repository should be two: the dependency in superset-frontend/package.json and the registration in MainPreset.js. Every upgrade is a rebase of those two lines onto the new tag. If your fork has grown beyond that, the upgrade cost is no longer about the plugin, and it is worth moving the extra code into the plugin package or a separate extension before the next major version.

Frequently Asked Questions

Will my Superset 3.x plugin work on 4.x without any changes?

Sometimes, but do not plan for it. A plugin that only uses stable @superset-ui/core exports and no time-range controls often survives untouched. Anything that uses the legacy time section, the filter box, hard-coded colours or the older cross-filter callbacks will need work. The only reliable answer comes from building the plugin against the target version and reading the errors.

What is the most common cause of a plugin breaking after a Superset upgrade?

Two copies of @superset-ui/core in the same bundle. Registries and theme contexts are singletons, so when the plugin’s copy differs from Superset’s copy, the plugin cannot see the charts, colour schemes or theme that Superset registered. It usually surfaces as “theme is undefined” or a chart that never appears in the picker. Align the peer dependency pins before investigating anything else.

The filter box was removed in 4.x. Do I have to rewrite my plugin?

Only if your plugin depended on it. The filter box was a chart type, not a plugin API, so most custom charts are unaffected. What does change is the surrounding dashboard: filters now come from native dashboard filters and cross-filters, so a plugin that read filter state the old way needs updating to the current hooks.

How long does a 3.x to 4.x plugin migration usually take?

For a single plugin with current dependency pins and no legacy time controls, a day is realistic, most of it spent on the build tooling and the test pass. Plugins that span more than one major version, borrow from legacy chart code, or have no tests take substantially longer, and the estimate is worth getting before the upgrade is scheduled rather than after.

Build It With Andolasoft

If the plugin was written by someone who has left, if it borrows from legacy chart code, or if the upgrade spans more than one major version, the cheapest first step is a scoping conversation rather than a spike. Our Plugin Scoping Call is free and 45 minutes; you leave with a written estimate of the migration effort. The wider service is described on the Superset plugin development page, and if the upgrade itself is the worry, the Architecture Review covers upgrade readiness for the whole deployment.

If a Superset upgrade is already on the calendar and the custom charts are the unknown, book a Plugin Scoping Call. We will review the plugins you have, tell you which of the changes above apply to each, and give you a written migration estimate before you commit to an upgrade date.