Tag: BFSI

Why the prompt is the blocker

A text-to-SQL request is not just the user’s question. To write SQL that runs, the model needs to know what exists. For example, the model needs table and column names, types, and which columns are measures or dimensions. It often also needs metric definitions and the conversation so far. For a regulated organisation every part of that is sensitive:

• The question. “Show NPA movement for accounts linked to customer group X” names a customer group. Questions leak intent, investigations and names.
• The schema. Column names such as aadhaar_hash or diagnosis_code describe what you hold and how you process it. Outsourcing and data-protection reviews treat that as information about personal data processing, not harmless metadata.
• Sample values. If the AI layer sends distinct values so the model can match “Bhubaneswar” to a branch code, it is sending data.
• Results. Some designs send query results back to the model to write a narrative. If those are row-level, that is a data transfer.

Good design shrinks the payload: send aggregates rather than rows, never send sample values from sensitive columns. It does not take it to zero. The question and a schema description always have to reach the model. As a result, for many regulated teams, that means a model inside the perimeter and no path to any public AI endpoint.

The reference architecture

Three components, each with one job.

Apache Superset is the governed execution layer. It holds the database connections, datasets, roles, row-level security rules, query timeouts and cache. Every SQL statement runs through Superset’s permissions as the user who asked. Therefore, it does not matter whether a person wrote it or a model did. This is the most important design choice in the stack: the model never gets a database connection of its own.

The AI service sits between the user and Superset. It works out which datasets are relevant, builds the prompt from a schema description and calls the model. It then validates the returned SQL (read-only, within allowed datasets, row-limited) and submits it for execution. Finally, it asks the model to choose a chart and explain the aggregated result. It also keeps conversation state so follow-ups inherit the previous filters.

The model server runs the model on your hardware and exposes an OpenAI-compatible HTTP API. That property matters more than which server you choose. If the AI service speaks the OpenAI chat completions protocol, switching model servers needs no code. Moving from a hosted provider to a local server is just a change of base URL and model name. Similarly, switching between local servers works the same way.

How one question flows through the stack

The request path for one question:

• The user asks a question, authenticated through the same identity provider as Superset.
• The AI service selects candidate datasets the user’s role can already see and builds a prompt from the question and their description.
• The prompt goes to the local model server over the internal network.
• The returned SQL is parsed, checked, and executed through Superset under the user’s permissions, so row-level security applies.
• The aggregated result, not raw rows, goes back to the model to pick the visual and draft the insight.
• Chart and text return to the user. Nothing crosses the perimeter.

Steps 2 and 4 are where governance lives. Suppose the model is only offered datasets the user is entitled to, and Superset enforces the same entitlements at execution. Then a cleverly worded question cannot reach data the user could not query by hand. If your role model is not yet trustworthy enough for that, fix it first; our notes on RBAC design for a 500-user enterprise are a reasonable start.

Ollama or vLLM

Both are open-source, both run offline once the weights are present, and both expose OpenAI-compatible endpoints. Ollama serves /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models on port 11434. vLLM’s vllm serve starts an OpenAI-compatible server on port 8000 with chat completions, completions, embeddings and more. The AI service can talk to either; the differences are in what each is optimised for.

Which one to start with

Ollama is the right first step. A pilot with a handful of analysts or a department of a few dozen users on one server fits it well. Importing a model from local files is a Modelfile and one command. Before anyone else uses it, bind it to an internal interface with OLLAMA_HOST, and raise OLLAMA_NUM_PARALLEL deliberately: its documentation notes that required memory scales with parallel requests times context length.

vLLM is the production answer once concurrency matters. When a branch network starts asking questions at 09:30 on a Monday, continuous batching is what keeps the server busy and the queue moving, and multi-GPU parallelism handles models that do not fit on one card.

Authentication is your job in both cases. Ollama’s local server does not check keys, and vLLM’s documentation says --api-key should not be relied on alone. Put the model server on a private segment reachable only from the AI service, behind a reverse proxy that terminates TLS, enforces authentication and logs requests. Treat it like a database port.

Sizing: the considerations, not a benchmark

We will not quote tokens per second for a model on a card. Those numbers depend on model, quantisation, context length, concurrency, driver and server version, and they go stale within months. How to reason about memory does not.

Weights. Roughly parameter count times bytes per parameter: two bytes at 16-bit, half that at 8-bit, about half again at 4-bit. That arithmetic tells you quickly whether a model fits on the GPUs you have.

KV cache and context. Each concurrent request holds attention state that grows with context length, and text-to-SQL prompts are long because schema descriptions are long. Ollama defaults to a 4,096-token context; a prompt describing a dozen wide tables can exceed it, and a truncated schema produces confident, wrong SQL. Measure real prompt sizes from the AI service, set the context to fit, then size memory for that context at your expected concurrency.

Capability versus size. Larger models generally write better SQL against unfamiliar schemas, but a curated semantic layer narrows the gap. Clear dataset and column descriptions and certified metrics in Superset help every model. Before buying hardware for a bigger one, check whether the smaller one fails because your schema is ambiguous.

Latency budget. One question means at least two model calls (SQL, then chart and explanation) plus the warehouse query. Decide what end-to-end time users will accept and work backwards.

The practical step is a short sizing exercise on your own hardware: a few candidate models at a couple of quantisation levels, run against your question set at expected concurrency, measured end to end. It takes days, and the numbers are yours.

Offline packaging: nothing calls home

An air gap is a property of the whole install, not the model server. Everything normally pulled from the internet has to come through your transfer process, and everything that reaches out has to be found and disabled.

Images. Pull every image on a connected build machine (Superset, its metadata database and Redis, the AI service, the model server), scan, sign and push to the internal registry. Pin versions and digests.

Model weights. Download on the connected side, record checksums, verify on arrival. For Ollama, build the model from a local GGUF file or Safetensors directory with ollama create, and point OLLAMA_MODELS at the volume where models should live. For vLLM, serve from a local directory path and set HF_HUB_OFFLINE=1 so it does not try to resolve anything from the Hugging Face Hub.

Superset outbound calls and dependencies

Superset’s own outbound calls. Superset’s FAQ states that it uses Scarf by default to collect basic telemetry through three channels: the Scarf Gateway for container images, the scarf-js npm package, and a page-view pixel. SCARF_ANALYTICS=false disables the pixel at runtime on the pre-built images; the documentation is explicit that the other two channels are opted out separately. Pulling images from your own registry removes the gateway path, and building the frontend yourself is where you control the npm package. Mapbox-based map charts need a MAPBOX_API_KEY for the external Mapbox service, so disable them or plan an internal alternative.

Dependencies. If you build any image yourself, mirror the Python and npm indexes internally. A build that fails in the air gap is the good outcome; one that succeeds because someone allowed an exception is the bad one.

Prove it. Run the full stack in a staging segment with default-deny egress and log every blocked connection for a week of real use. The list should be empty. Each entry is a setting to change or a component to replace.

Where DashboardGPT fits

DashboardGPT is the product we built on this architecture. It is managed Apache Superset 6.0 hosting with an AI layer: users ask questions in plain English, Superset runs the query, and the answer comes back as a chart with a written explanation. The default is a managed cloud workspace. For regulated teams, the on-premise edition installs with Docker Compose on a single host or on Kubernetes for larger rollouts, and the air-gapped option ships as an offline bundle and licence file, with the language model on your own GPUs and no outbound calls.

It makes the design choices described above. Only the question and a description of the relevant schema go to the model; row-level data does not, because the query runs in the analytics layer and only aggregates come back. On-premise installs point at Ollama, vLLM or any OpenAI-compatible endpoint, so the server choice stays yours. It is its own Superset workspace rather than a plugin for an existing instance: teams already running Superset often keep it for governed dashboards and use DashboardGPT for ad-hoc questions.

Evaluation and updates inside an air gap

A model that writes good SQL in January can write worse SQL after an upgrade, and in an air gap you cannot try the new version in production and roll back tomorrow. Treat models as versioned dependencies.

Build a question set early. Collect 100 to 200 real questions from pilot users, each with an answer an analyst agrees is correct. Include relative dates (“last quarter”, “month to date”), ambiguous terms your metric definitions resolve, questions that should be refused because they reach outside the user’s datasets, and follow-ups that depend on the previous answer.

Score on results, not text. Two correct SQL statements rarely match character for character. Execute both against a fixed data snapshot and compare results. Track pass rate, correct refusals, invalid SQL and latency. The harness runs inside the air gap, because the questions and snapshot are sensitive.

Promote like software. A new model, quantisation, Ollama or vLLM release, or prompt template change is a release: transferred, checksummed, evaluated in staging, promoted only if it matches or beats the current version. Keep the previous weights and image so rollback is a configuration change. Because the AI service talks to an OpenAI-compatible endpoint, old and new models can run side by side under different names.

Fix the semantic layer first. When the model gets a question wrong, the fix is often a better column description, a certified metric or a clearer dataset name, not a bigger model. Those fixes help every model you will ever run.

Agree the cadence. Quarterly model and server updates are common; security patches to Superset, the model server or base images follow your emergency change process.

Frequently Asked Questions

Can Apache Superset use a local LLM for text-to-SQL?

Yes, through a text-to-SQL service in front of Superset that calls a model server you run yourself. Superset remains the execution layer, enforcing roles and row-level security on every query. The model server, typically Ollama or vLLM, exposes an OpenAI-compatible API, so the AI service calls it as it would a hosted provider, with a different base URL and no internet access.

Should we use Ollama or vLLM for an air-gapped deployment?

Ollama is simpler for pilots and smaller groups on one server. vLLM is built for high-throughput serving, with continuous batching and multi-GPU parallelism, which matters once many users ask questions at once. Both expose OpenAI-compatible endpoints, so teams can start on Ollama and move to vLLM without changing the AI service.

What does a text-to-SQL prompt actually contain?

At minimum the user’s question and a description of the relevant schema: table and column names and types, often metric definitions and the conversation so far. Some designs also send sample values or query results. In a regulated environment all of that is sensitive, which is why the model needs to run inside the network.

Does Apache Superset make outbound calls that need to be disabled?

By default Superset collects basic telemetry through Scarf: the container image gateway, the scarf-js npm package and a page-view pixel. SCARF_ANALYTICS=false disables the pixel, and pulling images from your internal registry avoids the gateway. Mapbox-based map charts depend on the external Mapbox service. Test in a default-deny egress segment before calling the stack air-gapped.

Build It With Andolasoft

We design and run Apache Superset for organisations that cannot put their data, or their questions about it, on the public internet. A Superset Architecture Review covers what this post depends on: your role and row-level security model, the semantic layer the model will read, egress and telemetry settings, and where an AI service and model server would sit in your network. Our AI services team handles model selection, the evaluation harness and the sizing exercise on your hardware, and Superset Managed Support can run the update and promotion cycle afterwards.

If you have been asked for AI analytics and your answer so far has been “not on a public API”, talk to us. We will work out what a local-model deployment looks like in your environment, and whether DashboardGPT’s on-premise edition or a custom AI service alongside your existing Superset is the better fit.