great_expectations/.agents/skills/gx-configure-data-source/SKILL.md
This skill takes a user from "here is my data" to a verified working batch definition — a named, saved way of pulling a specific slice of their data that you have proven works by actually reading through it.
Three objects, in order, each built on the one before:
Everything here goes through Great Expectations' public configuration API and produces ordinary project artifacts. Nothing depends on this skill being present afterwards.
references/preflight.md.Do not skip step 1, and do not stop before step 4. A configuration that was never read through is not a result worth reporting.
Follow references/preflight.md in full before configuring anything. It
establishes whether you are working against a project on disk or an in-memory
session, tells you what to announce to the user, and covers the environment
problems that silently masquerade as "no project found".
The outcome you carry forward is a context object and one fact: whether the
session is file-backed or in memory. Both are fully supported paths.
If anything here, or in step 3 below, turns up a missing driver or client library, report it and hand over the install command per the standing rule above. Do not install it yourself — not now, before anything has failed, and not later without the user's own go-ahead.
Four things, in this order. Ask for them together where you can; don't interrogate the user one field at a time.
1. Which data source type. Read the shipped catalog rather than working
from memory — see references/datasource-catalog.md. It gives you the exact
factory name for each type, the arguments that type accepts, and which of them
are mandatory. It also covers how to steer the conversation when the user
describes their data rather than naming a backend.
2. The connection details the chosen type's schema marks as required. Read the secrets rule below before you write any of them down.
3. What the asset is. A table name, a SQL query, a file-name pattern, or a dataframe variable in the user's session. Which of these are available depends on the type; the catalog reference enumerates them.
4. The batching cadence. Whole collection, or sliced by day, month, or year. Ask what the natural unit of the user's data is — if they check data daily, a daily batch definition matches how they work.
A user who expresses no preference has not chosen the whole collection.
Don't let silence settle it. Carry the question forward to step 4 instead,
where the probe's own output tells you whether the data can carry a time slice
at all, and an open question becomes a recommendation: "ordered_at looks
like your time column — monthly?" is something a user can answer, where "how
would you like to batch this?" mostly is not.
If nothing in the data can carry a time slice, the whole collection is the right answer and there is nothing to apologize for — say that you looked and found nothing, so the user hears a finding rather than a default.
Connection strings routinely carry passwords, tokens, and account identifiers. Great Expectations already has a mechanism for this, and it is the only one to use.
Put a ${VARIABLE_NAME} reference in the configuration; never the literal
value. The reference is what gets stored, and it is resolved at connection
time:
connection_string = "postgresql+psycopg2://${DB_USER}:${DB_PASSWORD}@warehouse.internal:5432/analytics"
Three rules follow from this, and none of them bend:
${DB_PASSWORD}), never what it resolved to.Where the variable is read from depends on the session, and the split matters:
${VARIABLE_NAME} from process
environment variables only. It has no project on disk, so it has no
uncommitted config file to read from. If the user wants to supply the value,
it has to be an environment variable.Name whichever applies when you ask for a credential, so the user knows where
to put it. references/write-out.md covers what changes when an in-memory
session later becomes a project.
The user may be adding to a project that already has some of this configured — possibly from a teammate, a previous conversation, or an earlier run of this same flow. Fetch each object before creating it, and create only what is missing.
Adding a duplicate is not an option the API offers: add_<type>_asset and
add_batch_definition_* raise ValueError: "<name>" already exists on a
second call with the same name. There is no add_or_update_* variant for
assets or batch definitions.
All three fetch calls signal absence with a LookupError subclass, so a
single except LookupError is the correct catch for each:
| Fetch | Raises when missing |
|---|---|
context.data_sources.get(name) | KeyError (a LookupError) |
datasource.get_asset(name) | LookupError |
asset.get_batch_definition(name) | KeyError (a LookupError) |
add_or_update_<type> replaces a data source wholesale. It does not merge.
Calling it against a name that already exists drops every asset and batch
definition attached to that data source — including ones this session never
created. A run that "updates" a data source in order to add one asset to it
destroys the user's other assets silently: no error, no warning, and the flow
carries on and reports success.
Verified directly: a data source carrying assets ['orders', 'products'] has
[] assets immediately after a second add_or_update_sqlite under the same
name.
So:
Note also that the factory tests the connection as part of the call, so it
can be slow or hang on an unreachable host. Run it inside the duration-tracked
wrapper in references/robustness.md like any other data-touching call.
That connection test is exactly where a missing driver or client library surfaces. Report it and hand over the install command — do not install it yourself to get the connection working, and do not offer to.
DATASOURCE_NAME, ASSET_NAME, BATCH_DEFINITION_NAME = "warehouse", "orders", "by_month"
# --- data source: create only if absent; never replace an existing one ---
try:
datasource = context.data_sources.get(DATASOURCE_NAME)
except LookupError:
datasource = context.data_sources.add_or_update_sqlite(
name=DATASOURCE_NAME,
connection_string="sqlite:///${WAREHOUSE_PATH}",
)
# --- asset: reuse, or delete and re-add when its configuration must change ---
replace_asset = False # set True only if the user wants different asset config
try:
asset = datasource.get_asset(ASSET_NAME)
asset_exists = True
except LookupError:
asset_exists = False
if asset_exists and replace_asset:
datasource.delete_asset(ASSET_NAME)
asset_exists = False
if not asset_exists:
asset = datasource.add_table_asset(name=ASSET_NAME, table_name="orders")
# --- batch definition: same shape ---
replace_batch_definition = False
try:
batch_definition = asset.get_batch_definition(BATCH_DEFINITION_NAME)
batch_definition_exists = True
except LookupError:
batch_definition_exists = False
if batch_definition_exists and replace_batch_definition:
asset.delete_batch_definition(BATCH_DEFINITION_NAME)
batch_definition_exists = False
if not batch_definition_exists:
batch_definition = asset.add_batch_definition_monthly(
name=BATCH_DEFINITION_NAME, column="ordered_at"
)
Two things to keep intact when you adapt this:
except of the fetch's own try. Keeping them apart means a failure in
the create is reported as a create failure rather than being swallowed by
the same handler.Set replace_asset / replace_batch_definition from what the user actually
asked for. If they want a different table, different files, or a different
partitioning column than what is already configured under that name, the
existing object has to be replaced. If they described what is already there,
reuse it and say so.
Retrieving a batch does not prove the configuration works. For a SQL query
or table asset, building the batch touches nothing — get_batch() returns a
real Batch object even when the table does not exist. Reporting success on
that basis means telling the user their setup works when it does not.
Always follow retrieval with a probe that actually reads data:
batch = batch_definition.get_batch() # add batch_parameters=... for a partitioned definition
head = batch.head(n_rows=5) # this is the step that touches the data
head is a small object with a .data attribute holding a pandas DataFrame
of the sampled rows; printing it renders the rows directly.
Run this probe inside the duration-tracked, exception-catching wrapper in
references/robustness.md. Do not write your own — that reference already
handles the parts that are easy to get wrong: checking in with the user while
a slow query is still running rather than after it returns, never cancelling
work that continues to run and bill on the data platform, and recovering the
real database error out of the bare KeyError a broken probe raises.
Four outcomes, and three of them are not failures:
head returns a frame with the
right columns and no rows. Say both things plainly. Do not report it as a
configuration failure, and do not start changing the configuration to make
rows appear.get_batch() raises NoAvailableBatchesError: No available batches found. — the configuration works, but the window you asked for holds no
data. Both partitioned SQL assets and file-based assets fail this way when
the requested year/month matches nothing, and it is the normal answer for a
window outside the data's range. Report it as an empty window, name the
batch parameters you used, and offer to try a window that exists — not as a
broken setup.references/robustness.md's rules: what failed, why as far as it is
known, one concrete next step. A broken table or query surfaces here as a
bare KeyError with no readable message; that reference explains how to
recover the real database error behind it. Do not report success, and do not
retry with different parameters hoping something sticks.If the cadence is still open from step 2, close it here. The probe's frame
is the survey you needed: head.data.dtypes names every column and its type,
so a date or timestamp among them is the recommendation to bring back. A
whole-collection definition standing where a time column exists should be
something the user chose, not something they were never offered — say what you
found, say what it would slice by, and let them decide. Switching is the
replace_batch_definition path in step 3, which leaves the asset and every
other batch definition on it untouched.
Tell the user, concretely:
batch_parameters the batch
definition needs.If the session is in memory, offer to write it out to a real project so
the work survives — see references/write-out.md for the procedure and for
what the user needs to know about dataframe assets, which carry configuration
but no data. Offer it; don't do it unprompted, and don't pick the location.
The offer ends this flow. Write-out creates a project on the user's disk,
so it starts only from their reply — a separate run, after they have agreed
and named a directory. Reporting the result and then writing it out in the
same breath means they were never asked; putting the write-out call in the
same program as the configure and verify calls means the same thing, because
there was no point in it where an answer could have arrived.
references/write-out.md opens with the gate this depends on.
The verified batch definition is the end state. Do not build a suite of "smoke test" expectations to prove the setup works — the probe in step 4 has already proven it, an invented expectation asserts something the user never asked for, and a failing one would report a data-quality problem that is really just a guess of yours.
When the user wants to assert things about their data, that is expectation
work: hand off to the gx-configure-expectations skill with the batch
definition you just verified. If they ask for both in one breath, finish this
flow, report it, and then move on.
Each of these shows the object chain — data source, asset, batch definition, verified batch. Preflight comes first in all three.
All three carry step 3's fetch-first guard on the data-source call, because
these are the blocks that get copied. A bare add_or_update_<type> sitting in
an example reads as the shape to follow, and followed against a name that
already exists it silently drops every asset on that data source — the caveat
in the prose above it does not survive the copy. The asset and
batch-definition calls are left bare to keep the per-backend differences
legible: those raise on a name that already exists rather than destroying
anything, and step 3 carries the reuse pattern for them.
Take the factory names and arguments for the user's actual backend from
references/datasource-catalog.md.
The date lives in the file name, so the monthly batch definition takes a
regex with named groups. Batch parameters are integers, and they are not
padded to match the file name: {"month": 2} selects sales_2024-02.csv, even
though the regex reads that group as two zero-padded digits.
try:
datasource = context.data_sources.get("sales_files")
except LookupError:
datasource = context.data_sources.add_or_update_pandas_filesystem(
name="sales_files",
base_directory="/data/sales",
)
asset = datasource.add_csv_asset(name="monthly_sales")
batch_definition = asset.add_batch_definition_monthly(
name="by_month",
regex=r"sales_(?P<year>\d{4})-(?P<month>\d{2})\.csv",
)
batch = batch_definition.get_batch(batch_parameters={"year": 2024, "month": 2})
print(batch.head(n_rows=5))
The date lives in a column, so the monthly batch definition takes column.
Batch parameters are integers here too — the two families take the same window,
so one set of parameters drives both. The credential-bearing part of the
connection string is a ${VARIABLE_NAME} reference, never a literal.
try:
datasource = context.data_sources.get("warehouse")
except LookupError:
datasource = context.data_sources.add_or_update_postgres(
name="warehouse",
connection_string="postgresql+psycopg2://${DB_USER}:${DB_PASSWORD}@warehouse.internal:5432/analytics",
)
asset = datasource.add_table_asset(name="orders", table_name="orders")
batch_definition = asset.add_batch_definition_monthly(name="by_month", column="ordered_at")
batch = batch_definition.get_batch(batch_parameters={"year": 2024, "month": 3})
print(batch.head(n_rows=5))
A query asset is the alternative when the logical collection is not a whole
table — datasource.add_query_asset(name=..., query=...). Use it to express
what the data is, not to bolt a LIMIT onto a slow table; see
references/robustness.md for why row-limited query assets are a last-resort
exploration tool rather than a batching mechanism.
A dataframe asset stores configuration only — the data itself is handed over at retrieval time, every time, in this session and in every future one:
try:
datasource = context.data_sources.get("in_memory")
except LookupError:
datasource = context.data_sources.add_or_update_pandas(name="in_memory")
asset = datasource.add_dataframe_asset(name="customers")
batch_definition = asset.add_batch_definition_whole_dataframe(name="all_rows")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
print(batch.head(n_rows=5))
Say this out loud to the user when you configure one: the dataframe is not saved, and anything reading this asset later must supply a dataframe itself.
references/preflight.md — establishing and announcing the session context.references/datasource-catalog.md — every configurable type, its factory,
its arguments, and its asset and batch-definition surface, read from the
shipped catalog.references/robustness.md — the time-budget wrapper, scope-reduction
levers, and how to report a failure helpfully.references/write-out.md — turning an in-memory session into a real
project.