dev/README_RELEASE_PYTHON_CLIENT.md
Table of Contents generated with DocToc
We are keeping track of security issues in the Security Issues repository currently. As a release manager, you should have access to the repository. Please review and ensure that all security issues marked for the release have been addressed and resolved. Ping security team (comment in the issues) if anything missing or the issue does not seem to be addressed.
Additionally, the dependabot alerts and code scanning alerts should be reviewed and security team should be pinged to review and resolve them.
The client versioning is independent of the Airflow versioning.
The Python client is generated using Airflow's openapi spec. To update the client for new APIs do the following steps:
# If you have not done so yet
git clone [email protected]/apache/airflow
cd airflow
# Checkout the right branch
git checkout v2-8-test
export AIRFLOW_REPO_ROOT=$(pwd -P)
export TEST_BRANCH=v2-8-test
export STABLE_BRANCH=v2-8-stable
cd ..
git checkout ${TEST_BRANCH}
# Find the commit from where to cut off and check it out
git checkout COMMIT_NUM
# Create the sync branch
git checkout -b changes313rc1
export SYNC_BRANCH="changes313rc1"
# If you have not done so yet
git clone [email protected]:apache/airflow-client-python
cd airflow-client-python
# Checkout the right branch
git checkout main
export CLIENT_REPO_ROOT=$(pwd -P)
cd ..
clients/python/version.txt in the Airflow repository.
Set the VERSION and VERSION_SUFFIX environment variables. Note that version.txt should contain
the target version - without the suffix.cd ${AIRFLOW_REPO_ROOT}
export VERSION="3.1.3"
export VERSION_SUFFIX="rc1"
export VERSION_RC=${VERSION}${VERSION_SUFFIX}
echo "${VERSION}" > clients/python/version.txt
cd ${AIRFLOW_REPO_ROOT}
git log 2.8.0..HEAD --pretty=oneline -- airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
cd ${AIRFLOW_REPO_ROOT}
git add .
git commit -m "Prepare release ${VERSION_RC}"
Create a PR where you add the commit above in main branch and cherry-pick it to the v2-test branch.
This PR should also contain the change to clients/python/version.txt. Due to that, you might want to do this PR from
the cloned airflow repo for the release and not your dev airflow clone.
Merge it to the v2-*-stable branch with the command below. You will release API client from the latest v2-*-stable branch
of Airflow repository - same branch that is used to release Airflow:
git checkout ${STABLE_BRANCH}
# make sure you are up to date
git fetch origin ${STABLE_BRANCH}
git reset --hard origin/${STABLE_BRANCH}
# merge the changes from the sync branch
git merge --ff-only ${SYNC_BRANCH}
# push the changes to the stable branch
git push origin ${STABLE_BRANCH}
Build the sdist and wheel packages to be added to SVN and copy generated client sources to the Python Client repository.
cd ${AIRFLOW_REPO_ROOT}
rm dist/*
breeze release-management prepare-python-client --distribution-format both --python-client-repo "${CLIENT_REPO_ROOT}" --version-suffix ""
This should generate both sdist and .whl package in dist folder of the Airflow repository. It should
also override the client sources in the Python Client repository.
Commit generated code to the main branch of the Python Client repository. No need to have code reviews
for that code if it is automatically generated. However make sure that git diff HEAD shows expected
changes only.
cd ${CLIENT_REPO_ROOT}
git diff HEAD
git checkout -b release-${VERSION}
git add .
git commit -m "Update Python Client to ${VERSION_RC}"
git push apache release-${VERSION}
Then open a PR and merge it into main.
cd ${AIRFLOW_REPO_ROOT}
git tag -s python-client/${VERSION_RC} -m "Airflow Python Client ${VERSION_RC}"
git push apache python-client/${VERSION_RC}
cd ${CLIENT_REPO_ROOT}
git tag -s ${VERSION_RC} -m "Airflow Python Client ${VERSION_RC}"
git push apache tag ${VERSION_RC}
cd ${AIRFLOW_REPO_ROOT}
breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}"
cd ${AIRFLOW_REPO_ROOT}
pushd dist
../dev/sign.sh *
popd
# First clone the repo somewhere if you have not done it yet
[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
svn update --set-depth=infinity asf-dist/dev/airflow
# Create new folder for the release
cd asf-dist/dev/airflow/clients/python
svn mkdir ${VERSION}${VERSION_SUFFIX}
# Move the artifacts to svn folder & commit
mv ${AIRFLOW_REPO_ROOT}/dist/apache_airflow_client-* ${VERSION_RC}/
mv ${AIRFLOW_REPO_ROOT}/dist/apache_airflow_python_client-* ${VERSION_RC}/
cd ${VERSION_RC}
svn add *
svn commit -m "Add artifacts for Apache Airflow Python Client ${VERSION_RC}"
# Remove old version
cd ..
export PREVIOUS_VERSION_WITH_SUFFIX=2.8.0rc1
svn rm ${PREVIOUS_VERSION_WITH_SUFFIX}
svn commit -m "Remove old Apache Airflow Python Client ${PREVIOUS_VERSION_WITH_SUFFIX}"
At this point we have the artefact that we vote on, but as a convenience to developers we also want to publish "snapshots" of the RC builds to pypi for installing via pip. Note that these packages must be generated with RC version.
To do this we need to:
cd ${AIRFLOW_REPO_ROOT}
rm dist/*
breeze release-management prepare-python-client --distribution-format both --version-suffix "${VERSION_SUFFIX}"
twine check dist/*
twine upload -r pypi dist/*
Subject:
cat <<EOF
[VOTE] Release Apache Airflow Python Client ${VERSION} from ${VERSION_RC}
EOF
Body:
cat <<EOF
Hey fellow Airflowers,
The release candidate for Apache Airflow Python Client ${VERSION_RC} is now available for testing!
This email is calling for a vote on the release, which will last at least 72 hours, from Friday, October 8, 2025 at 4:00 pm UTC
until Monday, October 11, 2025 at 4:00 pm UTC, and until 3 binding +1 votes have been received.
https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20211011T1600&p1=1440
Consider this my (binding) +1.
Airflow Client ${VERSION}${VERSION_SUFFIX} is available at:
https://dist.apache.org/repos/dist/dev/airflow/clients/python/${VERSION_RC}/
The apache_airflow_client-${VERSION}.tar.gz is an sdist release that contains INSTALL instructions, and also
is the official source release.
The apache_airflow_client-${VERSION}-py3-none-any.whl is a binary wheel release that pip can install.
Those packages do not contain .rc* version as, when approved, they will be released as the final version.
The rc packages are also available at PyPI (with rc suffix) and you can install it with pip as usual:
https://pypi.org/project/apache-airflow-client/${VERSION_RC}/
Public keys are available at:
https://dist.apache.org/repos/dist/release/airflow/KEYS
Only votes from PMC members are binding, but all members of the community
are encouraged to test the release and vote with "(non-binding)".
The test procedure for PMC members is described in:
https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PYTHON_CLIENT.md#verify-the-release-candidate-by-pmc-members
The test procedure for contributors and members of the community who would like to test this RC is described in:
https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PYTHON_CLIENT.md#verify-the-release-candidate-by-contributors
*Changelog:*
*Major changes:*
...
*Major fixes:*
...
*New API supported:*
...
Cheers,
<your name>
EOF
PMC members should verify the releases in order to make sure the release is following the Apache Legal Release Policy.
At least 3 (+1) votes should be recorded in accordance to Votes on Package Releases
The legal checks include:
The files should be present in the sub-folder of Airflow dist
The following files should be present (6 files):
As a PMC member, you should be able to clone the SVN repository
svn co https://dist.apache.org/repos/dist/dev/airflow/clients/python
Or update it if you already checked it out:
svn update .
Airflow Python client supports reproducible builds, which means that the packages prepared from the same sources should produce binary identical packages in reproducible way. You should check if the packages can be binary-reproduced when built from the sources.
Go to directory where your airflow sources are checked out and set the following environment variables:
export AIRFLOW_REPO_ROOT="$(pwd -P)"
VERSION=X.Y.Z
VERSION_SUFFIX=rc1
VERSION_RC=${VERSION}${VERSION_SUFFIX}
cd "${AIRFLOW_REPO_ROOT}"
python-client tag (assume apache is the remote name of the repository):git fetch apache --tags
git checkout python-client/${VERSION_RC}
rm -rf dist/*
breeze release-management prepare-python-client --distribution-format both --version-suffix ""
breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}"
The last - build step - by default will use Dockerized build and building of Python client packages
will be done in a docker container. However, if you have hatch installed locally you can use
--use-local-hatch flag and it will build and use docker image that has hatch installed.
breeze release-management prepare-python-client --distribution-format both --use-local-hatch --version-suffix ""
breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}"
This is generally faster and requires less resources/network bandwidth.
Both commands should produce reproducible .whl, .tar.gz packages in dist folder and "-source.tar.gz"
file containing airflow sources in dist folder.
# First clone the repo if you do not have it
cd ..
[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
svn update --set-depth=infinity asf-dist/dev/airflow/clients/python
export PATH_TO_AIRFLOW_SVN="${PWD}/asf-dist/dev/airflow/"
# Then compare the packages
cd ${PATH_TO_AIRFLOW_SVN}/clients/python/${VERSION_RC}
for i in ${AIRFLOW_REPO_ROOT}/dist/*
do
echo "Checking if $(basename $i) is the same as $i"
diff "$(basename $i)" "$i" && echo "OK"
done
The output should be empty (files are identical). In case the files are different, you should see:
Binary files apache_airflow-client-2.9.0.tar.gz and .../apache_airflow-2.9.0.tar.gz differ
You can use the breeze release-management check-release-files command to verify that all expected files are
present in SVN. This command may also help with verifying installation of the packages.
breeze release-management check-release-files python-client --version ${VERSION_RC} --path-to-airflow-svn=${PATH_TO_AIRFLOW_SVN}
You will see commands that you can execute to check installation of the distributions in containers.
This can be done with the Apache RAT tool.
Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the binary, the jar is inside)
You can run this command to do it for you (including checksum verification for your own security):
# Checksum value is taken from https://downloads.apache.org/creadur/apache-rat-0.17/apache-rat-0.17-bin.tar.gz.sha512
wget -q https://archive.apache.org/dist/creadur/apache-rat-0.17/apache-rat-0.17-bin.tar.gz -O /tmp/apache-rat-0.17-bin.tar.gz
echo "32848673dc4fb639c33ad85172dfa9d7a4441a0144e407771c9f7eb6a9a0b7a9b557b9722af968500fae84a6e60775449d538e36e342f786f20945b1645294a0 /tmp/apache-rat-0.17-bin.tar.gz" | sha512sum -c -
tar -xzf /tmp/apache-rat-0.17-bin.tar.gz -C /tmp
Unpack the release source archive (the <package + version>-source.tar.gz file) to a folder
rm -rf /tmp/apache/airflow-python-client-src && mkdir -p /tmp/apache-airflow-python-client-src && tar -xzf ${PATH_TO_AIRFLOW_SVN}/clients/python/${VERSION_RC}/apache_airflow_python_client-*-source.tar.gz --strip-components 1 -C /tmp/apache-airflow-python-client-src
Run the check:
cp ${AIRFLOW_REPO_ROOT}/.rat-excludes /tmp/apache-airflow-python-client-src/.rat-excludes
java -jar /tmp/apache-rat-0.17/apache-rat-0.17.jar --input-exclude-file /tmp/apache-airflow-python-client-src/.rat-excludes /tmp/apache-airflow-python-client-src/ | grep -E "! |INFO: "
You should see no files reported as Unknown or with wrong licence and summary of the check similar to:
INFO: Apache Creadur RAT 0.17 (Apache Software Foundation)
INFO: Excluding patterns: .git-blame-ignore-revs, .github/*, .git ...
INFO: Excluding MISC collection.
INFO: Excluding HIDDEN_DIR collection.
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
INFO: RAT summary:
INFO: Approved: 15615
INFO: Archives: 2
INFO: Binaries: 813
INFO: Document types: 5
INFO: Ignored: 2392
INFO: License categories: 2
INFO: License names: 2
INFO: Notices: 216
INFO: Standards: 15609
INFO: Unapproved: 0
INFO: Unknown: 0
There should be no files reported as Unknown or Unapproved. The files that are unknown or unapproved should be shown with a line starting with !.
For example:
! Unapproved: 1 A count of unapproved licenses.
! /CODE_OF_CONDUCT.md
Make sure you have imported into your GPG the PGP key of the person signing the release. You can find the valid keys in KEYS.
You can import the whole KEYS file:
wget https://dist.apache.org/repos/dist/release/airflow/KEYS
gpg --import KEYS
You can also import the keys individually from a keyserver. The below one uses Kaxil's key and retrieves it from the default GPG keyserver OpenPGP.org:
gpg --keyserver keys.openpgp.org --receive-keys CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F
You should choose to import the key when asked.
Note that by being default, the OpenPGP server tends to be overloaded often and might respond with errors or timeouts. Many of the release managers also uploaded their keys to the GNUPG.net keyserver, and you can retrieve it from there.
gpg --keyserver keys.gnupg.net --receive-keys CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F
Once you have the keys, the signatures can be verified by running this:
cd ${PATH_TO_AIRFLOW_SVN}/clients/python/${VERSION_RC}
for i in *.asc
do
echo -e "Checking $i\n"; gpg --verify $i
done
This should produce results similar to the below. The "Good signature from ..." is indication
that the signatures are correct. Do not worry about the "not certified with a trusted signature"
warning. Most of the certificates used by release managers are self-signed, and that's why you get this
warning. By importing the key either from the server in the previous step or from the
KEYS page, you know that
this is a valid key already. To suppress the warning you may edit the key's trust level
by running gpg --edit-key <key id> trust and entering 5 to assign trust level ultimate.
Checking apache-airflow-2.0.2rc4.tar.gz.asc
gpg: assuming signed data in 'apache-airflow-2.0.2rc4.tar.gz'
gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
gpg: Good signature from "Kaxil Naik <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
Checking apache_airflow-2.0.2rc4-py2.py3-none-any.whl.asc
gpg: assuming signed data in 'apache_airflow-2.0.2rc4-py2.py3-none-any.whl'
gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
gpg: Good signature from "Kaxil Naik <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
Run this:
cd ${PATH_TO_AIRFLOW_SVN}/clients/python/${VERSION_RC}
for i in *.sha512
do
echo "Checking $i"; shasum -a 512 `basename $i .sha512 ` | diff - $i
done
You should get output similar to:
Checking apache-airflow-client-2.0.2rc4.tar.gz.sha512
Checking apache_airflow-client-2.0.2rc4-py2.py3-none-any.whl.sha512
This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the actual users of Airflow Client test it in their own staging/test installations. Each release candidate is available on PyPI apart from SVN packages, so everyone should be able to install the release candidate version of Airflow Client via simply (<VERSION> is 2.0.0 for example, and <X> is release candidate number 1,2,3,....).
Once you install and run Airflow Client, you should perform any verification you see as necessary to check that the client works as you expected.
You can test the client by running the start-airflow command from Breeze. This will start Airflow
and allows you to test the client in a real environment.
files/airflow-breeze-config/init.sh:export AIRFLOW__API__AUTH_BACKENDS=airflow.providers.fab.auth_manager.api.auth.backend.session,airflow.providers.fab.auth_manager.api.auth.backend.basic_auth
export AIRFLOW__API__EXPOSE_CONFIG=True
When you enter Breeze the webserver will be available at http://localhost:8080 (from inside the container)
or http://localhost:28080 from the host) and you should be able to access the API
with admin/admin credentials. The http://localhost:8080 and admin/admin credentials are
default in the clients/python/test_python_client.py test.
The AIRFLOW__API__EXPOSE_CONFIG is optional - the script will also succeed when
(default setting) exposing configuration is disabled.
breeze start-airflow --load-example-dags
Give the server 20-30 seconds to serialize the example DAGs to DB
In the meantime install the python client you want to test - in the terminal window in the container.
It can be installed from PyPI via pip install apache-airflow-client==X.Y.Zrc1 or installed
from a file (for example if you nust built the client with breeze release-management prepare-python-client,
there will be /dist/apache_airflow_client-py*.whl file in in your /dist folder (mapped from dist
folder in your checked out airflow repository and you can install it with:
pip install /dist/apache_airflow_client-py*.whl.
The client script is available in /opt/airflow/clients/python/test_python_client.py and you can run it
from inside the container with:
python /opt/airflow/clients/python/test_python_client.py
Subject:
[RESULT][VOTE] Release Airflow Python Client 3.1.3 from 3.1.3rc1
Hello,
The vote to release Apache Airflow Python Client version 3.1.3 based on 3.1.3rc1 is now closed.
The vote PASSED with 3 binding "+1", 1 non-binding "+1" and 0 "-1" votes:
3 "+1" binding votes received:
- Ephraim Anierobi
- Jarek Potiuk
- Jed Cunningham
1 "+1" non-binding votes received:
- Pierre Jeambrun
Vote thread:
https://lists.apache.org/thread/1qcj0r67dff3zg0w2vyfhr30fx9xtp3y
I'll continue with the release process, and the release announcement will follow shortly.
Cheers,
<your name>
# Go to Airflow sources first
cd <YOUR_AIRFLOW_REPO_ROOT>
export AIRFLOW_REPO_ROOT="$(pwd)"
# Go to Airflow python client sources first
cd <YOUR_AIRFLOW_CLIENT_REPO_ROOT>
export CLIENT_REPO_ROOT="$(pwd)"
cd ..
# Clone the AS
[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
svn update --set-depth=infinity asf-dist/{release,dev}/airflow
CLIENT_DEV_SVN="${PWD}/asf-dist/dev/airflow/clients/python"
CLIENT_RELEASE_SVN="${PWD}/asf-dist/release/airflow/clients/python"
cd "${CLIENT_RELEASE_SVN}"
export VERSION="2.8.1"
# Update the approved RC version here
export VERSION_SUFFIX="rc1"
# Update the previous version that have been released here
# There should be only one version in https://downloads.apache.org/airflow/clients/python/
# Policy here: http://www.apache.org/legal/release-policy.html#when-to-archive
export PREVIOUS_VERSION=2.8.0
# Create new folder for the release
svn mkdir ${VERSION}
cd ${VERSION}
# Move the artifacts to svn folder & commit
for f in ${CLIENT_DEV_SVN}/${VERSION}${VERSION_SUFFIX}/*; do
svn cp $f . ;
done
# Remove old release
cd ..
svn rm ${PREVIOUS_VERSION}
svn commit -m "Release Apache Airflow Python Client ${VERSION} from ${VERSION}${VERSION_SUFFIX}"
Verify that the packages appear in airflow
We need to upload the packages to PyPI. Note that we are not copying the generated sources, we just
cd ${VERSION}
twine check *${VERSION}.tar.gz *.whl
twine upload -r pypi *${VERSION}.tar.gz *.whl
Confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow-client
Push Tag for the final version to the Airflow repository and the Python Client repository:
cd ${AIRFLOW_REPO_ROOT}
git checkout python-client/${VERSION}${VERSION_SUFFIX}
git tag -s python-client/${VERSION} -m "Airflow Python Client ${VERSION}"
git push apache tag python-client/${VERSION}
cd ${CLIENT_REPO_ROOT}
git checkout ${VERSION}${VERSION_SUFFIX}
git tag -s ${VERSION} -m ${VERSION}
git push origin tag ${VERSION}
Create a new release on GitHub "airflow-python-client" repo with the release notes and assets from the release svn.
Notify [email protected] (cc'ing [email protected]) that the artifacts have been published:
Subject:
cat <<EOF
[ANNOUNCE] Apache Airflow Python Client ${VERSION} Released
EOF
Body:
cat <<EOF
Dear Airflow community,
I'm happy to announce that Apache Airflow Python Client ${VERSION} was just released.
We made this version available on PyPI for convenience:
\`pip install apache-airflow-client==${VERSION}\`
https://pypi.org/project/apache-airflow-client/${VERSION}/
The documentation is available at:
https://github.com/apache/airflow-client-python/
Find the changelog here for more details:
https://github.com/apache/airflow-client-python/blob/main/CHANGELOG.md
Thanks,
<your name>
EOF
Add the release data (version and date) at: https://reporter.apache.org/addrelease.html?airflow