providers/papermill/docs/operators.rst
.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Apache Airflow supports integration with Papermill_. Papermill is a tool for
parameterizing and executing Jupyter Notebooks. Perhaps you have a financial
report that you wish to run with different values on the first or last day of
a month or at the beginning or end of the year. Using parameters in your
notebook and using the :class:~airflow.providers.papermill.operators.papermill.PapermillOperator makes this a breeze.
.. _Papermill: https://papermill.readthedocs.io/en/latest/
Creating a notebook '''''''''''''''''''
To parameterize your notebook designate a cell with the tag parameters. Papermill looks for the parameters cell and treats this cell as defaults for the parameters passed in at execution time. Papermill will add a new cell tagged with injected-parameters with input parameters in order to overwrite the values in parameters. If no cell is tagged with parameters the injected cell will be inserted at the top of the notebook.
Make sure that you save your notebook somewhere so that Airflow can access it. Papermill supports S3, GCS, Azure and Local. HDFS is not supported.
Example Dag '''''''''''
Use the :class:~airflow.providers.papermill.operators.papermill.PapermillOperator
to execute a jupyter notebook:
.. exampleinclude:: /../../papermill/tests/system/papermill/example_papermill.py :language: python :dedent: 4 :start-after: [START howto_operator_papermill] :end-before: [END howto_operator_papermill]
Example Dag to Verify the message in the notebook:
.. exampleinclude:: /../../papermill/tests/system/papermill/example_papermill_verify.py :language: python :start-after: [START howto_verify_operator_papermill] :end-before: [END howto_verify_operator_papermill]
Example Dag to Verify the message in the notebook using a remote jupyter kernel:
.. exampleinclude:: /../../papermill/tests/system/papermill/example_papermill_remote_verify.py :language: python :start-after: [START howto_verify_operator_papermill_remote_kernel] :end-before: [END howto_verify_operator_papermill_remote_kernel]