[Colab-specific instructions] Installing Pathway with Python 3.8+

In the cell below we install pathway into a Python 3.8+ Linux runtime. Please:

Insert in the form below the pip install link given to you with your beta access.

Run the colab notebook (Ctrl+F9), disregarding the 'not authored by Google' warning. The installation and loading time is less than 1 minute.

python

#@title ⚙️ Pathway installer. Please provide the pip install link for Pathway:
# Please copy here the installation line:
PATHWAY_INSTALL_LINE='pip install https://packages.pathway.com/...' #@param {type:"string"}

if PATHWAY_INSTALL_LINE.startswith('pip install '):
    PATHWAY_INSTALL_LINE=PATHWAY_INSTALL_LINE[len('pip install '):]

class InterruptExecution(Exception):
    def _render_traceback_(self):
        pass

if '...' in PATHWAY_INSTALL_LINE or not PATHWAY_INSTALL_LINE.startswith('https://'):
    print(
        "⛔ Please register at https://pathway.com/developers/documentation/introduction/installation-and-first-steps\n"
        "to Copy & Paste the Linux pip install line for Pathway!"
    )
    raise InterruptExecution

if 'macosx_11_0_arm64' in PATHWAY_INSTALL_LINE:
    PATHWAY_INSTALL_LINE = PATHWAY_INSTALL_LINE.replace('macosx_11_0_arm64','manylinux_2_12_x86_64.manylinux2010_x86_64')
DO_INSTALL = False
import sys
if sys.version_info >= (3, 8):
    print(f'✅ Python {sys.version} is active.')
    try:
        import pathway as pw
        print('✅ Pathway successfully imported.')
    except:
        DO_INSTALL = True
else:
    print("⛔ Pathway requires Python 3.8 or higher.")
    raise InterruptExecution

if DO_INSTALL:
    !ls $(dirname $(which python))/../lib/python*/*-packages/pathway 1>/dev/null 2>/dev/null || echo "⌛ Installing Pathway. This usually takes a few seconds..."
    !ls $(dirname $(which python))/../lib/python*/*-packages/pathway 1>/dev/null 2>/dev/null || pip install {PATHWAY_INSTALL_LINE} 1>/dev/null 2>/dev/null
    !ls $(dirname $(which python))/../lib/python*/*-packages/pathway 1>/dev/null 2>/dev/null || echo "⛔ Installation failed. Don't be shy to reach out to the community at https://pathway.com !"
    !ls $(dirname $(which python))/../lib/python*/*-packages/pathway 1>/dev/null 2>/dev/null && echo "✅ All installed. Enjoy Pathway!"

Getting started with Pathway

In the following, you can find instructions on how to start using Pathway.

How to install Pathway

You can download the current Pathway release, which is now available in Open Beta on a free-to-use license: ::pip-install :: on a Python 3.8+ installation, and we are ready to roll!

To use Pathway, we only need to import it:

python

import pathway as pw

How to connect to your first table

The first thing you need to do is to access the data you want to manipulate. Pathway provides many connectors to access your data and manipulate them.

As an example, let's load a table using a csv connector:

python


table_dogs = pw.debug.table_from_markdown(
    """
    | name  | age
  1 | Ace   | 8
  2 | Bella | 5
  3 | Coco  | 13
 """
)

Now the table is loaded. But if we try to print it, we obtain a very generic output: Table['age', 'name']

That's perfectly normal: as explained in our introduction to programming in Pathway, Pathway is used to schedule the operations that will be later performed in realtime by the runtime engine. To process the actual data in our example, we need to use debug function called compute_and_print:

python

pw.debug.compute_and_print(table_dogs)

Some basic operations using Pathway

Now that we have a table, we are going to do some basic operations on it. You can find the full list of the supported operations in our API documentation.

The first thing we may want, is to filter on the age and keep only the dogs younger than 10 years old. We can use the operator filter on the column 'age'. To access a column, we can either use the notation table_name.column_name or use the more generic table['column_name'].

python

table_dogs_young = table_dogs.filter(
    table_dogs.age <= 10
)  # table_dogs['age'] also works
pw.debug.compute_and_print(table_dogs_young)

We can also apply a function to a given column. Let's say that we want to change the value of a column. Due to an error in rounding, all the age values are wrong and should be decreased by one. We can modify the table using the apply operation:

python

table_dogs_corrected = table_dogs.select(
    table_dogs.name, age=pw.apply((lambda x: x - 1), table_dogs["age"])
)
pw.debug.compute_and_print(table_dogs_corrected)

What happens here is that we select from table_dogs the column 'name' and a column 'age' which is obtained by the operator apply: pw.apply(f,col) applies f to each entry in col (there may be several such columns).

To do more complicated operations, we may need a second table:

python

table_dogs_owners = pw.debug.table_from_markdown(
    """
    | name  | owner
  1 | Ace   | Alice
  2 | Bella | Bob
  3 | Coco  | Alice
 """
)
pw.debug.compute_and_print(table_dogs_owners)

We can build a table with both information using the operator join:

python

table_dogs_full = table_dogs_corrected.join(
    table_dogs_owners, table_dogs_corrected.name == table_dogs_owners.name
).select(table_dogs_corrected.name, table_dogs_corrected.age, table_dogs_owners.owner)
pw.debug.compute_and_print(table_dogs_full)

To go further

Now you know the basis of Pathway! If you want to learn more about the Pathway, you can check out our survival guide, our manual about joins, or our introduction about transformers.

As we continue you will see some more advanced programming constructs which provide a lot of flexibility to Pathway:

Applying Machine Learning to data tables.
The ability to do iteration and recursion.

We will also use Pathway connectors to external data sources (for data inputs) and sinks (for data outputs).

This, and a lot more, is covered in recipes in the Pathway cookbook - try these for a start: