ContextQMD
Back to Pathway

Troubleshooting

docs/2.developers/4.user-guide/90.development/10.troubleshooting.md

0.30.16.2 KB
Original Source

Troubleshooting

This page provides a guide to common issues encountered when using a Pathway.

As with any powerful tool, there is a learning curve to using Pathway effectively. This guide will outline some of the most common problems developers encounter when working with Pathway, along with tips and best practices for avoiding these pitfalls. Whether you are new to Pathway or an experienced user, this guide will help you to optimize your workflow and avoid common mistakes. By following these guidelines, you can save yourself time and frustration and get the most out of Pathway.

So let's get started!

:article-toc-without-title

Package versioning

If a module is missing or you encounter an issue reproducing the examples displayed on the website, you will likely need the latest version of Pathway. The best solution is to reinstall it.

⚠️ Pathway requires Python 3.10 or higher and runs on Linux and MacOS.

Possible error messages:

  • This is not the real Pathway package.
  • ModuleNotFoundError: No module named 'pathway.stdlib'

Solution:

  • Reinstall Pathway by uninstalling it first:
bash
pip uninstall pathway && pip install pathway
  • If the above solution does not work, try installing Pathway with --force-reinstall:
bash
pip install --force-reinstall pathway

You can access your Pathway version using pw.__version_ or pathway --version in the CLI.

⚠️ Windows is currently not supported.

Docker on MacOS

When using docker on MacOS, make sure to have a linux/x86_64 platform:

  • FROM --platform:linux/x86_64 python:3.10

Windows users

⚠️ Pathway is currently not supported on Windows. Windows users may want to use Windows Subsystem for Linux (WSL), docker, or a VM.

You can also try these steps in an online notebook environment like Colab.

Nothing happens / missing output

The application terminates.

You launch your Pathway application, and nothing happens: no outputs, no errors. The application terminates without error, but the expected output, whether stdout or a CSV file, is empty.

Your application likely builds the dataflow, but doesn't launch a computation using it. You need to trigger the computation:

  • streaming mode: use pw.run() (in addition to the use of output connectors),
  • static mode: print your table using pw.debug.compute_and_print.

Explanation:

  • Pathway's operators are used to build a pipeline modeled by a dataflow. The data is ingested only when the computation is started. In the streaming mode, the computation is launched using pw.run(), while in the static mode, the computation is triggered at each output connector call. See our article about streaming and static modes for more details.

The application does not terminate

Similarly to above, you launch your Pathway application, and nothing happens. The application is running, there is no error but, at the same time, the output remains empty.

You input data is likely to be at fault:

  • is your incoming data stream on?
  • are you correctly connected to it? For example, let's suppose you have a data stream with a column colA. If you are connecting to this data source with an incorrect schema, expecting a colB instead of colA for instance, the data may be coming but it will not be ingested as the system expects values for colB and not colA. This will not trigger any error as the connection is working, expecting correct entries.

Different universes

Error message:

  • ValueError: universes do not match

Explanation:

  • The error comes from the combination of two tables with different universes. The universe is the set of indexes of each table. Some operations, such as update_cells or update_rows, require the universes to be identical, or at least one should be a subset of the other. Pathway will raise an error when it is impossible to infer whether two tables have the same universe. You can manually assert that the two universes are compatible with unsafe_promise_same_universe_as or unsafe_promise_universe_is_subset_of.

Solutions:

  • You can force the given operation by giving a manual guarantee that the universe will be the same:
python
T1=T1.unsafe_promise_same_universe_as(T2)
# OR
T1=T1.unsafe_promise_universe_is_subset_of(T2)

Common errors with RAG applications

UnstructuredParser UnsupportedFileFormatError or FileType.UNK exceptions

UnstructuredParser utilizes the unstructured.partition.auto module for parsing files under the hood. This module calls the magic library for determining the file mime type, in case the magic dependency is missing, unstructured library will throw the following exception:

Partitioning is not supported for the FileType.UNK file type.

To resolve this, you should install libmagic, or magic library. If you are running into this issue on MacOS, you can install it with:

brew install libmagic

If you are running it on Google Colab or a Debian-based Linux environment, you can install it with:

apt install libmagic1

For more information, see: filemagic installation, and python-magic.

<!-- REMOVING datetime type errors ## Type errors ### Datetime Possible error message: - `TypeError: argument 'values': unsupported value type: Timestamp` Solution: - Cast to a regular timestamp (int) using `datetime.timestamp` Explanation: - Pathway does not currently support `datetime.datetime`, and using those should result in such `TypeError`. You can cast the `datetime.datetime` to integer using a `pw.apply` with `datetime.timestamp`. -->

Still blocked?

We hope this guide has helped identify and avoid common mistakes when using Pathway. Our team is always happy to help you find a solution to your problem and ensure that you get the most out of Pathway. If you have any questions or encounter issues not covered in this guide, don't hesitate to get in touch with us on our Discord channel.