The Easiest Solution for Python Stream Processing, Data Indexing, and Real-Time AI Analytics.

<!-- https://www.canva.com/design/DAGEibqmR74/6QHiszH-xUcKeBSCciNuYA/edit?utm_content=DAGEibqmR74&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton --> <!--  -->

::article-img

src: '/assets/content/documentation/why-live-data-framework/why-pathway-key-points.svg' alt: 'Pathway three key assets: simplicity, low latency, and batch & stream' zoomable: true class: 'mx-auto' width: '300'

::

• Simplicity at all stages: Installation, integration, development, and deployment.
• Low latency: Pathway Live Data Framework is the fastest data processing framework, thanks to its powerful Rust engine.
• Batch and stream processing alike: Pathway Live Data Framework handles batch and streaming data for you in the same way.

It's simple. From installation to deployment.

<!-- https://www.canva.com/design/DAGEirGYDO0/YtOm02qpZF52oy6C022low/edit?utm_content=DAGEirGYDO0&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton --> <!--  -->

::article-img

src: '/assets/content/documentation/why-live-data-framework/why-pathway-python.svg' alt: 'Pathway is easy to use thanks to its full compatibility with Python' zoomable: true class: 'mx-auto' width: '580'

::

With its full Python compatibility, Pathway Live Data Framework is easy to use, from the installation to the maintenance.

1. Python native: Pathway Live Data Framework is a Python framework, and as such it is compatible with the whole Python ecosystem. It will integrate perfectly into your Python architecture and will allow you to use your favorite libraries.
2. Installation: you can install Pathway Live Data Framework with a simple pip install pathway.
3. Many data sources: Pathway Live Data Framework provides a multitude of connectors to access your favorite data sources. You can also set up your own.
4. Transformation and Machine Learning: you can easily design your data pipeline using Pathway Live Data Framework transformations. You can define your own UDFs, use any Python library, and integrate Machine Learning models.
5. Many destinations: Pathway Live Data Framework provides output connectors to send the results to the destination you want. You can also create your own.
6. RAG and LLM-ready: Pathway Live Data Framework provides most of the common utilities to develop your LLM applications and RAG. This includes complete AI pipelines with structured and unstructured data ingestion, chunking, and indexing.
7. Data indexing: Pathway Live Data Framework offers real-time data indexes (vector search, full text search, and more) allowing you to effortlessly synchronize your index with data sources in real time. Don't bother installing a dedicated vector store, Pathway Live Data Framework has got it covered for you!

It's fast, scalable, and safe.

<!-- https://www.canva.com/design/DAGEipyLAgo/kZCSb7DqOXTnVbvghT2m2w/edit?utm_content=DAGEipyLAgo&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton --> <!--  -->

::article-img

src: '/assets/content/documentation/why-live-data-framework/why-pathway-rust-new.svg' alt: 'Pathway Rust engine makes it fast, scalable, and safe.' zoomable: true class: 'mx-auto' width: '800'

::

1. A powerful Rust engine: Pathway Live Data Framework is not bound to Python limits as it relies on a powerful Rust engine. The engine ensures that the computations are fast.
2. Scalable: thanks to the Rust engine, Pathway Live Data Framework provides multi-threading, multi-processing and distributed computations. You can easily deploy your pipeline in the cloud.
3. Differential Dataflow and incremental computations: The engine incrementally processes data updates. Results are computed using the minimum work needed, ensuring high latency.
4. Stateful operations: Pathway Live Data Framework supports stateful operations such as groupby and windows.
5. Persistence: you can save the state of the ongoing computation, be it for updating your pipeline or for recovery.

<!-- 6. **Fault-tolerant** -->

It takes the pain out of temporal & event data

<!-- https://www.canva.com/design/DAGEiyGFyX4/3cMkRN6D5jmomK7vC-p0NQ/edit?utm_content=DAGEiyGFyX4&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton --> <!--  -->

::article-img

src: '/assets/content/documentation/why-live-data-framework/why-pathway-unified.svg' alt: 'Pathway is a unified data processing framework, doing both batch and stream processing' zoomable: true class: 'mx-auto' width: '400'

::

1. Batch and stream processing alike: Pathway Live Data Framework does both batch and stream processing. No matter your use case, it is a good fit.
2. Same syntax: Your pipeline can run on both batch and streaming data, without modifying your code.
3. Same engine: The unified Rust engine makes your computation fast and scalable, no matter if you choose batch or stream processing.
4. Consistent results: for stream processing, Pathway Live Data Framework returns an output in real-time, which is what you would have if you were processing the received data using batch processing.
5. Streaming complexity is hidden: All the challenges of stream processing, such as handling late and out-of-order data, are handled by the engine and hidden from the user.
6. Time-related operations: Pathway Live Data Framework offers advanced temporal operations such as as-of-join and temporal windows.

Python + Rust: the best of both worlds

The Pathway Live Data Framework efficiently associates the convenience of Python with the power of Rust.

<!-- https://www.canva.com/design/DAGEjAJ_cS0/wdoUAK8nJpp85jcjjbHLnQ/edit?utm_content=DAGEjAJ_cS0&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton --> <!--  -->

::article-img

src: '/assets/content/documentation/why-live-data-framework/why-pathway-python-rust.svg' alt: 'Pathway: simplicity of Python + power of Rust' zoomable: true class: 'mx-auto' width: '300'

::

Python makes everything easy

The Pathway Live Data Framework is a fully Python-compatible framework. You can install it with a simple pip install pathway and import it as any Python library. It provides a Python interface and experience created with data developers in mind. You can easily build pipelines by manipulating framework tables and rely on the vast resources and libraries of the Python ecosystem. Also, the framework can seamlessly be integrated into your CI/CD chain as it is inherently compatible with popular tools such as mypy or pytest. Your pipelines can be automatically tested, built, and deployed like any other Python workflow. The Pathway Live Data Framework can be easily deployed in any container-based method (docker, Kubernetes) supporting the deployment of Python-based projects.

Rust makes your pipeline fast and scalable

The Pathway Live Data Framework relies on a powerful Rust engine to ensure high performance for your pipelines, no matter if you are dealing with batch or streaming data. The engine makes the utmost of Rust speed and memory safety to provide efficient parallel and distributed processing without being limited by Python's GIL.

The engine is based on Differential dataflow, a computational framework known for its efficiency to process large volumes of data. Its incremental computations make it able to quickly process data updates. This means that the minimum work needed by any algorithm or transformation is performed to refresh its results when fresh data arrives.

A unified framework to end the debate between batch and stream processing

Batch processing and stream processing are seen as two distinct approaches to handling data.

<!-- However, when you think about it, all the data we interact with is, in essence, a data stream. Data is typically generated and updated over time. -->

The Pathway Live Data Framework is a unified data processing framework that allows you to use the same code for batch and streaming. All the complexity, including late data and consistency, are automatically handled and hidden from the user. The Pathway Live Data Framework provides advanced streaming operations, such as temporal windows, while keeping the simplicity of batch processing.

With Pathway Live Data Framework, you don't have to choose between batch and stream processing. You can make your pipeline and focus on the data transformation you want to do. The resulting pipeline will work with both batch and stream processing. Not having to distinguish between batch and stream --and use different tools for them-- highly simplifies your architecture (bye-bye Lambda architecture) and the development of your pipeline.

What can it be used for?

With its unified engine and full Python compatibility, Pathway Live Data Framework makes data processing as easy as possible. It's the ideal solution for a wide range of data processing pipelines, including:

• Real-time analytics on IoT and event data.
• AI RAG pipelines at scale.
• Real-time Document Indexing.
• ETL on unstructured data.

Learn more about the real-world applications of Pathway Live Data Framework on our solutions page and our success stories page.