ContextQMD
Back to Qlib

Qlib Client-Server Framework

docs/hidden/client.rst

0.9.78.2 KB
Original Source

.. _client:

Qlib Client-Server Framework

.. currentmodule:: qlib

Introduction

Client-Server is designed to solve following problems

  • Manage the data in a centralized way. Users don't have to manage data of different versions.
  • Reduce the amount of cache to be generated.
  • Make the data can be accessed in a remote way.

Therefore, we designed the client-server framework to solve these problems. We will maintain a server and provide the data.

You have to initialize you qlib with specific config for using the client-server framework. Here is a typical initialization process.

qlib init commonly used parameters; nfs-common must be installed on the server where the client is located, execute: sudo apt install nfs-common: - provider_uri: nfs-server path; the format is host: data_dir, for example: 172.23.233.89:/data2/gaochao/sync_qlib/qlib. If using offline, it can be a local data directory - mount_path: local data directory, provider_uri will be mounted to this directory - auto_mount: whether to automatically mount provider_uri to mount_path during qlib init; You can also mount it manually: sudo mount.nfs provider_uri mount_path. If on PAI, it is recommended to set auto_mount=True - flask_server: data service host; if you are on the intranet, you can use the default host: 172.23.233.89 - flask_port: data service port

If running on 10.150.144.153 or 10.150.144.154 server, it's recommended to use the following code to init qlib:

.. code-block:: python

import qlib qlib.init(auto_mount=False, mount_path='/data/csdesign/qlib') from qlib.data import D D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head() [39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [init.py:16] - default_conf: client. [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [init.py:54] - qlib successfully initialized based on client settings. [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [init.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib [39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710 [39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server! Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode Out[5]: $close instrument datetime SH600000 2008-01-02 119.079704 2008-01-03 113.120125 2008-01-04 117.878860 2008-01-07 124.505539 2008-01-08 125.395004

If running on PAI, it's recommended to use the following code to init qlib:

.. code-block:: python

import qlib qlib.init(auto_mount=True, mount_path='/data/csdesign/qlib', provider_uri='172.23.233.89:/data2/gaochao/sync_qlib/qlib') from qlib.data import D D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head() [39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [init.py:16] - default_conf: client. [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [init.py:54] - qlib successfully initialized based on client settings. [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [init.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib [39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710 [39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server! Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode Out[5]: $close instrument datetime SH600000 2008-01-02 119.079704 2008-01-03 113.120125 2008-01-04 117.878860 2008-01-07 124.505539 2008-01-08 125.395004

If running on Windows, open NFS features and write correct mount_path, it's recommended to use the following code to init qlib:

1.windows System open NFS Features * Open Programs and Features. * Click Turn Windows features on or off. * Scroll down and check the option Services for NFS, then click OK

Reference address: https://graspingtech.com/mount-nfs-share-windows-10/

2.config correct mount_path * In windows, mount path must be not exist path and root path, * correct format path eg: H, i... * error format path eg: C, C:/user/name, qlib_data...

.. code-block:: python

import qlib qlib.init(auto_mount=True, mount_path='H', provider_uri='172.23.233.89:/data2/gaochao/sync_qlib/qlib') from qlib.data import D D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head() [39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [init.py:16] - default_conf: client. [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [init.py:54] - qlib successfully initialized based on client settings. [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [init.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib [39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710 [39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server! Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode Out[5]: $close instrument datetime SH600000 2008-01-02 119.079704 2008-01-03 113.120125 2008-01-04 117.878860 2008-01-07 124.505539 2008-01-08 125.395004

The client will mount the data in provider_uri on mount_path. Then the server and client will communicate with flask and transporting data with this NFS.

If you have a local qlib data files and want to use the qlib data offline instead of online with client server framework. It is also possible with specific config. You can created such a config. client_config_local.yml

.. code-block:: YAML

provider_uri: /data/csdesign/qlib
calendar_provider: 'LocalCalendarProvider'
instrument_provider: 'LocalInstrumentProvider'
feature_provider: 'LocalFeatureProvider'
expression_provider: 'LocalExpressionProvider'
dataset_provider: 'LocalDatasetProvider'
provider: 'LocalProvider'
dataset_cache: 'SimpleDatasetCache'
local_cache_path: '~/.cache/qlib/'

provider_uri is the directory of your local data.

.. code-block:: python

import qlib qlib.init_from_yaml_conf('client_config_local.yml') from qlib.data import D D.features(['SH600001'], ['$close'], start_time='20180101', end_time='20190101').head() 21232:MainThread](2019-05-29 10:16:05,066) INFO - Initialization - [init.py:16] - default_conf: client. [21232:MainThread](2019-05-29 10:16:05,066) INFO - Initialization - [init.py:54] - qlib successfully initialized based on client settings. [21232:MainThread](2019-05-29 10:16:05,067) INFO - Initialization - [init.py:56] - provider_uri=/data/csdesign/qlib Out[9]: $close instrument datetime SH600001 2008-01-02 21.082111 2008-01-03 23.195362 2008-01-04 23.874615 2008-01-07 24.880930 2008-01-08 24.277143

Limitations

  1. The following API under the client-server module may not be as fast as the older off-line API.
    • Cal.calendar
    • Inst.list_instruments
  2. The rolling operation expression with parameter 0 can not be updated rightly under mechanism of the client-server framework.

API

The client is based on python-socketio <https://python-socketio.readthedocs.io>_ which is a framework that supports WebSocket client for Python language. The client can only propose requests and receive results, which do not include any calculating procedure.

Class

.. automodule:: qlib.data.client