Refreshing pre-aggregations

Pre-aggregation refresh is the process of building pre-aggregations and updating them with new data. Pre-aggregation refresh is the responsibility of the refresh worker.

Configuration

You can use the following environment variables to configure the refresh worker behavior:

<EnvVar>CUBEJS_REFRESH_WORKER</EnvVar> (see also <EnvVar>CUBEJS_PRE_AGGREGATIONS_BUILDER</EnvVar>)
<EnvVar>CUBEJS_PRE_AGGREGATIONS_SCHEMA</EnvVar>
<EnvVar>CUBEJS_SCHEDULED_REFRESH_TIMEZONES</EnvVar>
<EnvVar>CUBEJS_DB_QUERY_TIMEOUT</EnvVar>
<EnvVar>CUBEJS_REFRESH_WORKER_CONCURRENCY</EnvVar> (see also <EnvVar>CUBEJS_CONCURRENCY</EnvVar>)
<EnvVar>CUBEJS_SCHEDULED_REFRESH_QUERIES_PER_APP_ID</EnvVar>
<EnvVar>CUBEJS_DROP_PRE_AGG_WITHOUT_TOUCH</EnvVar>
<EnvVar>CUBEJS_PRE_AGGREGATIONS_BACKOFF_MAX_TIME</EnvVar>

Troubleshooting

`Refresh scheduler interval error`

Sometimes, you might come across the following error:

json

{
  "message": "Refresh Scheduler Interval Error",
  "error": "Previous interval #2 was not finished with 60000 interval"
}

It indicates that your refresh worker is overloaded. You probably have a lot of tenants, a lot of pre-aggregations to refresh, or both.

If you're using multitenancy, you'd need to deploy several Cube clusters (each one per a reduced set of tenants) so there will be multiple refresh workers which will work only on a subset of your tenants.

If you're using Cube Cloud, you can use a production multi-cluster that would automatically do this for you.