ContextQMD
Back to Airbyte

Google Search Console

docs/integrations/sources/google-search-console.md

2.0.037.9 KB
Original Source

Google Search Console

<HideInUI>

This page contains the setup guide and reference information for the Google Search Console source connector.

</HideInUI>

Prerequisites

  • Google Account
  • A verified property in Google Search Console (or the list of the Site URLs (Website URL Property))
<!-- env:oss -->
  • Google Search Console API enabled for your project (Airbyte Open Source only)
<!-- /env:oss -->

Setup guide

Step 1: Set up Google Search Console

To authenticate the Google Search Console connector, you will need to use one of the following methods:

<!-- env:cloud -->

OAuth (Recommended for Airbyte Cloud)

You can authenticate using your Google Account with OAuth if you are the owner of the Google Search Console property or have view permissions. Follow Google's instructions to ensure that your account has the necessary permissions (Owner or Full User) to view the Google Search Console property. This option is recommended for Airbyte Cloud users, as it significantly simplifies the setup process and allows you to authenticate the connection directly from the Airbyte UI.

<!-- /env:cloud --> <!-- env:oss -->

To authenticate with OAuth in Airbyte Open Source, you will need to create an authentication app and obtain the following credentials and tokens:

  • Client ID
  • Client Secret
  • Refresh Token
  • Access Token

More information on the steps to create an OAuth app to access Google APIs and obtain these credentials can be found in Google's documentation.

Google service account with JSON key file (Recommended for Airbyte Open Source)

You can authenticate the connection using a JSON key file associated with a Google service account. This option is recommended for Airbyte Open Source users. Follow the steps below to create a service account and generate the JSON key file:

  1. Open the Service Accounts page.
  2. Select an existing project, or create a new project.
  3. At the top of the page, click + Create service account.
  4. Enter a name and description for the service account, then click Create and Continue.
  5. Under Service account permissions, select the roles to grant to the service account, then click Continue. We recommend the Viewer role.
    • Optional: Under Grant users access to this service account, you may specify the users or groups that are allowed to use and manage the service account.
  6. Go to the API Console/Credentials and click on the email address of the service account you just created.
  7. In the Keys tab, click + Add key, then click Create new key.
  8. Select JSON as the Key type. This will generate and download the JSON key file that you'll use for authentication. Click Continue.

:::caution This file serves as the only copy of your JSON service key, and you will not be able to re-download it. Be sure to store it in a secure location. :::

:::note You can return to the API Console/Credentials at any time to manage your service account or generate additional JSON keys. For more details about service account credentials, see Google's IAM documentation. :::

Note on delegating domain-wide authority to the service account

Domain-wide delegation is a powerful feature that allows service accounts to access users' data across an organization's Google Workspace environment through 'impersonation'. This authority is necessary in certain use cases, such as when a service account needs broad access across multiple users and services within a domain.

:::note Only the super admin of your Google Workspace domain can enable domain-wide delegation of authority to a service account. :::

To enable delegated domain-wide authority, follow the steps listed in the Google documentation. Please make sure to grant the following OAuth scopes to the service account:

  • https://www.googleapis.com/auth/webmasters.readonly

For more information on this topic, please refer to this Google article.

<!-- /env:oss -->

Step 2: Set up the Google Search Console connector in Airbyte

<!-- env:cloud -->

For Airbyte Cloud:

  1. Log into your Airbyte Cloud account.
  2. Click Sources and then click + New source.
  3. On the Set up the source page, select Google Search Console from the Source type dropdown.
  4. Enter a name for the Google Search Console connector.
  5. For Website URL Property, enter the specific website property in Google Seach Console with data you want to replicate.
  6. For Start Date, by default the 2021-01-01 is set, use the provided datepicker or enter a date in the format YYYY-MM-DD. Any data created on or after this date will be replicated.
  7. To authenticate the connection:
<!-- env:cloud --> <!-- env:oss -->

For Airbyte Open Source:

  1. Navigate to the Airbyte Open Source dashboard.
  2. Click Sources and then click + New source.
  3. On the Set up the source page, select Google Search Console from the Source type dropdown.
  4. Enter a name for the Google Search Console connector.
<!-- /env:oss -->
  • For Airbyte Cloud:
    • Select Oauth from the Authentication dropdown, then click Sign in with Google to authorize your account. <!-- /env:cloud --> <!-- env:oss -->
  • For Airbyte Open Source:
    • (Recommended) Select Service Account Key Authorization from the Authentication dropdown, then enter the Admin Email and Service Account JSON Key. For the key, copy and paste the JSON key you obtained during the service account setup. It should begin with {"type": "service account", "project_id": YOUR_PROJECT_ID, "private_key_id": YOUR_PRIVATE_KEY, ...}
    • Select Oauth from the Authentication dropdown, then enter your Client ID, Client Secret, Access Token and Refresh Token.
    <!-- /env:oss -->
  1. (Optional) For End Date, you may optionally provide a date in the format YYYY-MM-DD. Any data created between the defined Start Date and End Date will be replicated. Leaving this field blank will replicate all data created on or after the Start Date to the present.
  2. (Optional) For Custom Reports, you may optionally provide an array of JSON objects representing any custom reports you wish to query the API with. Refer to the Custom reports section below for more information on formulating these reports.
  3. (Optional) For Data Freshness, you may choose whether to include "fresh" data that has not been finalized by Google, and may be subject to change. Please note that if you are using Incremental sync mode, we highly recommend leaving this option to its default value of final. Refer to the Data Freshness section below for more information on this parameter.
  4. (Optional) For Search Analytics API Requests Per Minute, you may configure the maximum number of requests per minute for Search Analytics API calls. The default value (1200) matches Google's documented maximum quota. If you are experiencing rate limit errors, you may need to lower this value. Most new Google Cloud projects start with a quota of 60 requests per minute. Check your Google Cloud Console quotas to see your actual limit. Refer to the Rate Limiting section below for more information.
  5. Click Set up source and wait for the tests to complete.
<HideInUI>

Supported sync modes

The Google Search Console source connector supports the following sync modes:

:::note The granularity for the cursor is 1 day, so Incremental Sync in Append mode may result in duplicating the data. :::

Supported Streams

Entity-Relationship Diagram (ERD)

<EntityRelationshipDiagram></EntityRelationshipDiagram>

Connector-specific configurations

Custom reports

Custom reports allow you to query the API with a custom set of dimensions to group results by. Results are grouped in the order that you supply these dimensions. Each custom report should be constructed like following:

  1. Click Add under the Custom Reports section
  2. Enter the Name of the report, this will be the name of the stream
  3. Select one or more Dimensions from the available dropdown list

The available Dimensions are:

  • country
  • date
  • device
  • page
  • query

For example, to query the API for a report that groups results by country, then by date, you could enter the following custom report:

  • Name: country_date
  • Dimensions: ["country", "date"]

Please note, that for technical reasons date is the default dimension which will be included in your query whether you specify it or not. By specifying it you can change the order the results are grouped in. Primary key will consist of your custom dimensions and the default dimension along with site_url and search_type.

The information you provide via UI Custom report builder will then be transformed into the custom stream by it's Name

You can use the Google APIS Explorer to build and test the reports you want to use.

Data Freshness

The Data Freshness parameter deals with the "freshness", or finality of the data that is being queried.

  • final: The query will include only finalized, stable data. This is data that has been processed, verified, and is unlikely to change. When you select this option, you are querying for the definitive statistics and information that Google has analyzed and confirmed.
  • all: The query will return both finalized data and what Google terms "fresh" data. Fresh data includes more recent data that hasn't gone through the full processing and verification that finalized data has. This option can give you more up-to-the-minute insights, but it may be subject to change as Google continues to process and analyze it.

:::caution When using Incremental Sync mode, we recommend leaving this parameter to its default state of final, as the all option may cause discrepancies between the data in your destination table and the finalized data in Google Search Console. :::

Data type map

Integration TypeAirbyte Type
stringstring
numbernumber
arrayarray
objectobject

Limitations & Troubleshooting

<details> <summary> Expand to see details about Google Search Console connector limitations and troubleshooting. </summary>

Connector limitations

Rate limiting

This connector attempts to back off gracefully when it hits Reports API's rate limits. To find more information about limits, see Usage Limits documentation.

While Google's public documentation states that the Search Console API allows up to 1,200 requests per minute, most Google Cloud projects start with a lower default quota of 60 requests per minute. This is especially common for new projects or projects without billing enabled.

To check your actual quota limits:

  1. Go to your Google Cloud Console.
  2. Navigate to APIs & Services then Quotas.
  3. Search for "Search Console API".
  4. Look for "Requests per minute per user" to see your current limit.

If you need higher limits, you can enable billing on your Google Cloud project or submit a quota increase request through the Google Cloud Console. You can then configure the API Requests Per Minute setting in the connector to match your actual quota.

Data retention

Google Search Console only retains data for websites from the last 16 months. Any data prior to this cutoff point will not be accessible. Please see this article for more information.

Troubleshooting

  • Check out common troubleshooting issues for the Google Search Console source connector on our Airbyte Forum.
</details>

Changelog

<details> <summary>Expand to review</summary>
VersionDatePull RequestSubject
1.10.252026-03-1074685Update dependencies
1.10.242026-02-2473935Update dependencies
1.10.232026-02-1773372Update dependencies
1.10.222026-02-1072683Update dependencies
1.10.212026-01-2071877Update dependencies
1.10.202026-01-1471722Update dependencies
1.10.192025-12-1870517Update dependencies
1.10.182025-12-0970444Add user-configurable API rate limit option for Search Analytics streams
1.10.172025-11-2570059Update dependencies
1.10.162025-11-1869392Update dependencies
1.10.152025-10-2968771Update dependencies
1.10.142025-10-2168231Update dependencies
1.10.132025-10-1468004Update dependencies
1.10.122025-10-0767263Update dependencies
1.10.112025-09-3065384Update dependencies
1.10.102025-09-1666010Update to CDK v7
1.10.92025-08-0964584Update dependencies
1.10.82025-08-0264187Update dependencies
1.10.72025-07-2663877Update dependencies
1.10.62025-07-2363720Ignore 403 "insufficient permission" errors
1.10.52025-07-1963471Update dependencies
1.10.42025-07-1563309Adds type property to config_normalization_rules in manifest
1.10.32025-07-1263094Update dependencies
1.10.22025-07-0562613Update dependencies
1.10.12025-06-2862194Update dependencies
1.10.02025-06-2361425Migrate to manifest-only
1.9.42025-06-2161789Update dependencies
1.9.32025-06-1861706Fix record serialization bug
1.9.22025-06-1460663Update dependencies
1.9.12025-06-1061514Promoting release candidate 1.9.1-rc.1 to a main version.
1.9.1-rc.12025-06-1061508Add API budget, reduce concurrency levels, and catch 403 rate limiting errors
1.9.02025-06-0661408Migrate custom reports streams to low-code
1.8.02025-06-0661335Migrate SearchAnalyticsAllFields, SearchAnalyticsByDate, SearchAnalyticsByDevice, SearchAnalyticsByPage, and SearchAnalyticsByQuery to low-code
1.7.02025-06-0561332Migrate search_by_keyword streams to low-code
1.6.02025-06-0461380Promoting release candidate 1.6.0-rc.1 to a main version.
1.6.0-rc.12025-06-0260928Migrate search_analytics_by_country, sites, and sitemaps to low-code
1.5.182025-05-2858121Prevent KeyError raise on check connetion when account does not have access to any site. Better Error message
1.5.172025-05-1054946Update dependencies
1.5.162025-05-0959737Promoting release candidate 1.5.16-rc.1 to a main version.
1.5.16-rc.12025-05-0259140Migrate to CDK V6
1.5.152025-02-2254426Update dependencies
1.5.142025-02-1553781Update dependencies
1.5.132025-02-0152854Update dependencies
1.5.122025-01-2552303Update dependencies
1.5.112025-01-1851629Update dependencies
1.5.102025-01-1151129Update dependencies
1.5.92025-01-0450919Update dependencies
1.5.82024-12-2850579Update dependencies
1.5.72024-12-2150040Update dependencies
1.5.62024-12-1449173Update dependencies
1.5.52024-11-2543730Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
1.5.42024-09-0645196Fix request body for report streams by keyword
1.5.32024-08-0343067Update dependencies
1.5.22024-07-2742786Update dependencies
1.5.12024-07-2042142Update dependencies
1.5.02024-07-1742073Migrate to CDK v1.8.0
1.4.132024-07-1341734Update dependencies
1.4.122024-07-1041440Update dependencies
1.4.112024-07-0941164Update dependencies
1.4.102024-07-0640981Update dependencies
1.4.92024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
1.4.82024-06-2640532Update dependencies
1.4.72024-06-2540312Update dependencies
1.4.62024-06-2240077Update dependencies
1.4.52024-06-1739516Update state handling for incremental streams
1.4.42024-06-0439059[autopull] Upgrade base image to v1.2.1
1.4.32024-05-2438649Update deprecated auth package
1.4.22024-04-1936639Updating to 0.80.0 CDK
1.4.12024-04-1236639Schema descriptions
1.4.02024-03-1936267Pin airbyte-cdk version to ^0
1.3.72024-02-1235163Manage dependencies with Poetry
1.3.62023-10-2631863Base image migration: remove Dockerfile and use the python-connector-base image
1.3.52023-09-2830822Fix primary key for custom reports
1.3.42023-09-2730785Do not migrate config for the newly created connections
1.3.32023-08-2929941Added primary key to each stream, added custom_report config migration
1.3.22023-08-2529829Make Start Date a non-required, added the suggested streams, corrected public docs
1.3.12023-08-2429329Update tooltip descriptions
1.3.02023-08-2429750Add new Keyword-Site-Report-By-Site stream
1.2.22023-08-2329741Handle HTTP-401, HTTP-403 errors
1.2.12023-07-0427952Removed deprecated searchType, added discover(Discover results) and googleNews(Results from news.google.com, etc.) types
1.2.02023-06-2927831Add new streams
1.1.02023-06-2627738License Update: Elv2
1.0.22023-06-1327307Fix data_state config typo
1.0.12023-05-3026746Remove authSpecification from connector spec in favour of advancedAuth
1.0.02023-05-2426452Add data_state parameter to specification
0.1.222023-03-2022295Update specification examples
0.1.212023-02-1422984Specified date formatting in specification
0.1.202023-02-0222334Turn on default HttpAvailabilityStrategy
0.1.192023-01-2722007Set AvailabilityStrategy for streams explicitly to None
0.1.182022-10-2718568Improved config validation: custom_reports.dimension
0.1.172022-10-0817751Improved config validation: start_date, end_date, site_urls
0.1.162022-09-2817304Migrate to per-stream state.
0.1.152022-09-1616819Check available site urls to avoid 403 error on sync
0.1.142022-09-0816433Add custom analytics stream.
0.1.132022-07-2114924Remove additionalProperties field from specs
0.1.122022-05-0412482Update input configuration copy
0.1.112022-01-059186Fix incremental sync: keep all urls in state object
0.1.102021-12-239073Add slicing by date range
0.1.92021-12-229047Add 'order' to spec.json props
0.1.82021-12-218248Enable Sentry for performance and errors tracking
0.1.72021-11-267431Add default end_date param value
0.1.62021-09-276460Update OAuth Spec File
0.1.42021-09-236394Update Doc link Spec File
0.1.32021-09-236405Correct Spec File
0.1.22021-09-176222Correct Spec File
0.1.12021-09-226315Verify access to all sites when performing connection check
0.1.02021-09-035350Initial Release

master

</details> </HideInUI>