Quickstart

This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details.

Let's create an indexer for the tzBTC FA1.2 token contract. Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.

A modern Linux/macOS distribution with Python 3.10 installed is required to run DipDup.

Create a new project

Interactively (recommended)

You can initialize a hello-world project interactively by choosing configuration options in the terminal. The following command will install DipDup for the current user:

curl -Lsf https://dipdup.io/install.py | python

Now, let's create a new project:

dipdup new

Follow the instructions; the project will be created in the current directory. You can skip reading the rest of this page and slap dipdup run instead.

From scratch

Currently, we mainly use Poetry for dependency management in DipDup. If you prefer hatch, pdb, piptools or others — use them instead. Below are some snippets to get you started.

# Create a new project directory
mkdir dipdup-indexer; cd dipdup-indexer

# Plain pip
python -m venv .venv
. .venv/bin/activate
pip install dipdup

# or Poetry
poetry init --python ">=3.10,<3.11"
poetry add dipdup
poetry shell

Write a configuration file

DipDup configuration is stored in YAML files of a specific format. Create a new file named dipdup.yml in your current working directory with the following content:

spec_version: 1.2
package: demo_token

database:
  kind: sqlite
  path: demo-token.sqlite3

contracts:
  tzbtc_mainnet:
    address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
    typename: tzbtc

datasources:
  tzkt_mainnet:
    kind: tzkt
    url: https://api.tzkt.io

indexes:
  tzbtc_holders_mainnet:
    template: tzbtc_holders
    values:
      contract: tzbtc_mainnet
      datasource: tzkt_mainnet

templates:
  tzbtc_holders:
    kind: operation
    datasource: <datasource>
    contracts:
      - <contract>
    handlers:
      - callback: on_transfer
        pattern:
          - destination: <contract>
            entrypoint: transfer
      - callback: on_mint
        pattern:
          - destination: <contract>
            entrypoint: mint

Initialize project tree

Now it's time to generate typeclasses and callback stubs. Run the following command:

dipdup init

DipDup will create a Python package demo_token having the following structure:

demo_token
├── graphql
├── handlers
│   ├── __init__.py
│   ├── on_mint.py
│   └── on_transfer.py
├── hooks
│   ├── __init__.py
│   ├── on_reindex.py
│   ├── on_restart.py
│   ├── on_index_rollback.py
│   └── on_synchronized.py
├── __init__.py
├── models.py
├── sql
│   ├── on_reindex
│   ├── on_restart
│   ├── on_index_rollback
│   └── on_synchronized
└── types
    ├── __init__.py
    └── tzbtc
        ├── __init__.py
        ├── parameter
        │   ├── __init__.py
        │   ├── mint.py
        │   └── transfer.py
        └── storage.py

That's a lot of files and directories! But don't worry, we will need only models.py and handlers modules in this guide.

Define data models

Our schema will consist of a single model Holder having several fields:

• address — account address
• balance — in tzBTC
• volume — total transfer/mint amount bypassed
• tx_count — number of transfers/mints
• last_seen — time of the last transfer/mint

Put the following content in the models.py file:

from tortoise import fields

from dipdup.models import Model


class Holder(Model):
    address = fields.CharField(max_length=36, pk=True)
    balance = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    turnover = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    tx_count = fields.BigIntField(default=0)
    last_seen = fields.DatetimeField(null=True)

Implement handlers

Everything's ready to implement an actual indexer logic.

Our task is to index all the balance updates, so we'll start with a helper method to handle them. Create a file named on_balance_update.py in the handlers package with the following content:

from datetime import datetime
from decimal import Decimal

import demo_token.models as models


async def on_balance_update(
    address: str,
    balance_update: Decimal,
    timestamp: datetime,
) -> None:
    holder, _ = await models.Holder.get_or_create(address=address)
    holder.balance += balance_update
    holder.turnover += abs(balance_update)
    holder.tx_count += 1
    holder.last_seen = timestamp
    await holder.save()

Three methods of tzBTC contract can alter token balances — transfer, mint, and burn. The last one is omitted in this tutorial for simplicity. Edit corresponding handlers to call the on_balance_update method with data from matched operations:

on_transfer.py

from decimal import Decimal

from demo_token.handlers.on_balance_update import on_balance_update
from demo_token.types.tzbtc.parameter.transfer import TransferParameter
from demo_token.types.tzbtc.storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models import Transaction


async def on_transfer(
    ctx: HandlerContext,
    transfer: Transaction[TransferParameter, TzbtcStorage],
) -> None:
    if transfer.parameter.from_ == transfer.parameter.to:
        # NOTE: Internal tzBTC transfer
        return

    amount = Decimal(transfer.parameter.value) / (10**8)
    await on_balance_update(
        address=transfer.parameter.from_,
        balance_update=-amount,
        timestamp=transfer.data.timestamp,
    )
    await on_balance_update(
        address=transfer.parameter.to,
        balance_update=amount,
        timestamp=transfer.data.timestamp,
    )

on_mint.py

from decimal import Decimal

from demo_token.handlers.on_balance_update import on_balance_update
from demo_token.types.tzbtc.parameter.mint import MintParameter
from demo_token.types.tzbtc.storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models import Transaction


async def on_mint(
    ctx: HandlerContext,
    mint: Transaction[MintParameter, TzbtcStorage],
) -> None:
    amount = Decimal(mint.parameter.value) / (10**8)
    await on_balance_update(
        address=mint.parameter.to,
        balance_update=amount,
        timestamp=mint.data.timestamp,
    )

And that's all! We can run the indexer now.

Run your indexer

dipdup run

DipDup will fetch all the historical data and then switch to realtime updates. Your application data has been successfully indexed!