# IOTA Documentation - Operators > Complete operator documentation for running full nodes, validators, and infrastructure This file contains all documentation content in a single document following the llmstxt.org standard. ## Activate DoS Protection for Validators # Activating DoS Protection for Validators Validator nodes are critical infrastructure in the IOTA network and should be protected against Denial of Service (DoS) attacks. The TrafficController is a built-in mechanism that helps protect validator nodes by monitoring and limiting traffic from potential attackers. :::warning This guide explains how to configure the TrafficController for your validator node. A node whose configuration has no `policy-config` runs the shipped default policy. That default runs with `dry-run: true`, thus it only counts and logs the clients it would block. Set `dry-run: false` to make the node block them, for `connection-blocklist-ttl-sec` seconds each. ::: ## TrafficController Overview The TrafficController monitors incoming requests to your validator node and can block IP addresses that exceed certain thresholds. It can operate in two modes: 1. **Non-proxy mode** - For validators that receive traffic directly from clients. 2. **Proxy mode** - For validators behind load balancers or proxies. ## Configuration Parameters The TrafficController is configured through two main configuration structures: ### PolicyConfig This structure defines the traffic control policies: ```yaml # The following parameters are defined in the PolicyConfig struct, which controls # how the TrafficController identifies clients and applies rate limiting policies. # The struct provides configuration for both direct client connections and proxied # connections through load balancers. policy-config: # How to identify client IPs (socket-addr or x-forwarded-for) client-id-source: socket-addr # or x-forwarded-for: , where hops is the number of proxy hops # Serve these client IPs only, and refuse every other request. Rate limiting # does not run in this mode. Omit the field to rate limit instead. # allow-list: ["10.0.0.1", "10.0.0.2"] # How long to block direct client IPs (in seconds) connection-blocklist-ttl-sec: 60 # How long to block proxied client IPs (in seconds) proxy-blocklist-ttl-sec: 60 # Policy for handling spam traffic spam-policy-type: freq-threshold: client-threshold: 1000 # Sustained tallies per second per direct client; requests are sampled by spam-sample-rate, so the tolerated request rate is proportionally higher proxied-client-threshold: 10 # Sustained tallies per second per proxied client burst-secs: 30 # Burst allowance: a client can send client-threshold * burst-secs tallies back to back # Policy for handling error traffic error-policy-type: NoOp # NoOp, or a freq-threshold map like the spam policy above # Sampling rate for spam detection (0.0 to 1.0) spam-sample-rate: 0.2 # Whether to run in dry-run mode (log but don't block) dry-run: false ``` The `burst-secs` key had the name `window-size-secs` before. The node continues to accept the old name, but the old name is deprecated. Change your configuration to `burst-secs`. ## Non-Proxy Mode Configuration If your validator node receives traffic directly from clients (no proxy or load balancer in front), use this configuration: ```yaml policy-config: client-id-source: socket-addr connection-blocklist-ttl-sec: 60 proxy-blocklist-ttl-sec: 0 spam-policy-type: freq-threshold: client-threshold: 100 # Sustained tallies per second proxied-client-threshold: 10 # Currently ignored, as proxied client tallies are not reported; 0 would block every proxied client burst-secs: 30 error-policy-type: NoOp spam-sample-rate: 0.2 dry-run: false # Set to true initially to monitor without blocking ``` ## Proxy Mode Configuration If your validator is behind a load balancer or proxy, use this configuration: ```yaml policy-config: # Configure with the number of proxy hops between client and node # For example, if you have one proxy: client -> proxy -> node, use 1 client-id-source: x-forwarded-for: 1 # Adjust based on your infrastructure connection-blocklist-ttl-sec: 60 proxy-blocklist-ttl-sec: 60 spam-policy-type: freq-threshold: client-threshold: 1000 # Higher threshold for proxies that aggregate traffic proxied-client-threshold: 10 # Threshold for individual clients behind proxies burst-secs: 30 error-policy-type: NoOp spam-sample-rate: 0.2 dry-run: false ``` The node reads the entry your proxy appended, so a client cannot hide itself by putting unreadable bytes in the part of the header it sent. A proxy writes only addresses, so the node refuses a request whose selected entry is not valid UTF-8, whatever the policy. It likewise refuses a request whose connection has no peer address. The node still ends up with no client when the header is absent, when the entry it selects is not an address, or when the header holds fewer entries than the configured hop count. With a rate-limiting policy such a request is served and charged to no client, which means it is not rate limited. In allowlist mode it is refused, because an allowlist admits the clients it names. The `traffic_control_unresolved_client_requests` metric counts these requests. On a validator it does not sit at zero: other nodes call the validator directly, their requests carry no header, and each one is counted. Read it as a change from its usual level rather than as an absolute number. A rise above that level means that client traffic reaches the node on another path, or that the configured hop count does not match the number of proxies. On a node that serves clients only, and where every request arrives through the proxy, the metric does sit at zero. An `allow-list` on a validator has to name every other node that calls it, because those calls arrive on the same listener. Two configurations are possible, and only one of them works. With `client-id-source: socket-addr` the node resolves the address of each caller, so listing the other nodes does admit them, at the cost of a list that has to change with the committee. With `x-forwarded-for` listing them does not help at all: their requests carry no header, the node resolves no address to match against the list, and it refuses them whatever the list holds. Prefer to leave an `allow-list` off a validator. ### Determining the Correct Number of Hops While `x-forwarded-for` is set to `0` the node determines no client at all. With a rate-limiting policy that means no client is limited, and in allowlist mode it means every request is refused. Use this setting only to read the hop count off the logs, and set the correct value straight after. To determine the correct number of hops for the `x-forwarded-for` setting: 1. Set `x-forwarded-for: 0` temporarily. 2. Run the node and query any endpoint from a known IP address. 3. Check the logs for lines containing `x-forwarded-for`. 4. The correct value is 1 + the number of IP addresses that appear after your known IP in the header. ## What if I'm Unsure of the Number of Proxies? If you're unsure about the correct number of proxies to configure, you can use the [`config-traffic-control.sh`](https://github.com/iotaledger/iota/blob/develop/setups/validator/config-traffic-control.sh) script included in the IOTA codebase. This script analyzes your node logs to determine the correct configuration. Follow these steps: 1. Configure `x-forwarded-for` in the policy config to `0`, which is a special case setting that allows the node to provide logging details of the `x-forwarded-for` header contents: ```yaml policy-config: client-id-source: x-forwarded-for: 0 ``` 2. Restart your node. 3. Tail the node logs and pipe them into the script, providing as an additional argument an IP address of a known host that you can use to query any endpoint of your validator: ```bash journalctl -fu iota-node | setups/validator/config-traffic-control.sh "YOUR_KNOWN_IP_ADDRESS" ``` 4. Using the host whose IP was provided to the script, query any endpoint of your validator. The script will collect request data and provide a recommended configuration: ```bash journalctl -fu iota-node | setups/validator/config-traffic-control.sh "YOUR_KNOWN_IP_ADDRESS" x-forwarded-for contents: YOUR_KNOWN_IP_ADDRESS Configuration: client-id-source: x-forwarded-for: 1 ``` 5. Set the configuration as specified by the script. 6. Restart your node to apply the new configuration. The script automatically calculates the correct number of hops by analyzing the position of your known IP address in the `x-forwarded-for` header. ## Recommended Production Settings For production validator nodes, we recommend: ```yaml policy-config: # Use socket-addr for direct connections or x-forwarded-for for proxied setups client-id-source: socket-addr # Adjust based on your setup connection-blocklist-ttl-sec: 300 # Block for 5 minutes proxy-blocklist-ttl-sec: 300 # Block for 5 minutes spam-policy-type: freq-threshold: # Adjust these thresholds based on your expected legitimate traffic client-threshold: 50 # Sustained tallies per second proxied-client-threshold: 10 # Sustained tallies per second per client behind proxy burst-secs: 30 # Consider enabling error policy for production error-policy-type: freq-threshold: client-threshold: 20 # Error tallies per second proxied-client-threshold: 5 burst-secs: 30 spam-sample-rate: 0.5 # Sample half of all requests dry-run: false # Ensure this is false for actual protection ``` ## Monitoring and Tuning After enabling the `TrafficController`, monitor your node's metrics to ensure legitimate traffic isn't being blocked. Key metrics to watch: - `tallies` - `connection_ip_blocklist_len` - `proxy_ip_blocklist_len` - `requests_blocked_at_protocol` - `traffic_control_num_dry_run_blocked_requests` - `traffic_control_unresolved_client_requests` - `connection_blocklist_ttl_sec` - `proxy_blocklist_ttl_sec` Start with `dry-run: true` to observe what would be blocked without actually blocking traffic, then adjust thresholds as needed before setting `dry-run: false`. ## Troubleshooting If legitimate traffic is being blocked: 1. Increase the `client-threshold` and `proxied-client-threshold` values. 2. Decrease the `connection-blocklist-ttl-sec` and `proxy-blocklist-ttl-sec` values. 3. Increase `burst-secs` to permit larger bursts. If you're still experiencing DoS attacks: 1. Decrease the thresholds. 2. Increase the blocklist TTL values. --- ## Archives The checkpoint archive is a remote object store holding IOTA's checkpoint history since genesis — checkpoint summaries and their full contents — trailing the latest checkpoint by roughly 10 minutes. When a full node cannot retrieve checkpoints from its peers via state sync (for example, because they have pruned the range it needs), it falls back to downloading them from a pre-configured checkpoint archive, letting it catch up — even from genesis — regardless of its peers' pruning policies. Configure it with the `checkpoint-archive-config` block in `fullnode.yaml`, described in [Configure Checkpoint Archive Fallback](#configure-checkpoint-archive-fallback). A full node can also fall back to a cloud key-value store for individual historic transaction data (transaction digests, effects, events, and checkpoints) that it has pruned locally, described in [Key-value Store Backup](#iota-full-node-key-value-store-backup). ## IOTA Full Node Key-value Store Backup To enable historic data queries for the IOTA full nodes that prune old transactional data, full node RPC implementation is configured to fallback for querying missing transactional data from a remote store. If the information about the transaction digest, effects, events, or checkpoints is not available locally, a full node automatically retrieves the historical data from a cloud-based key-value store (currently managed by the IOTA Foundation). Note that the current key-value store implementation keeps historic transactional data only. ## Configure Checkpoint Archive Fallback To enable your node to fall back to the checkpoint archive when peers no longer serve the range it needs, add this to your `fullnode.yaml` file: ```yaml checkpoint-archive-config: # Use https://checkpoints.mainnet.iota.cafe/ingestion/historical for mainnet # Use https://checkpoints.testnet.iota.cafe/ingestion/historical for testnet url: "https://checkpoints..iota.cafe/ingestion/historical" # Optional. Number of checkpoints to download in parallel; defaults to 10. download-concurrency: 5 ``` --- ## Genesis Genesis is the initial state of the IOTA blockchain. To launch a network, the initial committee of validators collaborate by providing their validator information (public keys, network addresses, and so on) to a shared workspace. After all of the initial validators have contributed their information, IOTA generates the initial, unsigned genesis checkpoint (checkpoint with sequence number 0) and each validator provides their signature. IOTA aggregates these signatures to form a _certificate_ on the genesis checkpoint. IOTA bundles this checkpoint, as well as the initial objects, together into a single genesis.blob file that is used to initialize the state when running the `iota-node` binary for both validators and full nodes. ## Genesis blob locations The genesis.blob files for each network are located at: ```bash curl -fLJO https://dbfiles.mainnet.iota.cafe/genesis.blob curl -fLJO https://dbfiles.mainnet.iota.cafe/migration.blob ``` ```bash curl -fLJO https://dbfiles.testnet.iota.cafe/genesis.blob ``` --- ## Enable the JSON-RPC API, including the index it relies on for queries Sustainable disk usage requires IOTA full nodes to prune the information about historic object versions as well as historic transactions with the corresponding effects and events, including old checkpoint data. All three kinds of pruning below run in the background right after each epoch boundary and once when the node starts, where an expired epoch's data is dropped as a whole set of RocksDB column families; the node keeps executing transactions meanwhile. A lowered retention therefore takes effect at the next start. Most of the disk space comes back with those files. What is left is a small amount of row-by-row deletion in the live object table, which RocksDB reclaims through background compaction, so the last part of the effect on disk usage is not immediate — see `periodic-compaction-threshold-days` below. Once the node has caught up with the network and no read holds back the expiry of historic objects, the same background task also compacts the data of each epoch that has ended, once, so a lookup in it reads a single sorted run of SST files instead of many level-0 files. An epoch is skipped while the disk has less free space than its data takes plus 10%. On the first start after the upgrade, once the node has caught up, this compacts every epoch the node holds, which takes disk I/O for a while on a node with a long history. :::tip Testing indicates that aggressive pruning results in more efficient full node operation. ::: ## Types of Pruning ### Object Pruning IOTA adds new object versions to the database as part of transaction execution, which makes previous versions ready for garbage collection. Without pruning, this would result in database performance degradation and require large amounts of storage space. A superseded version is moved into its epoch's store as it is superseded, and the whole epoch is dropped once it falls outside the retention window. ### Transaction Pruning Transaction pruning removes previous transactions, their effects and events, and the corresponding checkpoint data from the database. Each is written into the store of the epoch that executed it, and the whole epoch is dropped once it falls outside the retention window. ### Index Pruning Index pruning drops the expired epochs of the `rpc_indexes` store, the single secondary index store both the JSON-RPC and the gRPC API read from, so older entries are no longer served by the history queries (`iotax_queryTransactionBlocks()` and `iotax_queryEvents()`). Balances, coins, owned objects and dynamic fields are read from tables of the live state, which index pruning does not touch. Without it, the index adds a set of tables per epoch and grows without bound. Index pruning is relevant for every node that maintains the store, which is any node serving the JSON-RPC API (`enable-jsonrpc-api: true`, the default) or the gRPC API (`enable-grpc-api: true`). On the first start after the upgrade to the per-epoch storage, a full node rebuilds its RPC index: the live state up front, the history in the background from the transactions the node still holds. Index history older than the node's checkpoint retention is not brought back, so `iotax_queryTransactionBlocks()` and `iotax_queryEvents()` reach no further back than that retention afterwards. Raising the retention before the upgrade does not help with transactions the node has already pruned; history queries older than that are for an indexer to answer. Back up the database before the first start after the upgrade. That start changes the database layout for good, and no older release must be started on the database afterwards: going back means restoring the backup or a snapshot. Before it changes anything an older release reads, the first start also refuses to start when `num-epochs-to-retain-for-checkpoints` is not set and the node holds transactions older than its default of 30 epochs keeps, since that start would delete them. Set it to `18446744073709551615` to keep them all, or to the number of epochs to keep. A start refused for this reason, or for too little disk space as described below, changes nothing an older release reads, so the older release can still be started on the database. The first start after the upgrade also needs free disk space for a second copy of what it keeps: - **Transactions and checkpoints:** the one-time migration copies the transactions, effects, events and checkpoints of the retained epochs into the per-epoch layout, and frees the old tables only once it is done. The node checks for this space before it starts the migration, and refuses to start with a message naming the space it needs if the disk is too small. The copy goes to `/historic`. - **Object versions on a node that never pruned them:** a node that ran with `num-epochs-to-retain: 18446744073709551615` copies every superseded object version once. The old tables' space comes back through later compaction, not at once. If the one-time move of superseded object versions fails on a database row it cannot read, and does so on every start, `iota-tool db-tool --db-path /live mark-object-backlog-swept`, run while the node is stopped, lets the node start without it. The versions it had not moved then stay in the live object table. The index window may be shorter than the checkpoint window, and the only thing that costs is query reach: `iotax_queryTransactionBlocks()` and other query-style endpoints stop returning entries older than the index window, while every transaction the checkpoint window still retains remains fetchable by digest on both APIs and still reports the checkpoint that confirmed it — that answer is read from the ledger, not from the index. ## Pruning Parameters Object pruning and transaction pruning are independent of one another, and each is gated by its own parameter. Index pruning is not independent: the index holds pointers into the transaction history, so it follows transaction pruning unless you give it a shorter window of its own. | Pruning task | Controlling parameter | Enabled when | | --------------------- | -------------------------------------- | --------------------------------------------------- | | Object pruning | `num-epochs-to-retain` | value is not `u64::MAX` | | Transaction pruning | `num-epochs-to-retain-for-checkpoints` | value is not `u64::MAX` | | Index pruning | `num-epochs-to-retain-for-indexes` | value is not `u64::MAX`, or transaction pruning is on when unset | All pruning parameters are configured in `fullnode.yaml`. The following annotated block shows every parameter with its default value — copy it into your config and adjust as needed: ```yaml # Enable the JSON-RPC API, including the index it relies on for queries # like ownership, balances, and events. Default: true. # Set to false for validators or nodes that do not serve JSON-RPC requests. # The gRPC API maintains the same index under its own flag (enable-grpc-api, # default: false), so a node with either of the two prunes its indexes. enable-jsonrpc-api: true authority-store-pruning-config: # Number of historic epochs of superseded object versions to keep, on top # of the epoch the node is in. Controls object pruning. # 0 — aggressive pruning (default). Keeps the versions superseded # during the epoch the node is in, and drops them at the next # epoch boundary. Lowest disk usage. Recommended for validators # and full nodes that do not serve historic object queries. # N (>= 1) — keeps the epoch the node is in plus the N epochs before it. # Use this when the node must serve lookups by object ID + # version. # u64::MAX — disables object pruning entirely (18446744073709551615). # Independent of num-epochs-to-retain-for-checkpoints below: historic object # versions are retained per epoch and dropped a whole epoch at a time, so # object pruning never reads a checkpoint's contents. num-epochs-to-retain: 0 # Number of historic epochs of transactions, effects, events, and checkpoint # data to keep, on top of the epoch the node is in. Controls transaction # pruning. Counted the same way as num-epochs-to-retain above. # 0 — refused at startup: peers still sync the previous epoch's # checkpoints right after an epoch boundary. # N — keeps the epoch the node is in plus the N epochs before it, # so N + 1 epochs in total. # u64::MAX — disables transaction pruning entirely (18446744073709551615). # unset — 30 (default), about a month at one-day epochs. The first # start after the upgrade to the per-epoch storage refuses to # start while unset if the node holds older transactions. # num-epochs-to-retain-for-checkpoints: 30 # Number of historic epochs to retain in the rpc_indexes store, on top of # the current one. The store serves both the JSON-RPC and the gRPC API. # Controls index pruning. Relevant whenever the node maintains that store # (enable-jsonrpc-api: true or enable-grpc-api: true). # 0 — keeps the epoch the node is in only. # N — keeps the epoch the node is in plus the N epochs before it. # u64::MAX — disables index pruning entirely (18446744073709551615). # unset — follows num-epochs-to-retain-for-checkpoints (default). # The epoch currently being executed is always retained, whatever this is # set to: a client that submits a transaction needs to read it back — the # transaction, its effects, and the index rows behind queries — seconds # later. Only at an epoch boundary is dropping that data acceptable. # May be shorter than num-epochs-to-retain-for-checkpoints above — the # only cost is query reach, since every retained transaction stays # fetchable by digest and still reports its checkpoint regardless of # index retention. It must not exceed that value: a node that maintains # the store refuses to start otherwise, since iotax_queryTransactionBlocks() # would then return transactions the ledger has already pruned. This is why # leaving it unset follows the transaction retention rather than turning # index pruning off. # num-epochs-to-retain-for-indexes: 2 # Number of recent per-epoch RocksDB databases to keep. Default: 3. num-latest-epoch-dbs-to-retain: 3 # Trigger periodic compaction for SST files older than this many days. # Reclaims disk space and avoids fragmentation. Default: 1 (when read from # config). Set to "null" to disable periodic compaction entirely. periodic-compaction-threshold-days: 1 ``` All three run right after the epoch boundary and once at the node's start, where the node drops the epochs that have fallen outside each retention window, and all three count that window the same way: `N` historic epochs on top of the epoch the node is in, keeping `N + 1` epochs in total. `u64::MAX` turns a pruning task off. Leaving `num-epochs-to-retain-for-checkpoints` unset keeps `30` historic epochs, and leaving `num-epochs-to-retain-for-indexes` unset follows the transaction retention: the index is a set of pointers into the transaction history, so it follows that history's retention rather than outliving it. :::info From `2` up, `num-epochs-to-retain-for-checkpoints` keeps one epoch more than it did in earlier releases, where the same value kept `N` epochs in total, and values below `2` were raised to `2`. Transaction and checkpoint history therefore takes roughly 50% more disk at the commonly used value of `2` — size disks accordingly, or lower the value by one to keep the previous window. Two further changes in the same release: `0` is now refused at startup, where it used to be raised to `2` like `1`, and leaving `num-epochs-to-retain-for-indexes` unset now follows the transaction retention instead of turning index pruning off. A node that prunes transactions and left the index setting unset therefore starts pruning its index history, which had been growing without bound. ::: :::warning Leaving `num-epochs-to-retain-for-checkpoints` unset used to keep the whole transaction and checkpoint history. It now keeps `30` historic epochs. A node that relied on the old default no longer carries the older epochs over in the one-time migration at its first start on this release, and drops them for good. To keep the whole history, set the value to `18446744073709551615` before upgrading. ::: :::info Even at `num-epochs-to-retain: 0`, the versions superseded during the epoch the node is in stay on disk until the next epoch boundary, where earlier releases deleted them within a few checkpoints. Disk usage therefore climbs over the course of an epoch and falls back at the boundary. Size disks for one epoch of superseded versions on top of the live state: the size of the newest `/live/historic/epoch_/objects` directory just before an epoch boundary shows how much that is on your network. ::: ## Where the History Is Kept The retained epochs of all three kinds of history are kept in one directory inside the live database directory, `/live/historic`, with a subdirectory per epoch: ```text /live/historic/ epoch_594/ objects/ superseded object versions ledger/ transactions, effects and events checkpoints/ checkpoint contents and summaries indexes/ JSON-RPC and gRPC index history epoch_595/ ... ``` An expired epoch's directory is removed together with its data. ## Configuration Examples The following examples cover common operator use cases. Each example shows the key parameters for the scenario — parameters not listed use their defaults from the [reference block above](#pruning-parameters). ### Validator or Minimal Full Node This configuration keeps disk usage to a minimum. A node with this setup cannot answer queries that require indexing or historic data. It is the recommended setup for IOTA validators and for full nodes that do not serve RPC requests. ```yaml # Do not serve the JSON-RPC API, or maintain the index it depends on enable-jsonrpc-api: false authority-store-pruning-config: # Aggressively prune historic object versions num-epochs-to-retain: 0 # Prune historic transactions of past epochs num-epochs-to-retain-for-checkpoints: 2 ``` ### Full Node with Indexing but no History This setup manages secondary indexing in addition to the latest state, but aggressively prunes historic data. A full node with this configuration: - Answers RPC queries that require indexing, like `iotax_getBalance()`. - Answers RPC queries that require historic transactions via a fallback to retrieve the data from a remote key-value store: `iota_getTransactionBlock()`. - Cannot answer RPC queries that require historic object versions: `iota_tryGetPastObject()`. - The `showBalanceChanges` filter of `iota_getTransactionBlock()` relies on historic object versions, so it cannot work with this configuration. ```yaml authority-store-pruning-config: # Aggressively prune historic object versions num-epochs-to-retain: 0 # Prune historic transactions of past epochs num-epochs-to-retain-for-checkpoints: 2 ``` ### Full Node with Full Object History but Pruned Transaction History This configuration manages the full object history while still pruning historic transactions. A full node with this configuration can answer all historic and indexing queries (using the transaction query fallback for transactional data), including the ones that require historic objects such as the `showBalanceChanges` filter of `iota_getTransactionBlock()`. The main caveat is disk usage: the full object history grows without bound. You should closely monitor the disk space growth on a full node with this configuration. ```yaml authority-store-pruning-config: # No pruning of object versions (use u64::max for num of epochs) num-epochs-to-retain: 18446744073709551615 # Prune historic transactions of the past epochs num-epochs-to-retain-for-checkpoints: 2 ``` --- ## Snapshots import FormalSnapshotPicker from '@site/src/components/FormalSnapshotPicker'; Snapshots capture a node's view of the IOTA network at the end of an epoch. :::info While validators can enable snapshots, they are typically most valuable for full node operators. Therefore, in the following sections, we'll focus on snapshots from the perspective of full node operators. ::: Snapshots of the IOTA network enable full node operators a way to bootstrap a full node without having to execute all the transactions that occurred after genesis. You can upload snapshots to remote object stores like S3, Google Cloud Storage, Azure Blob Storage, and similar services. These services typically run the export process in the background so there is no degradation in performance for your full node. With snapshots stored in the cloud, you're more easily able to recover quickly from catastrophic failures in your system or hardware. To maintain a healthy IOTA network, IOTA encourages the IOTA community to bring up additional snapshots to ensure stronger data availability across the network. ## Formal snapshots :::info Formal snapshots are not suitable for use if you are running an RPC node that does any historical data lookups. For more information on node data management, see [Data Management](../data-management.mdx). ::: Formal snapshots are database-agnostic, minimalistic snapshots containing only essential data for node restoration at a specific epoch, keeping their storage footprint small and restoration fast. Natively supported by IOTA protocol, this means the snapshot contents can be cryptographically verified against the committee's epoch-end commitment, a process that occurs automatically during restoration (unless explicitly bypassed). The formal snapshots have the following properties: 1. **Minimalism:** Formal snapshots contain the end of epoch live object set (the set of all object versions eligible for use as input objects for future transactions) plus a small per-epoch metadata file covering every epoch since genesis. IOTA syncs all other critical chain information from the chain or derives it. Thus, formal snapshots contain only the necessary data required for a node to startup at epoch boundary and participate in the network. 2. **Agnosticism:** Formal snapshots are by nature agnostic to the underlying database choice or implementation of the protocol. As the live object set is protocol defined, so is the formal snapshot. 3. **Verifiability:** Formal snapshots have first-class support in the core IOTA protocol. As such, they must be trustless/verifiable by the node operator upon download. To support this, the protocol signs a commitment to the live object state at end of epoch, which formal snapshots are checked against at restore time. If this verification fails, the node state is moved back to the state before the snapshot restore attempt. Because these snapshots do not contain transaction history, they are most immediately useful for validators and state sync full nodes (SSFNs). Restoring from a formal snapshot also builds the RPC index store by default (pass `--skip-rpc-indexes` to opt out), so a full node started with the JSON-RPC API, the gRPC API, or both, serves them right away without re-indexing the restored state. You can upload snapshots to remote object stores like S3, Google Cloud Storage, Azure Blob Storage, and similar services. These services typically run the export process in the background so there is no degradation in performance for your full node. With snapshots stored in the cloud, you can recover from catastrophic failures in your system or hardware more efficiently. ## Enabling snapshots A nodes does not take snapshots by default. To enable this feature you must apply specific configs to your node. Follow these steps to change the configs for the node: 1. Stop your node, if it's running. 2. Open your `fullnode.yaml` file and apply config updates as the following. Using Amazon's S3 service as an example: ```yaml state-snapshot-write-config: object-store-config: object-store: "S3" bucket: "``" aws-access-key-id: "``" aws-secret-access-key: "``" aws-region: "``" object-store-connection-limit: 200 concurrency: 10 ``` - `object-store`: The remote object store to upload snapshots. Set as Amazon's `S3` service in the example. - `bucket`: The S3 bucket name to store the snapshots. - `aws-access-key-id` and `aws-secret-access-key`: AWS authentication information with write access to the bucket. - `aws-region`: Region where bucket exists. - `object-store-connection-limit`: Number of simultaneous connections to the object store. - `concurrency`: Number of concurrent uploads of snapshots to the object store. 3. Save the `fullnode.yaml` file and restart the node. ## Restoring a full node from snapshots To restore from a Formal snapshot, use the `iota-tool` binary. `iota-tool` can be downloaded along with other iota binaries. See [Install IOTA](../../developer/getting-started/install-iota.mdx#install-from-binaries) for more details. The following steps can be used to restore a node from a Formal snapshot: 1. If it's running, stop the node. 2. Pick your setup, network, and a snapshot epoch. Copy the resulting command and run it. The Docker command mounts `$PWD/data/db` and `$PWD/data/config/genesis.blob`, the layout used by the [Docker setup](../full-node/docker.mdx), so run it from that setup's directory or adjust the mounts to your own paths: - `--epoch` / `--latest`: Pin a specific snapshot epoch, or always grab the most recent one. - `--genesis`: The path to the location of the network's `genesis.blob`. - `--network`: Network to download the snapshot for: `mainnet`, `testnet`, or `devnet`. Defaults to `mainnet`. - `--path`: Must equal the `db-path` from your node's configuration (default `/opt/iota/db/authorities_db`). - `--skip-rpc-indexes`: Skips building the RPC index store during the restore. By default it is built from the same object stream that restores the state, so a full node started with the JSON-RPC API, the gRPC API, or both, opens it in place instead of re-indexing the whole restored state on first start. Both index groups are always built together, since the restore cannot know which of the two APIs the node will end up serving; skip it only if the node will serve neither. - `--disable-progress-bar`: Reports progress as a status line logged once per second instead of drawing progress bars. ### Backfilling the full checkpoint summary history A node restored from a formal snapshot holds only the end-of-epoch checkpoint summaries (one per epoch). That is enough to run, but the node cannot serve historical checkpoint queries below its restore point, nor act as a complete summary source for peers syncing the header chain. To fill in every intermediate summary from the checkpoint archive, stop the node and run: ```shell iota-tool backfill-checkpoint-summaries \ --path "/live" \ --ingestion-url "https://checkpoints..iota.cafe/ingestion/historical" \ --verbose ``` - `--path`: The node's live database directory (the one containing `checkpoints/`, `store/`, and `epochs/`). - `--ingestion-url`: URL of the checkpoint archive to download summaries from — the same archive the node's state-sync fallback reads from (for example `https://checkpoints..iota.cafe/ingestion/historical`). - `--num-parallel-downloads`: Number of summaries to download in parallel. Defaults to a value based on the number of available cores, capped at 8. - `--verbose`: Enable verbose logging. - `--disable-progress-bar`: Reports progress as a status line logged once per second instead of drawing a progress bar. Summaries are taken as-is from `--ingestion-url` without chain verification, so point it at the checkpoint archive you trust to serve this node's own chain. This only adds historical summaries below the node's existing watermarks — it never changes how far the node has synced or executed, and it never overwrites the summaries the node already has. It is safe to re-run. ## IOTA Foundation managed snapshots The IOTA Foundation provides the public snapshot access: - **Formal**: https://formal-snapshot.mainnet.iota.cafe - **Formal**: https://formal-snapshot.testnet.iota.cafe - **Formal**: https://formal-snapshot.devnet.iota.cafe --- ## Data Management Managing the data on your IOTA full node is critical to ensuring a healthy IOTA network. This topic provides a high-level description of data management on IOTA full nodes that you can use to optimize your full node configuration. ## Basic IOTA Full Node Functionality The minimal version of an IOTA full node executes all of the transactions IOTA validators commit. IOTA full nodes also orchestrate the submitting of new transactions to the system: ![Basic IOTA full node functionality](/img/operator/data-management/fn-basic-functionality.png "A diagram that shows the operations of a basic IOTA full node.") The preceding image shows how data flows through a full node: 1. **`State sync` protocol:** An IOTA full node performs the following to achieve state synchronization: - Retrieves the information about the committed checkpoints via the p2p gossip-like protocol - Executes the transactions locally to verify that effects match the effects certified by the quorum of the validators - Updates the local state with the latest object values accordingly. 2. **RPCs:** An IOTA full node exposes IOTA RPC endpoints for querying the latest state of the system, including both the latest state metadata (such as, `iota_getProtocolConfig`), and the latest state object data (`iota_getObject)`. 3. **Transaction submission:** Each IOTA full node orchestrates transaction submission to the quorum of the IOTA Validators, and, optionally if configured, locally executes the finalized transactions (called fast path execution), which circumvents the wait for checkpoint synchronization. ## IOTA full node Data and RPC types An IOTA full node stores multiple categories of data in its permanent store. :::info The per-epoch IOTA store is beyond the scope of this topic. IOTA uses the per-epoch store (resets at the start of each epoch) internally for authority and consensus operations. ::: IOTA full nodes store the following types of data: 1. **Transactions with associated effects and events:** IOTA uses a unique transaction digest to retrieve information about a transaction, including its effects and emitted events. IOTA full nodes don't require the historic transaction information for basic full node operations. To conserve disk space, you can enable pruning to remove this historical data. 2. **Checkpoints:** IOTA groups committed transactions in checkpoints, and then uses those checkpoints to achieve state synchronization. Checkpoints keep transaction digests that contain additional integrity metadata. IOTA full nodes don't require data from checkpoints to execute and submit transactions. 3. **Objects:** Transactions that mutate objects create new object versions. Each object has a unique pair of `(objectId, version)` used to identify the object. IOTA full nodes don't require historic object versions to execute and submit transactions. 4. **Indexing information:** A full node default configuration is to post-process the committed transactions: it indexes the committed information to enable efficient aggregation and filtering queries. For example, the indexing can be useful for retrieving all the historic transactions of a given sender, or finding all the objects owned by an address. IOTA full nodes support more than 40 RPC types that includes the following categories: * **General metadata**, such as `iota_getProtocolConfig` and `iota_getChainIdentifier`. These requests don't depend on additional indexing and don't require historic data to process. * **Direct lookups**, such as `iota_getObject`, `iota_getEvents`. These requests don't depend on additional indexing, but require historic data in some cases, such as `iota_tryGetPastObject` and `iota_getTransactionBlock`. * **Accumulation and filtering queries**,such as `iotax_getOwnedObjects` and `iotax_getCoins`. These requests depend on additional indexing, and require historic data in some cases, such as `iotax_queryTransactionBlocks`. :::info IOTA plans to migrate the RPC endpoints that require additional indexing away from IOTA full nodes in the near future. This plan decouples the storage that is backing transaction execution from the storage that is better suited for data indexing. ::: --- ## GraphQL The `iota-graphql-rpc` binary serves a GraphQL endpoint backed by the IOTA Indexer database. See the [Extended Data Services overview](./overview.mdx) for how it fits together with the indexer and the JSON-RPC service. ## Scope The GraphQL service answers GraphQL queries by reading data from the IOTA Indexer database. It exposes reads of checkpoints, transactions, events, objects (including past versions within the retained window), Move packages and types, balances, dynamic fields, and the on-chain name registry. GraphQL is ideal for applications that require rich query patterns over structured data. Two operations are not served from the database: `executeTransactionBlock` and `dryRunTransactionBlock`. Both are forwarded to a fullnode over gRPC. The GraphQL service holds no state of its own. Multiple instances can run in parallel against the same database. ## Dependencies - An indexer Postgres database. The schema version in the database must match the one compiled into the GraphQL binary; otherwise the GraphQL service refuses to start. For the `executeTransactionBlock` operation to work properly the indexer writer should be running and synced with the tip of the network. See [IOTA Indexer](./iota-indexer.mdx) for details on running the indexer writer. - A fullnode gRPC endpoint. Required at start-up — the server fails to start if `--node-rpc-url` is not set, even when no client issues a mutation or a dry run. - An optional REST KV store for historic fallback. When configured, it lets the GraphQL service serve data already pruned by the indexer writer. Currently experimental. See [Pruned data](#pruned-data) below and [Pruning](./iota-indexer.mdx#pruning) for the indexer-side retention model. ## Hardware requirements - CPU: 2 cores per instance - Memory: 4 GB per instance Scale up the number of instances when client request load grows. ## Running the GraphQL service A minimal start command: ```sh iota-graphql-rpc start-server \ --db-url postgres://:@:5432/ \ --node-rpc-url http://: \ --config service.toml ``` Other subcommands: - `iota-graphql-rpc generate-config ` writes a fully-populated TOML with the current defaults. - `iota-graphql-rpc generate-schema --file ` writes the active GraphQL schema in SDL format. Useful when generating client code. ## Configuration Settings come from two places: - CLI flags on the `start-server` subcommand. - A TOML file passed with `--config `. If no file is passed, the defaults apply. A starting TOML with every field set to its default can be generated with: ```sh iota-graphql-rpc generate-config ``` The most important settings are covered in subsections below. All settings are listed in the [reference table](#all-settings) at the bottom of the page. ### Database connection `--db-url` sets the Postgres connection URL (e.g. `postgres://:@:5432/`); see the [Postgres docs](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) for the full URI syntax. `--db-pool-size` sets the connection pool size — raise it for higher concurrency, but stay within the Postgres `max_connections` setting and account for other clients of the same database. ### Fullnode endpoint `--node-rpc-url` is the gRPC URL of a fullnode that the GraphQL service forwards `executeTransactionBlock` and `dryRunTransactionBlock` to. It is required at start-up. ### Pruned data :::warning Experimental The historic fallback depends on the API from the `iota-rest-kv` crate which is still being finalised. Treat the flags below as experimental — they may change or be replaced. ::: Data already [pruned](./iota-indexer.mdx#pruning) by the indexer writer can be served by the GraphQL service using a historic fallback feature. To use it, the operator runs a separate self-hosted REST KV store and points the GraphQL service at it via `--fallback-kv-url`. When `--fallback-kv-url` is unset (the default), no fallback is performed and pruned data is unreachable via GraphQL. These queries fall back to the KV store when their data has been pruned from the database: - `checkpoint`, `checkpoints`, `Epoch.checkpoints` - `transactionBlock`, `transactionBlocksByDigests` - `transactionBlocks` with filter `atCheckpoint`, `affectedAddress`, or `transactionIds`, and the matching `Checkpoint.transactionBlocks` and `Address.transactionBlocks` (`AFFECTED` relation) - `events` with filter `transactionDigest` - `object` at a past version, including dynamic fields resolved at a parent object's version :::note Resolving a dynamic field or dynamic object field at a parent object's version needs that version recorded in the indexer's `objects_version` table. On indexers restored from a snapshot, versions below the restore point cannot be served this way and return `DATA_PRUNED`. ::: The `--fallback-kv-multi-fetch-batch-size`, `--fallback-kv-concurrent-fetches`, and `--fallback-kv-cache-size` knobs tune the KV client's request batching, concurrency, and in-memory cache. `--fallback-kv-max-response-size-bytes` caps the size of a single response body accepted from the KV store, larger responses are rejected. ### HTTP server `--host` and `--port` set the address the GraphQL endpoint binds to (default `127.0.0.1:8000`). The interactive GraphiQL IDE is served on the same address; its page title comes from `--ide-title`. ### Response headers Every response carries two headers that clients use to check compatibility: | Header | Value | | ------------------------ | --------------------------------------------------------------------------------- | | `x-iota-rpc-version` | Full version of the running binary, including the git sha. | | `x-iota-min-sdk-version` | Lowest `iota-sdk-graphql-client` version able to decode this service's responses. | The minimum SDK version is fixed at build time and cannot be configured. It is raised in releases whose responses an older SDK cannot decode, so clients on an older SDK reject the service instead of panicking on unknown values. ### Prometheus metrics The Prometheus endpoint exposes histograms and counters covering request rate, latency, errors, database pool utilisation, and watermark lag. `--prom-host` and `--prom-port` set the address the Prometheus metrics are exposed on (default `0.0.0.0:9184`). ### Disabled features Operators can disable selected groups of queries using the `disabled-features` TOML parameter. The available groups are: | Name | Paths | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `coins` | `Query.coinMetadata`; `Address.balance`/`balances`/`coins`; `Object.balance`/`balances`/`coins`; `Owner.balance`/`balances`/`coins` | | `dynamic-fields` | `Object.dynamicField`/`dynamicFields`/`dynamicObjectField`; `Owner.dynamicField`/`dynamicFields`/`dynamicObjectField` | | `subscriptions` | `Subscription.events`; `Subscription.transactions` | | `system-state` | `Query.protocolConfig`; `Epoch.protocolConfigs`/`referenceGasPrice`/`validatorSet`; `SystemStateSummary.safeMode`/`storageFund`/`systemParameters`/`systemStateVersion` | For instance, with `coins` disabled, the following query is rejected because `Query.coinMetadata` and `Address.balance` are both part of the `coins` group: ```graphql { coinMetadata(coinType: "0x2::iota::IOTA") { decimals symbol } address(address: "0x...") { balance(type: "0x2::iota::IOTA") { totalBalance } } } ``` Multiple values allowed, example: ```toml disabled-features = ["subscriptions", "coins"] ``` ### Watermark task The service maintains an in-process watermark that tracks the maximum checkpoint and epoch readable from the database. The `background-tasks.watermark-update-ms` TOML key sets how often the watermark is refreshed (default `500` ms). This controls how fresh data is served through GraphQL queries as data above the watermark is filtered out from the responses. ### Max available range A *consistent view* fixes the database state at a single checkpoint for the duration of a paginated query. Without it, paginating through objects could mix state from different checkpoints — page 2 might miss objects that were deleted after page 1 was returned, or include objects that were created in the meantime. GraphQL keeps a window of recent checkpoints over which it can reconstruct historical object state from the indexer's `objects_backward_history` table. `--max-available-range` (also settable via the `MAX_AVAILABLE_RANGE` environment variable) sets the size of that window, in checkpoints. Larger values let pagination cursors on objects stay valid longer, at a cost that older cursors introduce more database load. The value must not exceed the retention of the historical tables on the indexer side (`objects_backward_history`). If it does, GraphQL may return incorrect results for data already pruned by indexer writer. Default value of 9000 allows pagination cursors on objects to stay alive for ≈ 30 minutes. ### IOTA Names resolution The `[iota-names]` TOML table configures the on-chain name registry the service resolves names against. The defaults match mainnet. Override the package address and registry IDs when running against a different network: ```toml [iota-names] package-address = "0x..." object-id = "0x..." payments-package-address = "0x..." registry-id = "0x..." reverse-registry-id = "0x..." ``` ## All settings ### CLI flags | Flag | Default | Description | | ------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `--host` | `127.0.0.1` | Host the GraphQL endpoint binds to. | | `--port`, `-p` | `8000` | Port the GraphQL endpoint binds to. | | `--db-url`, `-d` | `postgres://postgres:postgrespw@localhost:5432/iota_indexer` | Postgres connection URL. | | `--db-pool-size` | `10` | Number of Postgres connections held in the pool. | | `--prom-host` | `0.0.0.0` | Host the Prometheus metrics endpoint binds to. | | `--prom-port` | `9184` | Port the Prometheus metrics endpoint binds to. | | `--skip-migration-consistency-check` | `false` | Skip the start-up check that local migrations match the database. | | `--max-available-range` | `9000` | Maximum window (in checkpoints) in which consistent view on objects can be served. Also `MAX_AVAILABLE_RANGE` env var. | | `--node-rpc-url` | _(required)_ | Fullnode gRPC URL used for `executeTransactionBlock` and `dryRunTransactionBlock`. | | `--fallback-kv-url` | _(none)_ | Experimental. REST KV store URL for historic fallback. Disabled when unset. | | `--fallback-kv-multi-fetch-batch-size` | `100` | Experimental. Maximum number of keys per batch request to the fallback KV store. Also `FALLBACK_KV_MULTI_FETCH_BATCH_SIZE` env var. | | `--fallback-kv-concurrent-fetches` | `10` | Experimental. Maximum number of concurrent batch requests to the fallback KV store. Also `FALLBACK_KV_CONCURRENT_FETCHES` env var. | | `--fallback-kv-cache-size` | `100000` | Experimental. In-memory cache size for historic fallback. Also `FALLBACK_KV_CACHE_SIZE` env var. | | `--fallback-kv-max-response-size-bytes` | `~ 134 MB` | Experimental. Maximum size in bytes of a single response body accepted from the fallback KV store. Also `FALLBACK_KV_MAX_RESPONSE_SIZE_BYTES` env var. | | `--ide-title`, `-i` | `IOTA GraphQL IDE` | Title displayed at the top of the GraphiQL IDE. | | `--config`, `-c` | _(none)_ | Path to a TOML file with service configuration; see below. | ### TOML `[limits]` The `[limits]` table controls how heavy a single query can be. | Key | Default | Description | | ---------------------------- | ----------: | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `max-query-depth` | `20` | Maximum depth of nodes in a request. | | `max-query-nodes` | `300` | Maximum number of nodes in a request. | | `max-output-nodes` | `100000` | Maximum number of output nodes allowed in the response. | | `max-tx-payload-size` | `174763` | Maximum size in bytes for `txBytes` and `signatures` in `executeTransactionBlock`, and `txBytes` in `dryRunTransactionBlock`. | | `max-query-payload-size` | `5000` | Maximum size in bytes of the JSON payload of a read request (excluding `max-tx-payload-size`). | | `max-db-query-cost` | `20000` | Queries whose `EXPLAIN` cost exceeds this are logged. Units are Postgres planner units (≈ one sequential page access). | | `default-page-size` | `20` | Page size used when a paginated query does not specify one. | | `max-page-size` | `50` | Maximum page size a paginated query may request. | | `mutation-timeout-ms` | `74000` | Server-side timeout for mutation requests, in milliseconds. The underlying transaction may continue to execute on the fullnode after the timeout fires. | | `request-timeout-ms` | `40000` | Server-side timeout for read requests, in milliseconds. | | `max-type-argument-depth` | `16` | Maximum nesting of generic type arguments. | | `max-type-argument-width` | `32` | Maximum number of type parameters a type can have. | | `max-type-nodes` | `256` | Maximum size of a fully qualified type. | | `max-move-value-depth` | `128` | Maximum depth of a Move value. | | `max-transaction-ids` | `1000` | Maximum number of transaction ids accepted by `TransactionBlockFilter` and `transactionBlocksByDigests`. | | `max-scan-limit` | `100000000` | Maximum number of candidate rows the database scans when assembling a page. | | `max-subscription-recovery-checkpoints` | `20000` | Maximum number of checkpoints a subscription replays from the database when resuming from `startAfter`. An older item is rejected. | | `max-active-subscriptions` | `1000` | Maximum number of subscriptions active at the same time across all connections. | ### TOML — other tables | Key | Default | Description | | ------------------------------------ | ------------------------ | ------------------------------------------------------------------------------------------ | | `disabled-features` | `[]` | List of functional groups removed from the schema. See [Disabled features](#disabled-features). | | `background-tasks.watermark-update-ms` | `500` | How often the watermark task refreshes the maximum readable checkpoint and epoch, in milliseconds. | | `iota-names.*` | mainnet IDs | On-chain name registry the service resolves names against. See [IOTA Names resolution](#iota-names-resolution). | | `versions` | crate `major.minor` | GraphQL schema versions advertised through `serviceConfig.availableVersions`. | --- ## IOTA Indexer import IndexerRestorePicker from '@site/src/components/IndexerRestorePicker'; The `iota-indexer` binary in writer mode ingests checkpoints from an IOTA checkpoint source, writes them into PostgreSQL tables, and maintains that database over time (migrations on start-up, pruning per retention policy). See the [Extended Data Services overview](./overview.mdx) for how it fits together with the JSON-RPC and GraphQL reader services. The `iota-indexer` binary also has a `run-backfill` subcommand — a one-off maintenance tool for filling specific tables over a historical range, typically after a schema change. Not part of normal operation. It also has a `restore` subcommand. This is a one-off tool that bootstraps the database from a formal snapshot instead of replaying every checkpoint from genesis. See [Restoring from a formal snapshot](#restoring-from-a-formal-snapshot). ## Scope The writer: - Downloads checkpoints from a remote source (a fullnode gRPC endpoint or a historical checkpoint store). - Indexes each checkpoint into structured Postgres tables (objects, transactions, events, indexes for `tx_*` and `event_*` filters, and more). - Runs the needed schema migrations on the database on start-up. A database ahead of the binary (migrations the binary doesn't know about) fails to start; a database behind the binary is migrated forward. - Optionally runs background pruning tasks that enforce per-table retention policies. See [Pruning](#pruning) below. - Exposes Prometheus metrics for monitoring, e.g. ingestion lag, indexing throughput, etc. The writer is the main component that writes to the indexer database. Reader services ([JSON-RPC](./json-rpc.mdx), [GraphQL](./graphql.mdx)) read from the same database. They write only to `optimistic_transactions`, `objects`, and `display`, as part of optimistic indexing. Run a single writer instance per database. Running more than one against the same database is not supported. ## Dependencies - A PostgreSQL database. The database must not be ahead of the binary; on start-up the writer applies any pending migrations and refuses to start if the database contains migrations the binary doesn't know about. See [Appendix: PostgreSQL configuration](#appendix-postgresql-configuration) for a recommended `postgresql.conf`. - A checkpoint source. At least one of: - `--remote-store-url` pointing at a [fullnode](../full-node/overview.mdx) gRPC endpoint (e.g. `http://0.0.0.0:50051`). - `--remote-store-url` pointing at a historical checkpoint store hosting batched checkpoint files. - `--data-ingestion-path` pointing at a local directory that a colocated fullnode populates with checkpoint files. - Optionally a live checkpoint store (`--live-checkpoints-store-url`) that serves the latest checkpoints not yet in the historical archive. Combine it with a historical `--remote-store-url` to get full coverage from genesis with low tip latency. See the **Hybrid Historical Checkpoint Store** section in the [overview](./overview.mdx). URLs for the publicly hosted historical and live checkpoint stores are listed under [Checkpoint sources](#checkpoint-sources). ## Hardware requirements Disk is the main requirement and depends on the configured retention. See the [sizing table](#sizing-and-retention) under Pruning for predicted database sizes. Run the writer on the same network as Postgres to keep latency low. ## Running the writer A minimal start command, ingesting from a fullnode gRPC endpoint into a local Postgres database, without pruning: ```sh iota-indexer \ --db-url postgres://:@:5432/ \ indexer \ --remote-store-url http://:50051 ``` Alternatively, ingesting from a historical checkpoint store combined with a live checkpoint store for minimal tip latency: ```sh iota-indexer \ --db-url postgres://:@:5432/ \ indexer \ --remote-store-url https://checkpoints.mainnet.iota.cafe/ingestion/historical \ --live-checkpoints-store-url https://checkpoints.mainnet.iota.cafe/ingestion/live ``` For networks other than `mainnet`, use the matching URLs from the [Checkpoint sources](#checkpoint-sources) section below. ## Configuration Settings come from CLI flags and matching environment variables on the `indexer` subcommand. The only configuration file the writer reads is the optional pruning retention TOML described in the [Pruning configuration file (TOML)](#pruning-configuration-file-toml) section below. All flags are listed in the [reference tables](#all-settings) at the bottom of the page. ### Database connection - `--db-url` — Postgres connection URL (e.g. `postgres://:@:5432/`). See the [Postgres docs](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) for the full URI syntax. - `--pool-size` — connection pool size. Raise it for higher concurrency, but stay within the Postgres `max_connections` setting and account for other clients of the same database. - `--connection-timeout` — caps how long a request waits for a free database connection before failing. - `--statement-timeout` — caps how long a single SQL query may run before it's cancelled. ### Checkpoint sources At least one source must be configured. The CLI accepts: - `--remote-store-url` — a fullnode gRPC URL or a historical checkpoint store URL. A historical checkpoint store covers all checkpoints from genesis; a fullnode only serves the checkpoints it hasn't pruned. Available historical stores: - Mainnet: `https://checkpoints.mainnet.iota.cafe/ingestion/historical` - Testnet: `https://checkpoints.testnet.iota.cafe/ingestion/historical` - Devnet: `https://checkpoints.devnet.iota.cafe/ingestion/historical` - `--data-ingestion-path` — a local filesystem directory that a colocated fullnode populates with checkpoint files. - `--live-checkpoints-store-url` — a live checkpoint store URL. Used as a fallback for the latest checkpoints that aren't yet in the historical archive. Requires `--remote-store-url` to also be set. Available live stores: - Mainnet: `https://checkpoints.mainnet.iota.cafe/ingestion/live` - Testnet: `https://checkpoints.testnet.iota.cafe/ingestion/live` - Devnet: `https://checkpoints.devnet.iota.cafe/ingestion/live` ### Colocated fullnode `--data-ingestion-path` requires a colocated fullnode dumping each executed checkpoint into a shared directory. Set `data-ingestion-dir` in the fullnode YAML and point the writer's `--data-ingestion-path` at the same path. ### Prometheus metrics The Prometheus endpoint exposes ingestion lag, counts of committed checkpoints, transactions, and epochs, indexing latencies, database connection pool size, etc. `--metrics-address` sets the address the Prometheus metrics endpoint binds to (default `0.0.0.0:9184`). ### Resetting the database Passing `--reset-db` on the `indexer` subcommand drops every table on start-up and re-applies all migrations from scratch. Use it only on a writer instance you intend to fully rebuild — the operation is destructive and affects any reader sharing the same database. It starts ingestion from scratch. Use with care. ## Pruning A running IOTA Indexer accumulates one row per checkpoint, transaction, and event the network produces, so the database grows without bound as long as the network progresses. For deployments that intend to run for months or years, that growth may not be sustainable. The indexer addresses this with **opt-in pruning**: the operator declares a **retention period** (a number of most recent epochs) per table, and the indexer deletes anything older on each epoch boundary. Data older than the retention period becomes inaccessible to reads against the indexer database. A few tables are never pruned; see [Non-prunable tables](#non-prunable-tables) for the list. Because retention is configured per table, the operator decides which data to retain. Shrinking the retention window of a table shortens the time range over which queries backed by that table return results. The effect of pruning a table depends on its role and the queries it backs: - **Point lookups** (`transactions`, `events`, `checkpoints`) - direct lookup of a transaction by digest, an event by digest, etc. Trying to fetch an old digest whose data has been pruned will result in an error, unless a KV Store fallback is configured. For point-lookup tables, configuring a fallback lets the data be served for both pruned and unpruned ranges. - **Filtered queries** (`tx_*`, `event_*`) - list transactions or events by attribute (sender, recipient, emitter package, etc.). Pruned transactions/events will be omitted from the result, with no explicit error returned. This case cannot be handled by the fallback service. - **Historical object tables** (`objects_backward_history`) - back queries that read past versions of objects (e.g., consistent reads at a checkpoint, dynamic-field state at a specific point in time). Pruning shortens the time range over which such historical-state queries can be served; queries that target a checkpoint older than the retained range will fail or return missing data. - **Optimistic tables** (`optimistic_transactions`) - Data in those tables is needed only for short periods of time and doesn't need long retention. It can be safely pruned with no side effects on data availability, as long as current and previous epochs are kept. The subsections below cover sizing and retention guidance, how to enable pruning, the configuration file format, the schedule on which deletions happen, how to monitor pruner progress, how to reclaim disk space, and a full inventory of prunable and non-prunable tables. ### Sizing and retention Assuming a sustained 500 user TPS, the predicted DB size at typical retention windows is: | Retention | DB size | | ---: | ---: | | 2 days | ~604 GB | | 7 days | ~1.6 TB | | 30 days | ~6.3 TB | | 1 year | ~74 TB | We recommend **30 epochs of retention** as a default for all tables, and **2 epochs** for `objects_backward_history` and `optimistic_transactions`. ### Enabling pruning Pruning is enabled by passing `--pruning-config-path ` to the `indexer` subcommand, pointing at a TOML file that describes retention policies. If no config path is supplied, pruning is disabled. A sync worker with pruning enabled is started like this: ```sh iota-indexer \ --db-url \ indexer \ --remote-store-url \ --pruning-config-path ``` ### Pruning configuration file (TOML) The TOML file contains a default retention (in epochs) and an optional table of per-table overrides: ```toml # Default retention applied to every prunable table (in epochs). epochs_to_keep = 10 # Per-table overrides. Keys are the snake_case table names listed below. [overrides] transactions = 5 events = 5 tx_senders = 3 ``` Every retention value — both the default and each override — must be strictly greater than zero. **Defaults.** When the indexer resolves a final retention map, every prunable table receives the configured `epochs_to_keep` value. Any value supplied in `[overrides]` replaces the default for that table. **Selective pruning.** The configuration model is all-or-nothing at the worker level: there is no flag to skip a specific table. Once pruning is enabled, the pruner evaluates every prunable table. To approximate per-table opt-out, set a very large `epochs_to_keep` override for any table you want to retain. For example, to prune only `tx_senders` and keep the rest effectively forever: ```toml # Default high enough that no real network will ever reach it. epochs_to_keep = 1_000_000 [overrides] tx_senders = 5 ``` **Inter-table dependencies.** Some tables only return useful data when their dependencies are present. For example, the `tx_*` index tables (`tx_senders`, `tx_calls_*`, `tx_changed_objects`, …) are looked up via the `tx_digests` table; setting a shorter retention on `tx_digests` than on the `tx_*` indexes means the indexes still hold rows that can no longer be resolved to a transaction digest. Keep retention on `tx_digests` at least as long as the retention on any other `tx_*` table that depends on it. ### How pruning is applied The pruner runs as a set of background tasks inside the sync worker process: - A **watermarks-update task** polls every few seconds. When it detects that the network has advanced to a new epoch, it recomputes each prunable table's lower-bound watermark from the retention policy and writes the new bound into the `watermarks` table. - A **per-table pruner task** runs in parallel for each prunable table, polling every few seconds. When that table's watermark has been advanced, the task waits a configurable delay (`--pruning-delay-ms`, default **2 hours**) before actually deleting data — the delay protects in-flight reads against losing rows mid-query. It then deletes in batches sized by `--pruning-batch-size` (default `1000`), sleeping briefly between chunks to limit pressure on the database. As a result, disk usage drops some time after the epoch transition, not at the boundary itself. Likewise, when a sync worker is started with pruning enabled for the first time, deletions will not begin sooner than the configured `--pruning-delay-ms` after the worker starts. ### Monitoring pruning progress Pruner state is exposed via the `watermarks` table — one row per table (every prunable table plus a few bookkeeping entries). Useful columns: | Column | Meaning | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `entity` | Name of the table this row describes. | | `current_epoch` | Most recent epoch ingestion has written into the table. | | `min_available_epoch` / `min_available_cp` / `min_available_tx` | Lowest epoch / checkpoint / transaction sequence number still queryable. Anything below is scheduled for pruning. | | `lowest_unpruned_key` | Next epoch / checkpoint / transaction sequence number the pruner will actually delete from the database. Anything below has already been removed. | | `min_bounds_updated_at_timestamp_ms` | Timestamp of the last lower-bound update. The 2-hour deletion delay is measured from this value. | ### Storage reclamation Enabling pruning on a previously unpruned indexer — or shortening retention on an already-pruned one — does not immediately return the freed disk space to the operating system. Pruning deletes rows from Postgres; autovacuum reclaims the dead tuples for future inserts but does not shrink the on-disk footprint of tables and indexes. What pruning does guarantee is bounded growth going forward: pruned tables stop accumulating data past the configured retention window. To recover the existing on-disk footprint, run a compaction tool against the database (for example [`VACUUM FULL`](https://www.postgresql.org/docs/18/routine-vacuuming.html#VACUUM-FOR-SPACE-RECOVERY), [`REINDEX TABLE CONCURRENTLY`](https://www.postgresql.org/docs/18/sql-reindex.html#SQL-REINDEX-CONCURRENTLY), [`pg_repack`](https://reorg.github.io/pg_repack/), or [`pg_squeeze`](https://github.com/cybertec-postgresql/pg_squeeze)). Each option has different operational caveats — `VACUUM FULL` holds an `ACCESS EXCLUSIVE` lock for the full table rewrite, while the others run online but need temporary disk space. Review the linked Postgres docs before running any of these against a live database. The IOTA Indexer does not invoke these itself. To prevent pruned tables from re-bloating over time, autovacuum must run aggressively enough to keep up with the pruner's deletes. The following per-table autovacuum settings are a good starting point for every prunable table: ``` vacuum_index_cleanup = on autovacuum_vacuum_scale_factor = 0.01 autovacuum_vacuum_cost_limit = 500 ``` On Docker-hosted Postgres, also raise `shm_size` to at least `2g` — the default `64M` is too small for autovacuum to complete on large prunable tables, which silently disables the cleanup. ### Prunable tables Tables are pruned using one of three strategies, determined by how the table is structured: | Strategy | Mechanism | Disk reclaimed | Tables | | ----------------------- | ------------------------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **By epoch partition** | Drop the partition for each retired epoch. | Immediately | `transactions`, `events` | | **By checkpoint** | `DELETE` rows below a checkpoint cutoff. | After compaction | `checkpoints`, `pruner_cp_watermark`, `objects_backward_history` | | **By transaction** | `DELETE` rows below a transaction-sequence cutoff. | After compaction | `event_emit_package`, `event_emit_module`, `event_senders`, `event_struct_instantiation`, `event_struct_module`, `event_struct_name`, `event_struct_package`, `tx_calls_pkg`, `tx_calls_mod`, `tx_calls_fun`, `tx_changed_objects`, `tx_digests`, `tx_input_objects`, `tx_kinds`, `tx_recipients`, `tx_senders`, `tx_wrapped_or_deleted_objects`, `tx_global_order`, `optimistic_transactions` | Tables not listed here are never pruned by the indexer. ### Non-prunable tables Several tables hold data that is retained for the full lifetime of the database. They are not affected by any retention setting and will not shrink when pruning is enabled. The largest contributors are `objects_version`, `objects_snapshot`, and `objects`. ## Restoring from a formal snapshot The IOTA Indexer has the ability to initialize the database from a [formal snapshot](../common/snapshots.mdx#formal-snapshots) of the network instead of ingesting every checkpoint from genesis. A formal snapshot is a minimal, cryptographically verifiable capture of the end-of-epoch object state. Full nodes bootstrap from the same snapshots (see [formal-snapshot restore](../common/snapshots.mdx#restoring-a-full-node-from-snapshots)). For a target epoch, restoring the Indexer involves downloading the snapshot, verifying it against the committee chain, and writing the live object state plus the epoch, protocol, and other data derived from it (display, packages, object-version mapping). The CLI command to manage the underlying operations is `restore`, which exposes `run` and `available-epochs`: ```sh iota-indexer restore --network run [OPTIONS] --staging-path Local directory to stage the downloaded MANIFEST and .ref files --genesis-path Path to the genesis blob used to verify the snapshot --epoch Epoch to download [default: latest available] --num-parallel-downloads Parallel downloads [default: logical cores - 1, capped at 8] ``` where `--genesis-path` is the network's [genesis blob](../common/genesis.mdx). It is the trusted resource the committee chain is verified against. The snapshot is in turn verified against that committee chain. To get the epochs for which a formal snapshot is available use the following command: ```sh # epochs are returned as a sorted JSON array, newest first iota-indexer restore --network available-epochs [| jq .[i] ] # i: 0,1,2,... ``` When the restore finishes it leaves the database at the target epoch's final checkpoint. So, all prunable history tables start populating from the checkpoints that follow the restore point. Thus, one caveat when selecting the target epoch and configuring a retention policy is that the restore epoch must sit at least a full retention window behind the tip of the network. To keep the recommended [30 epochs of data](#sizing-and-retention), restore from an epoch about 30 epochs behind the tip. For convenience, the interactive picker below generates the restore command. Leave the epoch on **Latest** for the most recent snapshot, and adjust `DATABASE_URL` and `GENESIS_PATH`. The snapshots are served from the public endpoints listed under [IOTA Foundation managed snapshots](../common/snapshots.mdx#iota-foundation-managed-snapshots). :::warning Destructive operation Like [`--reset-db`](#resetting-the-database), `restore` drops every table and re-applies all migrations before restoring. Run it only against a database you intend to rebuild or initialize, since it affects any reader sharing the same database. ::: ## Appendix: PostgreSQL configuration The `postgresql.conf` below is the recommended configuration for an indexer host with roughly 96 GB of RAM and NVMe storage (mainnet-class hardware). Scale memory and parallelism values to your hardware. ```conf # Connections max_connections = 2000 listen_addresses = '*' superuser_reserved_connections = 5 # Memory shared_buffers = 32GB effective_cache_size = 64GB work_mem = 64MB maintenance_work_mem = 2GB temp_buffers = 64MB wal_buffers = 64MB # Write-ahead log wal_level = replica synchronous_commit = off full_page_writes = on wal_compression = off wal_writer_delay = 200ms checkpoint_completion_target = 0.9 checkpoint_timeout = 15min min_wal_size = 4GB max_wal_size = 32GB # Query planner default_statistics_target = 200 random_page_cost = 1.0 effective_io_concurrency = 200 # Parallelism max_worker_processes = 24 max_parallel_workers = 16 max_parallel_workers_per_gather = 4 max_parallel_maintenance_workers = 4 # Logging log_min_duration_statement = 500 log_checkpoints = on log_autovacuum_min_duration = 250 log_temp_files = 512kB log_statement = 'none' log_line_prefix = '%t [%p]: [%l-1] user=%u db=%d app=%a client=%h ' # Other huge_pages = try max_files_per_process = 10000 archive_mode = off # Timeouts statement_timeout = 6000 ``` Per-table autovacuum tuning and `shm_size` guidance that pair with this config are documented in [Storage reclamation](#storage-reclamation) above. ## All settings ### Top-level `iota-indexer` flags | Flag | Env var | Default | Description | | ------------------------ | ------------------------ | ---------------- | ----------------------------------------------------------------- | | `--db-url` | — | _(required)_ | Postgres connection URL. Also accepted as `--database-url`. | | `--pool-size` | `DB_POOL_SIZE` | `100` | Number of parallel Postgres connections held in the pool. | | `--connection-timeout` | `DB_CONNECTION_TIMEOUT` | `30` (seconds) | How long a request waits for a free database connection before failing. | | `--statement-timeout` | `DB_STATEMENT_TIMEOUT` | `3600` (seconds) | Cap on how long a single SQL query may run before it's cancelled. | | `--metrics-address` | — | `0.0.0.0:9184` | Address the Prometheus metrics endpoint binds to. | ### `indexer` subcommand flags | Flag | Env var | Default | Description | | ------------------------------------------ | ---------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------ | | `--remote-store-url` | — | _(none)_ | Fullnode gRPC URL or historical checkpoint store URL serving batched checkpoint files. | | `--live-checkpoints-store-url` | — | _(none)_ | Live checkpoint store URL. Fallback for the latest checkpoints not yet in the historical archive. Requires `--remote-store-url`. | | `--data-ingestion-path` | — | _(none)_ | Local directory populated with checkpoint files by a colocated fullnode. | | `--checkpoint-download-queue-size` | `DOWNLOAD_QUEUE_SIZE` | `200` | Maximum number of checkpoints buffered for indexing. | | `--checkpoint-download-queue-size-bytes` | `CHECKPOINT_PROCESSING_BATCH_DATA_LIMIT` | `20000000` | Byte cap on the checkpoint download queue. | | `--checkpoint-download-timeout` | `INGESTION_READER_TIMEOUT_SECS` | `20` | Per-checkpoint download timeout, in seconds. | | `--pruning-config-path` | — | _(none)_ | Path to a TOML file describing per-table retention policies. Enables pruning when set. See [Pruning](#pruning). | | `--pruning-delay-ms` | `PRUNING_DELAY_MS` | `7200000` (ms) | Delay between a watermark's lower bound being advanced and the pruner deleting rows. Lets in-flight reads finish before their data disappears. | | `--pruning-batch-size` | `PRUNING_BATCH_SIZE` | `1000` | Upper bound on units (checkpoints, transactions, or global sequence numbers) pruned per chunk, and on rows deleted per statement. | | `--reset-db` | — | `false` | Drop all tables and re-apply migrations from scratch on start-up. Starts ingestion from scratch. Use with care. | At least one of `--remote-store-url` or `--data-ingestion-path` must be set. --- ## JSON-RPC The JSON-RPC reader is started by the `iota-indexer` binary with the `json-rpc-service` subcommand and serves JSON-RPC queries backed by the IOTA Indexer database. See the [Extended Data Services overview](./overview.mdx) for how it fits together with the indexer and the GraphQL service. ## Scope The JSON-RPC service answers JSON-RPC requests by reading data from the IOTA Indexer database. It exposes reads of checkpoints, transactions, events, objects, coins, balances, governance state, and Move utilities. The full list of methods is in the [IOTA JSON-RPC API reference](/iota-api-ref). Two operations are not served from the database: `iota_executeTransactionBlock` and `iota_dryRunTransactionBlock`. Both are forwarded to a fullnode over gRPC. A transaction executed via `iota_executeTransactionBlock` is visible immediately on point-lookup endpoints — `iota_getTransactionBlock`, `iota_getEvents`, and current-version object reads such as `iota_getObject` — because the write path stages its data into the indexer database before the call returns. Query-style endpoints (`iotax_queryTransactionBlocks`, `iotax_queryEvents`) and historical-version object reads (`iota_tryGetPastObject`) only see the new transaction once the indexer writer has processed the checkpoint that contains it, typically a few hundred milliseconds later. The JSON-RPC service holds no state of its own. Multiple instances can run in parallel against the same database. Paginated reads are not consistent across pages: the underlying tables can be updated by the indexer between requests, so a page may include or skip rows that a previous page would have. Clients that need consistent pagination should use the [GraphQL service](./graphql.mdx) instead. ## Dependencies - An indexer Postgres database. The database must already be migrated to at least the schema version compiled into the `iota-indexer` binary; otherwise the service refuses to start. A database with additional applied migrations beyond what the binary knows is tolerated, so a newer writer can be paired with an older JSON-RPC reader during rolling deployments as long as migrations are run first. For the `iota_executeTransactionBlock` method to work properly the indexer writer should be running and synced with the tip of the network. See [IOTA Indexer](./iota-indexer.mdx) for details on running the indexer writer. - A fullnode gRPC endpoint. Required at start-up — the server fails to start if `--rpc-client-url` is not set, even when no client issues an execute or dry-run request. - An optional REST KV store for historic fallback. When configured, it lets the JSON-RPC reader serve point lookups for data already pruned by the indexer writer. Currently experimental. See [Pruned data](#pruned-data) below and [Pruning](./iota-indexer.mdx#pruning) for the indexer-side retention model. - An optional analytical worker. A subset of `iotax_*` methods is served from tables populated by a separate `analytical-worker` process. When it isn't running, those methods return empty or zeroed results. See [Analytics](#analytics) below. ## Hardware requirements - CPU: 2 cores per instance - Memory: 4 GB per instance Scale up the number of instances when client request load grows. ## Running the JSON-RPC service A minimal start command: ```sh iota-indexer \ --db-url postgres://:@:5432/ \ json-rpc-service \ --rpc-client-url http://: ``` ## Configuration The JSON-RPC reader takes all configuration from CLI flags on the `iota-indexer` and `json-rpc-service` subcommand. There is no config file. The most important settings are covered in subsections below. All settings are listed in the [reference tables](#all-settings) at the bottom of the page. ### Database connection `--db-url` sets the Postgres connection URL (e.g. `postgres://:@:5432/`); see the [Postgres docs](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) for the full URI syntax. `--pool-size` sets the connection pool size — raise it for higher concurrency, but stay within the Postgres `max_connections` setting and account for other clients of the same database. `--connection-timeout` caps how long a request waits for a free database connection before failing. `--statement-timeout` caps how long a single SQL query may run before it's cancelled. ### Fullnode endpoint `--rpc-client-url` is the gRPC URL of a fullnode that the JSON-RPC reader forwards `iota_executeTransactionBlock` and `iota_dryRunTransactionBlock` to. It is required at start-up. ### RPC endpoint `--rpc-address` sets the address the JSON-RPC endpoint binds to (default `0.0.0.0:9000`). ### Prometheus metrics The Prometheus endpoint exposes histograms and counters covering request rate, latency, errors, and database pool utilisation. `--metrics-address` sets the address the Prometheus metrics endpoint binds to (default `0.0.0.0:9184`). ### Pruned data :::warning Experimental The historic fallback depends on the API from the `iota-rest-kv` crate which is still being finalised. Treat the flags below as experimental — they may change or be replaced. ::: Data already [pruned](./iota-indexer.mdx#pruning) by the indexer writer can be served by the JSON-RPC reader using a historic fallback feature. To use it, the operator runs a separate self-hosted REST KV store and points the JSON-RPC service at it via `--fallback-kv-url`. When `--fallback-kv-url` is unset (the default), no fallback is performed and pruned data is unreachable via JSON-RPC. The following methods consult the fallback when the indexer database has no matching data: - `iota_getCheckpoint`, `iota_getCheckpoints` - `iota_getTransactionBlock` - `iota_getEvents` - `iota_tryGetPastObject`, `iota_tryGetObjectBeforeVersion`, `iota_tryMultiGetPastObjects` - `iotax_queryTransactionBlocks` with filter `Checkpoint` or `FromOrToAddress` - `iotax_queryEvents` with filter `Transaction` The `--fallback-kv-multi-fetch-batch-size`, `--fallback-kv-concurrent-fetches`, and `--fallback-kv-cache-size` knobs tune the KV client's request batching, concurrency, and in-memory cache. `--fallback-kv-max-response-size-bytes` caps the size of a single response body accepted from the KV store, larger responses are rejected. ### Analytics A subset of `iotax_*` methods reads analytical metrics that the indexer writer does not populate. They are populated by a separate analytical worker process, started from the `iota-indexer` binary: ```sh iota-indexer \ --db-url postgres://:@:5432/ \ analytical-worker ``` The analytical worker runs on the same Postgres database as the indexer writer, so it increases load on the database. Operators should run it only when the methods below are needed. On a database whose history does not go back to genesis, for example one restored from a snapshot or one the pruner has moved past, the worker resumes from the first checkpoint and transaction still present. Metrics before that point stay absent. The cumulative address counts in `iotax_getLatestAddressMetrics`, `iotax_getCheckpointAddressMetrics`, `iotax_getAllEpochAddressMetrics` and `total_addresses` in `iotax_getNetworkMetrics` only include addresses seen from that point on. When the analytical worker is not running, these methods return empty or zeroed results: - `iotax_getMoveCallMetrics` - `iotax_getLatestAddressMetrics`, `iotax_getCheckpointAddressMetrics`, `iotax_getAllEpochAddressMetrics` - `current_tps` and `tps_30_days` in `iotax_getNetworkMetrics` ### IOTA Names resolution The `--iota-names-*` CLI flags configure the on-chain name registry the service resolves names against. The defaults match mainnet — override the package address and registry IDs when running against a different network. These settings are only used by the IOTA Names methods: `iotax_iotaNamesLookup`, `iotax_iotaNamesReverseLookup`, and `iotax_iotaNamesFindAllRegistrationNFTs`. ## All settings ### Top-level `iota-indexer` flags | Flag | Env var | Default | Description | | ------------------------ | ------------------------ | ---------------- | ----------------------------------------------------------------- | | `--db-url` | — | _(required)_ | Postgres connection URL. Also accepted as `--database-url`. | | `--pool-size` | `DB_POOL_SIZE` | `100` | Number of Postgres connections held in the pool. | | `--connection-timeout` | `DB_CONNECTION_TIMEOUT` | `30` (seconds) | How long a request waits for a free database connection before failing. | | `--statement-timeout` | `DB_STATEMENT_TIMEOUT` | `3600` (seconds) | Cap on how long a single SQL query may run before it's cancelled. | | `--metrics-address` | — | `0.0.0.0:9184` | Address the Prometheus metrics endpoint binds to. | ### `json-rpc-service` subcommand flags | Flag | Env var | Default | Description | | --------------------------------------- | -------------------------------------- | -------------- | -------------------------------------------------------------------------------------------- | | `--rpc-address` | — | `0.0.0.0:9000` | Address the JSON-RPC endpoint binds to. | | `--rpc-client-url` | — | _(required)_ | Fullnode gRPC URL used for `iota_executeTransactionBlock` and `iota_dryRunTransactionBlock`. | | `--fallback-kv-url` | — | _(unset)_ | Experimental. REST KV store URL for historic fallback. Disabled when unset. | | `--fallback-kv-multi-fetch-batch-size` | `FALLBACK_KV_MULTI_FETCH_BATCH_SIZE` | `100` | Experimental. Maximum number of keys per batch request to the fallback KV store. | | `--fallback-kv-concurrent-fetches` | `FALLBACK_KV_CONCURRENT_FETCHES` | `10` | Experimental. Maximum number of concurrent batch requests to the fallback KV store. | | `--fallback-kv-cache-size` | `FALLBACK_KV_CACHE_SIZE` | `100000` | Experimental. In-memory cache size for historic fallback. | | `--fallback-kv-max-response-size-bytes` | `FALLBACK_KV_MAX_RESPONSE_SIZE_BYTES` | `~134 MB` | Experimental. Maximum size in bytes of a single response body accepted from the fallback KV store. | | `--iota-names-package-address` | — | mainnet ID | Address of the `iota_names` package. | | `--iota-names-object-id` | — | mainnet ID | ID of the `IotaNames` object. | | `--iota-names-payments-package-address` | — | mainnet ID | Address of the `payments` package. | | `--iota-names-registry-id` | — | mainnet ID | ID of the registry table. | | `--iota-names-reverse-registry-id` | — | mainnet ID | ID of the reverse registry table. | --- ## Extended Data Services Overview The IOTA Indexer is an off-chain service that makes IOTA network data easy to query. It ingests data from an IOTA node, stores it in a PostgreSQL database, and serves it through query APIs [JSON-RPC](./json-rpc) and [GraphQL](./graphql). ## Functions The IOTA Indexer provides three main functionalities: - Ingest data from a full node to a relational database. Data on full nodes is stored as BCS bytes in an embedded RocksDB database, thus the query capabilities are very limited. The indexer pulls checkpoint blob data, schematizing them into various tables like objects, transactions, and so on, with proper indices. :::info Hybrid Historical Checkpoint Store If the full node does not retain its historical data, the indexer can be configured to ingest from a **hybrid historical checkpoint store** instead. This combines two sources: - A **historical checkpoint store** with the full archive from genesis to the network tip. - An optional **live checkpoint store** storing only current-epoch checkpoints, used for real-time ingestion. ```shell iota-indexer \ --db-url \ indexer \ --remote-store-url https://checkpoints..iota.cafe/ingestion/historical \ --live-checkpoints-store-url https://checkpoints..iota.cafe/ingestion/live ``` For more info see the [reference table](/developer/advanced/custom-indexer#historical-checkpoint-store). ::: - Serve online transaction processing (OLTP) RPC requests. With data in relational databases, IOTA indexer spins a stateless reader binary as JSON RPC server with the associated [interface](/iota-api-ref). - Other than OLTP data ingestion and requests, the Indexer also supports some analytical data ingestion like transactions per second (TPS) and daily active user (DAU) metrics. :::info Indexer Worker One Indexer instance can provide one of the above mentioned functionalities. They can be enabled with the corresponding subcommands in the Indexer CLI like: - `indexer` - `json-rpc-service` - `analytical-worker` Most RPC calls mentioned in [interface](/iota-api-ref) will only work with an indexer running. This is always mentioned in the RPC call description if it is the case. Indexer RPC calls that provide metrics are provided by the analytical worker. ::: ## Run an Indexer - [**IOTA Indexer**](./iota-indexer) — ingest checkpoint data into a local or managed Postgres-compatible database, with optional pruning to control storage growth. - [**IOTA Indexer JSON-RPC**](./json-rpc) — serve online transaction processing (OLTP) RPC requests against the indexed data. ## Run GraphQL Interface Alternatively to the JSON-RPC, a GraphQL [interface](../../developer/references/iota-graphql.mdx) can be used to provide read access to the Indexer database, and enable execution of transactions to the full node. A running Indexer database instance is required for the GraphQL API to be functional, so it is advised that an Indexer sync worker is setup and started first. For more information, see [GraphQL](./graphql). --- ## Archives(Configs) import ArchivesContent from '../../common/archives.mdx'; --- ## Genesis(Configs) import GenesisText from '../../common/genesis.mdx'; --- ## Network ## Connectivity To synchronize the state, the node needs to establish connections with other nodes that have already joined the network. Ensure that port `8084` is open and set up an external address for node discovery. The full node uses the following ports by default: :::info Default Firewall Ports Make sure to open these ports in your firewall configuration. (These are the defaults, but may vary for a custom setup) | Protocol/Port | Reachability | Purpose | Description | | ------------- | ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------- | | TCP/8080 | Outbound | Validator/transaction interface | full nodes connect to validators to collect partial signatures for transactions | | UDP/8084 | Inbound/Outbound | Peer-to-peer state sync interface | nodes sync their state with each other | | TCP/9000 | Inbound | JSON RPC endpoint | API for interaction with the network | | TCP/9184 | Localhost | Prometheus metrics scraping | for local monitoring only | ::: ## Known peers By default, your config file (`fullnode.yaml`) should include the following known peers (from public nodes run by the IOTA Foundation): ```yaml p2p-config: seed-peers: - address: /dns/access-0.r.mainnet.iota.cafe/udp/8084 peer-id: 66b55de43e925417761f3c66cd734451b5876e1496b961052095bdd8a7732189 - address: /dns/access-1.r.mainnet.iota.cafe/udp/8084 peer-id: 7f8dc46616df5b5640ffd118c6ef640b3f7f5cae8a2bedfa700769e12150a197 - address: /dns/access-2.r.mainnet.iota.cafe/udp/8084 peer-id: c9fbf6fb511fc5243989c7a455d36f277844288b1a1d8453331c200b6ac4d99f - address: /dns/access-3.r.mainnet.iota.cafe/udp/8084 peer-id: 2c1e05e9bddaf505b70184baf21a188a7c270b1b58f38d55e3aea42d067c5fb6 ``` ```yaml p2p-config: seed-peers: - address: /dns/access-0.r.testnet.iota.cafe/udp/8084 peer-id: 46064108d0b689ed89d1f44153e532bb101ce8f8ca3a3d01ab991d4dea122cfc - address: /dns/access-1.r.testnet.iota.cafe/udp/8084 peer-id: 8ffd25fa4e86c30c3f8da7092695e8a103462d7a213b815d77d6da7f0a2a52f5 ``` ```yaml p2p-config: seed-peers: - address: /dns/access-0.r.devnet.iota.cafe/udp/8084 peer-id: 01589ac910a5993f80fbc34a6e0c8b2041ddc5526a951c838df3037e11ab0188 - address: /dns/access-1.r.devnet.iota.cafe/udp/8084 peer-id: 32875c547ea3b44fa08a711646cedb70fa0c97959d236578131505da09723add ``` --- ## Pruning import PruningContent from '../../common/pruning.mdx'; --- ## Snapshots(Configs) import SnapshotsContent from '../../common/snapshots.mdx'; --- ## Full Node Configuration The IOTA full node configuration file is a YAML file named `fullnode.yaml`, located in your iota install folder (if you used docker) or in `/opt/iota/config` (if you used systemd). You can modify it to configure your IOTA node. To simplify the setup process, you can use the default configuration file: ```yaml file=/setups/fullnode/fullnode-mainnet.yaml ``` ```yaml file=/setups/fullnode/fullnode-testnet.yaml ``` ```yaml file=/setups/fullnode/fullnode-devnet.yaml ``` :::info If you used the installer script, it will have generated that config file for you, with default ports, archive and pruning settings, and also covers peering configuration for the selected network. ::: ## Customize your node's configuration ### Networking To synchronize the state, the node needs to establish connections with other nodes that have already joined the network. Ensure that port `8084` is open and set up an external address for node discovery. See more useful ports and network configuration options [here](configs/network.mdx) ### Storage and pruning (optional) To keep disk usage in check or save storage space, you can: - prune old transaction data (transactions, effects, events, checkpoints) - prune old object versions - turn off the built-in blockchain indexer Learn how to choose and set pruning settings [here](configs/pruning.mdx). ### Archival (optional) If the peers your node syncs with have already pruned data your node needs, your node can fallback onto using archives (files containing historical data). Learn how to configure archive settings for your node [here](configs/archives.mdx). ### Snapshots (optional) If your node fails, it can catch up faster from a recent database snapshot (on restart) than replaying all past transactions since genesis. Learn how to configure formal snapshots [here](configs/snapshots.mdx). ### Data storage paths (optional) To set paths for data files, change these values in `fullnode.yaml`: ```yaml # Set a custom path for storing the node's database db-path: "/db-files/iota-fullnode" # Set a custom path to the genesis blob genesis: genesis-file-location: "/iota-fullnode/genesis.blob" # Set a custom path for the migration blob migration-tx-data-path: "/opt/iota/config/migration.blob" ``` ## Restart your node after making changes After changing any of the node's settings, restart your node using: ```shell sudo systemctl restart iota-node ``` ```shell docker compose restart ``` --- ## Docker Setup Use this guide to install and configure an full node locally using Docker. :::warning Advanced These instructions are for setting up a full node for network participation. If you just need a local development environment, you should instead follow the instructions in [Create a Local IOTA Network](/developer/getting-started/local-network) to create a local full node, validators, and faucet. ::: ## Hardware requirements - **RAM:** 64 GB - **CPU:** 8 physical cores / 16 vCPUs - **Storage:** 512 GB NVMe SSD (or more depending on pruning settings) - **Network Uplink:** 1 Gbps - **RAM:** 32 GB - **CPU:** 4 physical cores / 8 vCPUs - **Storage:** 512 GB NVMe SSD (or more depending on pruning settings) - **Network Uplink:** 1 Gbps ## Software requirements Before you begin, ensure you have the following: - [Docker](https://docs.docker.com/get-docker/) installed. - [Docker Compose](https://docs.docker.com/compose/install/) installed. :::info Default Firewall Ports Make sure to open these ports in your firewall configuration. (These are the defaults, but may vary for a custom setup) | Protocol/Port | Reachability | Purpose | Description | | ------------- | ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------- | | TCP/8080 | Outbound | Validator/transaction interface | full nodes connect to validators to collect partial signatures for transactions | | UDP/8084 | Inbound/Outbound | Peer-to-peer state sync interface | nodes sync their state with each other | | TCP/9000 | Inbound | JSON RPC endpoint | API for interaction with the network | | TCP/9184 | Localhost | Prometheus metrics scraping | for local monitoring only | ::: ## Download the Full Node Docker Setup In a directory of your choice, run the following: ```shell mkdir iota-fullnode-docker-setup && cd iota-fullnode-docker-setup && curl -L https://fullnode-docker-setup.iota.org/mainnet | tar -zx ``` ```shell mkdir iota-fullnode-docker-setup && cd iota-fullnode-docker-setup && curl -L https://fullnode-docker-setup.iota.org/testnet | tar -zx ``` ```shell mkdir iota-fullnode-docker-setup && cd iota-fullnode-docker-setup && curl -L https://fullnode-docker-setup.iota.org/devnet | tar -zx ``` The command will perform the following steps: - creates and changes into the `iota-fullnode-docker-setup` directory - downloads the zipped configuration for the specific network, containing the docker-compose and node configuration files, and unzips them You can modify the node configuration file (`data/config/fullnode.yaml`) according to your needs. ## Prepare your Full Node Run the following command in the same directory: ```shell ./prepare.sh ``` This script will perform the following steps: - download the latest docker images specified in the docker-compose.yaml - download the network specific snapshot files that are needed to sync the node from genesis ## Configure P2P Settings Configure the P2P settings in your `fullnode.yaml` file by following these steps: 1. Update the external address: ```yaml p2p-config: external-address: /dns//udp/8084 ``` :::note Replace `` with your public hostname. In case you don't have a public hostname, you can alternatively use a public IPv4 like `/ip4/1.2.3.4/udp/8084/` or IPv6 like `/ip6/2001:db8::1234:5678/udp/8084` This address must be accessible from the internet. ::: ## Start your Full Node In the same directory, start your node by running: ```shell docker compose up -d ``` You can check the logs of the full node by running: ```shell docker compose logs -f ``` :::tip These commands assume you are using Docker Compose V2. In Docker Compose V1, the `docker compose` command uses a dash (`docker-compose`). If you use Docker Compose V1, replace the space in each `docker compose` command with a dash (`docker-compose`). For more information, see [Docker Compose V2](https://docs.docker.com/compose/#compose-v2-and-the-new-docker-compose-command). ::: ## Stop the Full Node Run the following command to stop the full node when you finish using it: ```shell docker compose stop ``` ## Check your Installation ### Check your Full Node Configuration By default, the configuration file `fullnode.yaml` is located at `/data/config/`. If you changed your local configuration you can always go back to the default [Full Node Configuration](https://docs.iota.org/operator/full-node/configuration). ### Try reaching your Full Node on the RPC endpoint import {Networks} from '@site/src/components/constant' Your full node serves the read endpoints of the IOTA JSON-RPC API at: {Networks.iota_localnet.rpc.json.core}. Register a new iota client environment with your node's RPC endpoint ```shell iota client new-env --alias=local --rpc http://127.0.0.1:9000 iota client switch --env local ``` :::info The instructions for building/installing the iota client are [here](../../developer/references/cli) ::: Then query the chain identifier from the RPC endpoint: ```shell iota client chain-identifier ``` --- ## Monitoring After a successful setup of a node, the next important step is to set up monitoring for the node. Monitoring is important to keep track of the node's health and performance. ## JSON-RPC Endpoint ### Test JSON RPC Interface After the full node starts, you can test the [JSON-RPC interfaces](https://docs.iota.org/iota-api-ref). ### View Activity on Your Local Full Node with IOTA Explorer The IOTA Explorer supports connecting to any network as long as it has `https` enabled. To view activity on your local full node: - open the URL: [https://explorer.iota.org/](https://explorer.iota.org/) - then select `Custom RPC URL` from the network dropdown in the top right. ### Fetch latest checkpoint using JSON-RPC ```shell curl --json '{"jsonrpc":"2.0","id":1,"method":"iota_getLatestCheckpointSequenceNumber","params":[]}' localhost:9000 -s | jq .result ``` To ensure node health, you can check that this value matches the latest checkpoint known by the rest of the network by using the value found on https://explorer.iota.org/ ## Node Health Metrics IOTA nodes expose a wide range of metrics to be scraped by [Prometheus](https://prometheus.io/). By default, metrics are available at the `http://localhost:9184/metrics` endpoint. The best way to visualize these metrics is to use [Grafana](https://grafana.com/). Additionally, a common approach is to use [node exporter](https://github.com/prometheus/node_exporter) to scrape performance metrics from the node and push them to Prometheus. ### Fetch key health metrics Key health metrics via the `/metrics` HTTP endpoint: ```sh curl -s localhost:9184/metrics | grep -E "^last_executed_checkpoint|^highest_synced_checkpoint|^highest_known_checkpoint|^last_committed_round|^consensus_threshold_clock_round|^highest_received_round|^consensus_proposed_blocks|^uptime" ``` For instance, for a validator node, the output would be: ```sh consensus_proposed_blocks{force="false"} 247 consensus_proposed_blocks{force="true"} 1 consensus_threshold_clock_round 257 highest_known_checkpoint 555 highest_synced_checkpoint 886 last_executed_checkpoint 890 last_executed_checkpoint_age_bucket{le="0.001"} 0 last_executed_checkpoint_age_bucket{le="0.005"} 0 last_executed_checkpoint_age_bucket{le="0.01"} 0 ... last_executed_checkpoint_age_bucket{le="60"} 891 last_executed_checkpoint_age_bucket{le="90"} 891 last_executed_checkpoint_age_bucket{le="+Inf"} 891 last_executed_checkpoint_age_sum 156.52341099999992 last_executed_checkpoint_age_count 891 last_executed_checkpoint_timestamp_ms 1748335503888 uptime{chain_identifier="b5d7e5c8",is_docker="false",os_version="macOS 15.5 Sequoia",process="validator",version="1.1.0"} 196 ``` ### Ensure node health using last checkpoint timestamp To make sure your node runs properly, we check that the last processed checkpoint is recent enough: - 10 seconds is typical - 30 seconds is still fine - You want to check that the timestamp difference stays under 1 minute You can check that from the previous metric `last_executed_checkpoint_timestamp_ms`, and compare timestamps with now using this command: ```shell last_executed_checkpoint_timestamp_ms="$(curl -s localhost:9184/metrics | grep ^last_executed_checkpoint_timestamp_ms | awk '{print $2}')" now_timestamp="$(date +%s%3N)" if (( now_timestamp - last_executed_checkpoint_timestamp_ms < 60000 )); then echo "[OK] healthy & in sync" else echo "[ERROR] Node unhealthy. Last known checkpoint is too old." fi ``` ### Monitor consensus sync status To ensure your node's consensus module is properly synced with the network, monitor the difference between `consensus_commit_sync_local_index` and `consensus_commit_sync_quorum_index`: ```shell metrics="$(curl -s localhost:9184/metrics)" local_index="$(echo "$metrics" | grep ^consensus_commit_sync_local_index | awk '{print $2}')" quorum_index="$(echo "$metrics" | grep ^consensus_commit_sync_quorum_index | awk '{print $2}')" difference=$((quorum_index - local_index)) if (( difference > 100 )); then echo "[WARNING] Consensus module not in sync. Difference: $difference" echo "[INFO] Monitor this difference over time:" echo " - If growing: Node falling behind network" echo " - If shrinking: Node catching up to network" else echo "[OK] Consensus module in sync. Difference: $difference" fi ``` **Key indicators:** - **Difference > 100**: Node's consensus module is not in sync - **Growing difference**: Network is advancing faster than node is syncing - **Shrinking difference**: Node is correctly syncing and catching up ### Monitor skipped proposals Check the `consensus_core_skipped_proposals` metric to detect issues with block production: ```shell metrics="$(curl -s curl -s localhost:9184/metrics)" skipped_proposals="$(echo "$metrics" | grep ^consensus_core_skipped_proposals)" if [[ -z "$skipped_proposals" ]]; then echo "[OK] No skipped proposals metric found" else echo "Skipped proposals by reason:" echo "$skipped_proposals" echo "" echo "Monitor these values over time. Growing counts indicate issues:" echo " - no_quorum_subscriber: Insufficient connections (< 2f+1 stake) to produce blocks" echo " - Other reasons: Check the specific label for the cause" fi ``` **Key indicators:** - **Growing `no_quorum_subscriber`**: Insufficient validator connections (less than 2f+1 stake connected). Node cannot safely produce new blocks - **Growing other labels**: The corresponding label value indicates the specific cause of skipped proposals - Run the script periodically to compare values and identify trends ### Overall node health When all the above healthchecks pass, your node should be considered healthy: - ✅ Last checkpoint timestamp is recent (< 60 seconds) - ✅ Consensus sync difference is acceptable (< 100) - ✅ Validator connections are sufficient (≥ 80% of committee) - ✅ Connection symmetry is maintained ## Logs #### Configuring Logs Log level (error, warn, info, trace) is controlled using the `RUST_LOG` environment variable. The `RUST_LOG_JSON=1` environment variable can optionally be set to enable logging in JSON structured format. Depending on your deployment method, these are configured in the following places: ``` [Service] ... Environment=RUST_BACKTRACE=1 Environment=RUST_LOG=info,iota_core=debug,consensus=debug,jsonrpsee=error ``` Add the following to the node container settings: ```yaml environment: - RUST_BACKTRACE=1 - RUST_LOG=info,iota_core=debug,consensus=debug,jsonrpsee=error ``` It is possible to change the logging configuration while a node is running using the admin interface. #### Verify Configured Logging Values To view the currently configured logging values: ```sh curl -w "\n" localhost:1337/logging ``` To change the currently configured logging values: ```sh curl localhost:1337/logging -d "info" ``` Note that the admin port (`1337`) is only exposed to `localhost` by default, so the commands must be executed inside the container. To view the currently configured logging values: ```sh docker exec curl -w "\n" localhost:1337/logging ``` To change the currently configured logging values: ```sh docker exec curl localhost:1337/logging -d "info" ``` Replace `` with your actual container name, such as `iota-fullnode-docker-setup-fullnode-1`. #### Viewing Logs To view and follow the IOTA node logs: ```sh journalctl -u iota-node -f ``` To search for a particular match: ```sh $ journalctl -u iota-node -g ``` View and follow: ```sh sudo docker compose logs -f [node_container_name] ``` By default, all logs are output. Limit this using `--since`: ```sh sudo docker logs --since 10m -f [node_container_name] ``` ## Monitoring Services Implementing monitoring services is essential to ensure the reliability, security, and performance of the blockchain network by providing real-time insights, detecting anomalies, enabling proactive issue resolution, and receiving automatic alerts. **Prometheus and Grafana** (recommended) Example pre-made dashboards you can use: - Community-owned grafana dashboard: https://github.com/stakeme-team/grafana-iota. - Grafana setup for the [local private network](https://github.com/iotaledger/iota/tree/develop/dev-tools/grafana-local), which might be a good example of how to build your own setup. - Officially supported dashboards can be found [here](https://github.com/iotaledger/iota/tree/develop/setups/grafana) in the IOTA repository. **Dolphin** Dolphin is a CLI tool that provides high-level features for validator and fullnode monitoring. Under the hood, it uses the IOTA node Prometheus metric exporter to check the health of the node. More info: https://gitlab.com/blockscope-net/dolphin-v2. --- ## Full Node Overview :::warning Advanced These instructions are for setting up a full node for network participation. If you just need a local development environment, you should instead follow the instructions in [Create a Local IOTA Network](/developer/getting-started/local-network) to create a local full node, validators, and faucet. ::: ## Introduction IOTA full nodes validate blockchain activities such as transactions, checkpoints, and epoch changes. They store and serve queries for the blockchain's state and history based on their configuration. While they cannot commit transactions like validator nodes, full nodes ensure the chain's integrity by re-executing transactions validated by a quorum of validators. ### Features - Track and verify the state of the blockchain, independently and locally. - Serve read requests from clients. - State Synchronization: - keep up with new transactions / checkpoints, - these transactions / checkpoints are processed and finalized by validators, but can be relayed to the network by the full nodes - verify these new transactions - apply the new transactions' state changes to update the local database ## Running IOTA Full Node ### Setup & Configuration You can set up your IOTA full node [using Docker](docker.mdx) or [build it from source](source.mdx). ### Monitoring & Troubleshooting Operators can monitor their node operation as explained in [Monitoring](./monitoring.mdx). --- ## Build From Source import {Networks} from '@site/src/components/constant' :::warning Advanced These instructions are for setting up a full node for network participation. If you just need a local development environment, you should instead follow the instructions in [Create a Local IOTA Network](/developer/getting-started/local-network) to create a local full node, validators, and faucet. ::: ## Hardware requirements - **RAM:** 64 GB - **CPU:** 8 physical cores / 16 vCPUs - **Storage:** 512 GB NVMe SSD (or more depending on pruning settings) - **Network Uplink:** 1 Gbps - **RAM:** 32 GB - **CPU:** 4 physical cores / 8 vCPUs - **Storage:** 512 GB NVMe SSD (or more depending on pruning settings) - **Network Uplink:** 1 Gbps ## 1. Software requirements - Linux (recommended): - For this installer to work, use a Linux distribution with `systemd` and `apt-get`. - IOTA recommends and supports Ubuntu or Debian. - Building from source also works on macOS (but systemd-related steps won't). - Rust/cargo stable (recent version, check [rust-toolchain.toml](https://github.com/iotaledger/iota/blob/develop/rust-toolchain.toml) for the specific version) . If needed, [update Rust](https://www.rust-lang.org/tools/install). :::info Default Firewall Ports Make sure to open these ports in your firewall configuration. (These are the defaults, but may vary for a custom setup) | Protocol/Port | Reachability | Purpose | Description | | ------------- | ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------- | | TCP/8080 | Outbound | Validator/transaction interface | full nodes connect to validators to collect partial signatures for transactions | | UDP/8084 | Inbound/Outbound | Peer-to-peer state sync interface | nodes sync their state with each other | | TCP/9000 | Inbound | JSON RPC endpoint | API for interaction with the network | | TCP/9184 | Localhost | Prometheus metrics scraping | for local monitoring only | ::: ## 2. Run the installer script In any directory, run: ```shell NETWORK="testnet" bash -i <(curl -sSf https://raw.githubusercontent.com/iotaledger/iota/refs/heads/develop/setups/fullnode/systemd/setup.sh) ``` :::info Set the network environment variable to the network you want to connect to ("testnet", "mainnet", or "devnet") ::: The installer will perform the following steps: 1. Check your rust toolchain version 2. Install system packages (libraries & other dependencies): `tzdata`, `libprotobuf-dev`, `ca-certificates`, `build-essential`, `libssl-dev`, `libclang-dev`, `pkg-config`, `openssl`, `protobuf-compiler`, `git`, `clang`, `cmake` 3. Clone the [iota repo](https://github.com/iotaledger/iota), set to the right branch for the [network](../../developer/getting-started/connect.mdx#iota-networks) you chose 4. Build the iota-node binary The built binary is located at `/opt/iota/bin/iota-node` by default. 5. Create a user called iota, make it own directories for service binary, config, data 6. Create a node config file, download genesis/migration blobs The config and blob files are located at `/opt/iota/config` by default. 6. Create a systemd service unit file The service definition file is located at `/etc/systemd/system/iota-node.service`. 7. Start the service ## 3. Check your installation Your IOTA node should now be running and connected to the network you chose. Check that the iota-node service is active and running: ```shell sudo systemctl status iota-node ``` Check (and follow) the logs: ```shell journalctl -u iota-node -f ``` If you need to stop the service, you can run `sudo systemctl stop iota-node` Your IOTA node should now be running and connected to the network you chose. ### Check your node configuration By default, the configuration file `fullnode.yaml` is located at `/opt/iota/config/fullnode.yaml`. If you changed your local configuration you can always go back to the default [Full Node Configuration](https://docs.iota.org/operator/full-node/configuration). #### Configure P2P Settings Configure the P2P settings in your `fullnode.yaml` file by following these steps: 1. Update the external address: ```yaml p2p-config: external-address: /dns//udp/8084 ``` :::note Replace `` with your public hostname. In case you don't have a public hostname, you can alternatively use a public IPv4 like `/ip4/1.2.3.4/udp/8084/` or IPv6 like `/ip6/2001:db8::1234:5678/udp/8084` This address must be accessible from the internet. ::: ### Reach your node via the RPC endpoint import {Networks} from '@site/src/components/constant' Your full node serves the read endpoints of the IOTA JSON-RPC API at: {Networks.iota_localnet.rpc.json.core}. Register a new iota client environment with your node's RPC endpoint ```shell iota client new-env --alias=local --rpc http://127.0.0.1:9000 iota client switch --env local ``` :::info The instructions for building/installing the iota client are [here](../../developer/references/cli) ::: Then query the chain identifier from the RPC endpoint: ```shell iota client chain-identifier ``` ### Troubleshooting If you run into issues during the setup, or while checking your install, be sure to check common issues in the [troubleshooting section](./troubleshooting.mdx) ## 4. Updating Your Full Node When IOTA releases a new version, you must update your full node with the release to ensure compatibility with the network it connects to. Make sure your node stays on the same network (testnet, mainnet, devnet, ...) it was previously connected to. To upgrade your node, run the same script you used to install it: ```shell NETWORK="testnet" bash -i <(curl -sSf https://raw.githubusercontent.com/iotaledger/iota/refs/heads/develop/setups/fullnode/systemd/setup.sh) ``` :::info Re-use the same network (`testnet`, `mainnet`, `devnet`) your node is already running on. ::: Your full node restarts on: {Networks.iota_localnet.rpc.json.core}. :::info If your full node is configured to prune transactions, Archival nodes can help ensure lagging peer nodes don't lose any information. For more information, see [IOTA Archives](../common/archives.mdx). ::: ## 5. Customize your node's configuration :::warning Node logs can take a lot of space quickly. Make sure to configure your `iota-node` systemd service to rotate or prune logs to not run out of space. See [systemd docs](https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html#SystemMaxUse=) You can also lower the log level by editing your service definition file (changing RUST_LOG in `/etc/systemd/system/iota-node.service`), in order to produce less logs. ::: In the [next section](./configuration.mdx), we'll see which options are available to tune the node's configuration to your preferences. --- ## Troubleshooting If you encounter errors or your full node stops working, run the commands in the following section to resolve the issue. ## Issues during setup ### `Error: cannot find -lpq` This error should only occur when building indexer-related binaries (`iota-indexer` or `iota-localnet` with the `indexer` feature), which require the `libpq` library. It is not needed for building `iota-node`. Use `sudo apt-get install libpq-dev` to install on Linux, or `brew install libpq` on macOS. After you install on macOS, create a Homebrew link using `brew link --force libpq`. For further context, reference the [issue on Stack Overflow](https://stackoverflow.com/questions/70313347/ld-library-not-found-for-lpq-when-build-rust-in-macos?rq=1). ### `Error: Listener: Address already in use` If you receive the following error: ```shell panicked at error binding to 0.0.0.0:9184: error creating server listener: Address already in use (os error 98) ``` Update the metrics address in your `fullnode.yaml` file to use port `9180`. ```yaml metrics-address: "0.0.0.0:9180" ``` No known issues with the docker setup yet. ## Inspecting the state of a running node Looking at the logs for specific errors: ```shell journalctl -u iota-node -f ``` Looking at the logs for specific errors: ```shell docker compose logs iota-node -f ``` Connecting to a shell inside the container: ```shell docker exec -it $CONTAINER_ID /bin/bash ``` Inspecting the database: ```shell ls -la data/db/ ``` ## Reinstalling the node from scratch If you continue to see issues, do the following steps: To re-install your node from scratch, run the same script you used to install it: ```shell NETWORK="testnet" bash -i <(curl -sSf https://raw.githubusercontent.com/iotaledger/iota/refs/heads/develop/setups/fullnode/systemd/setup.sh) ``` :::info Re-use the same network (`testnet`, `mainnet`, `devnet`) your node is already running on. ::: Stop the full node ```shell docker compose down ``` Delete the data folder Make sure to first make a backup of your config files ```shell rm -Rf data/ ``` Pull the latest version of the container ```shell docker compose pull ``` Run the prepare script again ```shell ./prepare.sh ``` Start the container again ```shell docker compose up ``` ## Local RPC Connectivity Issues Update the `json-rpc-address` in the full node config to listen on all addresses: ```shell sed -i 's/127.0.0.1/0.0.0.0/' fullnode.yaml ``` ```shell -json-rpc-address: "127.0.0.1:9000" +json-rpc-address: "0.0.0.0:9000" ``` --- ## API Reference ## Overview The **Gas Station API** provides a simple and secure interface to manage gas reservations and execute transactions on the IOTA network. This API is exposed via HTTP and uses **Bearer Token Authentication** for secured access. Below are the available endpoints, request and response formats, and example payloads. --- ## Authentication All API endpoints require a **Bearer Token** for authorization. Include the token in the `Authorization` header of your requests. ``` Authorization: Bearer YOUR_TOKEN_HERE ``` --- ## Endpoints ### Health Check #### `GET /` Performs a basic health check of the Gas Station server. **Response:** ```json "OK" ``` --- ### Version #### `GET /version` Retrieves the current version of the Gas Station server. **Response:** ```json "1.0.0-abcdef123456" ``` --- ### Debug Health Check #### `POST /debug_health_check` Performs an in-depth health check, accessible only with valid authorization. **Headers:** ``` Authorization: Bearer YOUR_TOKEN_HERE ``` **Response:** ```json "OK" ``` Or, if unauthorized: ```json "Unauthorized" ``` --- ### Reserve Gas #### `POST /v1/reserve_gas` Reserves gas objects for a specified budget and duration. **Headers:** ``` Authorization: Bearer YOUR_TOKEN_HERE ``` **Request Body:** ```json { "gas_budget": 1000000, "reserve_duration_secs": 600 } ``` **Response (Success):** ```json { "result": { "sponsor_address": "0xc408cb821190caaa4d0e3a66d47f4d3ba4a5de8a3a46049f71192d43ccb3a70c", "reservation_id": 4, "gas_coins": [ { "objectId": "0x01dc25a229a0d68699525809d8efa303eb873195c0e0fd7bb06fbd0eaf8efdd4", "version": 3, "digest": "CVr5PFWNoi5i2SmwetXoxbUssgX23i3Lf2xciB7ybrY5" }, { "objectId": "0x01e80806f756c3f49d6a7c39277a9ed6dbe8d7090703b69a1bb6433514c67c24", "version": 3, "digest": "AsMpLULuJmvkSKBYqjqQ5f163356Gva9D38FPPrKrZTr" }, { "objectId": "0x01e903e2f24e6d1fa035194d513168bb0fd18d11139210174319b31d0e704f64", "version": 3, "digest": "7m8wctf4LoWJ8AwM3AyHVr1NRPF1fpmicNWd5DAcKrjq" }, { "objectId": "0x01f7d07ee4c0370ae188c1c7ddd8367c6400efd5612697140e13338b48ff7c6c", "version": 3, "digest": "xA77CCa7FU2ecapFuA3skUdgKGuksMFwQxi9QY1msV4" }, { "objectId": "0x020420418e31e1c3e14b8ab6b6eff8398f661a625005b9b14c37006c5568b606", "version": 3, "digest": "7rMvXzq8PLvmKufpz4PKBkd4y7jdiZ4SP6uUrpULdf8c" } ] } } ``` **Response (Error):** ```json { "error": "Invalid authorization token" } ``` **Validation Rules:** * `gas_budget` must be positive and less than `2_000_000_000`. * `reserve_duration_secs` must be between 1 and 600 seconds. --- ### Execute Transaction #### `POST /v1/execute_tx` Executes a transaction using reserved gas objects. **Headers:** Authorization: Bearer YOUR_TOKEN_HERE ``` **Request Body:** ```json { "reservation_id": 12345, "tx_bytes": "BASE64_ENCODED_TRANSACTION", "user_sig": "BASE64_ENCODED_SIGNATURE" } ``` **Response (Success):** ```json { "effects": { "messageVersion": "v1", "status": { "status": "success" }, "executedEpoch": "4720", "gasUsed": { "computationCost": "1000000", "computationCostBurned": "1000000", "storageCost": "6368800", "storageRebate": "4902000", "nonRefundableStorageFee": "0" }, "modifiedAtVersions": [ { "objectId": "0x01dc25a229a0d68699525809d8efa303eb873195c0e0fd7bb06fbd0eaf8efdd4", "sequenceNumber": "3" }, { "objectId": "0x01e80806f756c3f49d6a7c39277a9ed6dbe8d7090703b69a1bb6433514c67c24", "sequenceNumber": "3" }, { "objectId": "0x01e903e2f24e6d1fa035194d513168bb0fd18d11139210174319b31d0e704f64", "sequenceNumber": "3" }, { "objectId": "0x01f7d07ee4c0370ae188c1c7ddd8367c6400efd5612697140e13338b48ff7c6c", "sequenceNumber": "3" }, { "objectId": "0x020420418e31e1c3e14b8ab6b6eff8398f661a625005b9b14c37006c5568b606", "sequenceNumber": "3" } ], "sharedObjects": [ { "objectId": "0x0000000000000000000000000000000000000000000000000000000000000006", "version": 4347636, "digest": "J7BwPQsd5KHGeRPkPhNNiPTVJ2KGQEcfgWPi39v9LnvN" } ], "transactionDigest": "87zDiKXukrjTvBmhDENSnb2QrJTDhHpLAobAYAFJWvfw", "created": [ { "owner": { "Shared": { "initial_shared_version": 4347637 } }, "reference": { "objectId": "0x76417e6fec0a9c06a2f7872fc52d97cc199ef1cea71e37c7f61a73de81e81bb1", "version": 4347637, "digest": "BQWkcCLoCA6adKqnniPAfrKZdf9yErAehzKHf7nPW2xM" } }, { "owner": { "AddressOwner": "0x443b6425a154a0681ba29e06e4f59bca5a6dbf14a753e9ebe06eabc2ca3cabe9" }, "reference": { "objectId": "0x8a63067b819265a873fc318b169ebeace5e832b49894eca7b9d4f25743e0df41", "version": 4347637, "digest": "3ddyifb2AJurGJpo8TqrkpMMZSauwFVFzT3n7y6jZ2Nx" } } ], "mutated": [ { "owner": { "AddressOwner": "0xc408cb821190caaa4d0e3a66d47f4d3ba4a5de8a3a46049f71192d43ccb3a70c" }, "reference": { "objectId": "0x01dc25a229a0d68699525809d8efa303eb873195c0e0fd7bb06fbd0eaf8efdd4", "version": 4347637, "digest": "Dc1BchUUFZpDSHw36tdoQaYXqdT8R89hyPigNvMPouR3" } } ], "deleted": [ { "objectId": "0x01e80806f756c3f49d6a7c39277a9ed6dbe8d7090703b69a1bb6433514c67c24", "version": 4347637, "digest": "7gyGAp71YXQRoxmFBaHxofQXAipvgHyBKPyxmdSJxyvz" }, { "objectId": "0x01e903e2f24e6d1fa035194d513168bb0fd18d11139210174319b31d0e704f64", "version": 4347637, "digest": "7gyGAp71YXQRoxmFBaHxofQXAipvgHyBKPyxmdSJxyvz" }, { "objectId": "0x01f7d07ee4c0370ae188c1c7ddd8367c6400efd5612697140e13338b48ff7c6c", "version": 4347637, "digest": "7gyGAp71YXQRoxmFBaHxofQXAipvgHyBKPyxmdSJxyvz" }, { "objectId": "0x020420418e31e1c3e14b8ab6b6eff8398f661a625005b9b14c37006c5568b606", "version": 4347637, "digest": "7gyGAp71YXQRoxmFBaHxofQXAipvgHyBKPyxmdSJxyvz" } ], "gasObject": { "owner": { "AddressOwner": "0xc408cb821190caaa4d0e3a66d47f4d3ba4a5de8a3a46049f71192d43ccb3a70c" }, "reference": { "objectId": "0x01dc25a229a0d68699525809d8efa303eb873195c0e0fd7bb06fbd0eaf8efdd4", "version": 4347637, "digest": "Dc1BchUUFZpDSHw36tdoQaYXqdT8R89hyPigNvMPouR3" } }, "eventsDigest": "Au699PVCMgm7qA7xaYC3buv6qzCvTWZZjSxak53ZXgw6", "dependencies": [ "86Wh3jbx31jYYnvuRVMMxvFeV1fKAGt81GN6ZzRcPakr", "AjGJMbRPtJFPr84y1qqHy9DCCFwFaW3aEHjhKshKTJqn", "BjH9ntnSzydCYmYijXJh2nenbeVRhSiyJqJt4bzG7o5u", "FrfyHw9f1jjPzYaHcnNtTrM4iSycGKDzdQvXBdfHsoXd", "HrVH2REqTq5ZSm1JeW9APBgHomu7jEAKmpd9AkuUHQWE" ] }, "error": null } ``` The for information about the fields in the `effects` property, you can check the documentation of its internal type: [`IotaTransactionBlockEffects`](https://iotaledger.github.io/iota/iota_json_rpc_types/enum.IotaTransactionBlockEffects.html). **Response (Error):** ```json { "error": "Invalid transaction signature" } ``` **Validation Rules:** * The request body may also contain a `requestType` field, which is optional (defaults to `waitForEffectsCert`) and controls how transaction results are awaited in the gas station before returning a response. For details see [iota api ref](https://docs.iota.org/iota-api-ref#iota_executetransactionblock). Allowed values are: - waitForEffectsCert - waitForLocalExecution --- ### Reload Access Controller #### `GET /v1/reload_access_controller` Reloads the access controller configuration. **Headers:** Authorization: Bearer YOUR_TOKEN_HERE **Response:** ```json { "result": "success", } ``` **Response (Error):** ```json { "error": "Unauthorized" } ``` --- ## Error Codes | **Status Code** | **Description** | | --------------- | ----------------------------- | | 200 | Request succeeded | | 400 | Bad request (invalid payload) | | 401 | Unauthorized (invalid token) | | 403 | Forbidden (access denied) | | 500 | Internal server error | --- --- ## Architecture Overview The transition from Web2 to Web3 presents a significant challenge—how can applications provide seamless and user-friendly interactions while maintaining the security and decentralized principles of blockchain technology? This is where the **IOTA Gas Station** comes into play. The IOTA Gas Station is a mechanism that enables **sponsored transactions**, allowing users to interact with the IOTA network without directly managing transaction fees. This is particularly valuable for developers building decentralized applications (dApps) where the user experience must remain simple and intuitive. ## The Problem: Transaction Fees in Web3 Unlike traditional Web2 applications, where users do not need to worry about transaction costs, Web3 applications require users to pay network fees to validate and store transactions. However, expecting users to manage and maintain IOTA tokens solely for transaction fees introduces friction and discourages adoption. The **IOTA Gas Station** addresses this challenge by acting as an intermediary between the user and the network. It allows developers to cover transaction costs on behalf of users, making interactions frictionless while preserving the fundamental aspects of security and authentication in Web3. --- ## User Flow ### Standard User Flow (Without Gas Station) In a typical Web3 interaction, a user wishing to execute a transaction must first acquire IOTA tokens. This means they need to set up a wallet, fund it with IOTA tokens, and ensure they have enough balance to cover the transaction fees. When they prepare a transaction, the necessary fee is deducted from their wallet, and the transaction is submitted to the network for validation and processing. Once confirmed, the transaction is complete, and the updated balance reflects the fee deduction. While this process ensures security and decentralization, it also places a burden on users who may not be familiar with acquiring and managing tokens. The need to maintain a balance introduces friction, potentially discouraging mainstream adoption. ```mermaid sequenceDiagram actor User participant IOTA_Network User->>User: Prepare transaction with fees User->>IOTA_Network: Submit signed transaction IOTA_Network->>User: Validate and process transaction IOTA_Network->>User: Transaction complete ``` ### Enhanced User Flow with Gas Station With the introduction of the **IOTA Gas Station**, the transaction process becomes significantly more user-friendly. Instead of requiring users to hold IOTA tokens, the Gas Station acts as an intermediary that sponsors the transaction fees. The user still prepares their transaction as before, but instead of supplying their own gas fees, they request a **gas object** from the Gas Station. Once received, they sign the transaction, incorporating the provided gas object. The signed transaction is then sent back to the Gas Station, which finalizes it by adding its own signature and submitting it to the network. By abstracting the complexity of fee management, the Gas Station enables a **fee-less experience** for users while preserving the security and authentication guarantees of Web3. This model allows developers to integrate blockchain transactions into their applications without requiring users to engage with the underlying tokenomics. ```mermaid sequenceDiagram actor User participant Gas_Station participant IOTA_Network User->>Gas_Station: Request gas objects Gas_Station->>User: Provide gas objects User->>User: Prepare transaction with provided gas objects User->>Gas_Station: Send signed transaction Gas_Station->>IOTA_Network: Sign and submit transaction IOTA_Network->>Gas_Station: Validate and process transaction Gas_Station->>User: Transaction complete ``` --- --- ## Gas Station Components The **IOTA Gas Station** consists of several key components, each playing a crucial role in managing transactions efficiently and securely. ```mermaid flowchart LR subgraph subGraph0["Gas Station Pool"] GSS["Gas Station Servers"] end KMS[("Key Management Store")] -.-> GSS GSS --> REDIS[("Persistent Storage (Redis)")] style KMS stroke-width:1px,stroke-dasharray: 5 ``` ## Gas Station Server The **Gas Station Server** is the central component of the system. Each deployment of a Gas Station Server is associated with a single sponsor wallet, responsible for funding transactions. To scale operations, multiple Gas Station Servers can be deployed together as a **Gas Station Pool**, allowing them to share the same sponsor address and persistent storage. ### HTTP Endpoints: The server provides two main endpoints to facilitate transaction management: * [`/reserve_gas`](../api-reference/api-reference.mdx#reserve-gas) – Reserves gas objects that meet the specified amount required for a transaction. * [`/execute_tx`](../api-reference/api-reference.mdx#execute-transaction)** – Executes a transaction using the gas objects previously fetched from `/reserve_gas`. ### Transaction Lifecycle The lifecycle of a transaction within the Gas Station follows a structured flow. The process begins when the **user**, who does not hold IOTA tokens for fees, initiates an interaction with the Gas Station by making a request to the `/reserve_gas` endpoint. This request specifies the amount of gas required for the upcoming transaction. The Gas Station processes this request and allocates the necessary gas objects from its internal pool, returning them to the user. With the gas objects in hand, the user then prepares their transaction, embedding the allocated gas within it. Once the transaction is properly constructed, the user signs it, ensuring its authenticity and integrity. The signed transaction is then submitted to the Gas Station through the `/execute_tx` endpoint. At this stage, the Gas Station takes responsibility for finalizing the transaction. It verifies the validity of the submitted transaction, attaches its own signature, and submits the transaction to the IOTA network for processing. The network validates and executes the transaction, recording it permanently on the blockchain. Once the transaction is confirmed, the Gas Station relays the execution results back to the user, signaling the successful completion of the process. ```mermaid sequenceDiagram actor User as Transaction Creator box Gas Station Server participant ReserveGas as /reserve_gas participant ExecuteTx as /execute_tx end User->>ReserveGas: Request gas activate ReserveGas ReserveGas->>User: Provide gas objects deactivate ReserveGas activate User User->>User: Prepare transaction & sign it deactivate User User->>ExecuteTx: Submit signed transaction ``` ## Gas Station Storage The **Gas Station Storage** serves as the persistent database for managing gas objects and transaction states. The storage layer, powered by Redis, keeps track of all available gas coins, ensuring that transactions can be sponsored efficiently. Redis together with LUA scripts provide a built-in mechanism that supports concurrent access from multiple Gas Station Servers, allowing for scalable and high-performance operations. This ensures that multiple servers in a Gas Station Pool can efficiently share and allocate gas resources without conflicts. It is essential for ensuring continuity, especially when gas pools are depleted and must be replenished. To maintain consistency with the network, the storage state is periodically synchronized with the configured interval. This interval is defined in `config.yaml` under `refresh-interval-sec`, ensuring that the system regularly verifies and updates its stored state based on the actual conditions of the IOTA network. **Gas Station Storage usage process:** The Gas Station continuously synchronizes its storage with the network to ensure accurate gas coin tracking. It periodically fetches available gas coins from the IOTA Network Node, splits them into smaller denominations, and updates Redis with these entries. When a user requests gas, the Gas Station retrieves the required coins from Redis, assigns a reservation ID, and marks them as in use. Once the transaction is executed, the reservation ID is removed, and the list of available gas coins is updated accordingly. This process ensures that only valid gas coins are used while preventing inconsistencies between the Gas Station and the network. ```mermaid sequenceDiagram participant Network_Node as IOTA Network Node participant Gas_Station as Gas Station Server participant Redis_Storage as Redis Storage Gas_Station->>Network_Node: Fetch list of available gas coins Network_Node-->>Gas_Station: Return list of gas coins activate Gas_Station Gas_Station->>Gas_Station: Split gas coins into smaller denominations Gas_Station->>Redis_Storage: Store list of gas coins deactivate Gas_Station Note over Redis_Storage: Redis maintains available gas coins Gas_Station->>Redis_Storage: Reserve gas coins and generate reservation ID Redis_Storage-->>Gas_Station: Return reserved gas coins with reservation ID Note over Gas_Station: Sponsored transaction accepted Gas_Station->>Network_Node: Submit transaction to network Network_Node-->>Gas_Station: Transaction confirmed Gas_Station->>Redis_Storage: Remove reservation ID and update available coins Redis_Storage-->>Gas_Station: Storage updated ``` :::note **Important Considerations:** * A single Redis instance **should not** be used for Gas Station Servers that operate with different sponsor wallets. Doing so could lead to conflicts and incorrect transaction processing. * When switching between networks (e.g., from a testnet to a mainnet), all Redis entries should be cleared to prevent the Gas Station from returning outdated or non-existent objects. This can be done manually by flushing the Redis database before reinitializing the Gas Station instance. The storage layer, powered by Redis, keeps track of all available gas coins, ensuring that transactions can be sponsored efficiently. ::: ## Key Store Manager The **Key Store Manager (KSM)** provides secure key management for signing transactions. It offers two configurations: * **External Key Management Services (KMS)** – Uses external providers like AWS KMS for enhanced security. * **In-memory key storage** – Maintains private keys within the Gas Station instance for faster but less secure operations. **External Key Store** is an optional component that facilitates secure transaction signing by integrating an **external Key Management System (KMS)**. This feature simplifies key management, particularly when deploying multiple Gas Station Servers as a pool that shares the same signing key across multiple instances. ### External KMS Integration The Gas Station assumes that the external KMS will communicate using a **JSON API**. The KMS must expose the following two endpoints: * **GET `/get-pubkey-address`** This endpoint returns the public key address corresponding to the stored private key encoded in base64 * **POST `/sign-transaction`** This endpoint is responsible for signing transactions. The request payload is a JSON object containing the base64-encoded and BCS-serialized transaction data ```mermaid sequenceDiagram participant Network as IOTA Network Node participant Gas_Station as Gas Station Server participant KMS as Key Management System (KMS) Note right of Gas_Station: On startup Gas_Station->>KMS: Request public key address (/get-pubkey-address) KMS-->>Gas_Station: Return public key address loop For each transaction Gas_Station->>KMS: Request signature (/sign-transaction) KMS-->>Gas_Station: Return signed transaction Gas_Station->>Network: Submit signed transaction end ``` It is crucial that the serialized signature and keys adhere to the **IOTA transaction signature specification**. Developers should refer to the [IOTA documentation](https://docs.iota.org/developer/cryptography/transaction-auth/signatures#signature-requirements) for further details on signature requirements. For a reference implementation of a compatible KMS, developers can explore the sample implementation provided in the official repository: [IOTA Gas Station Sample KMS Sidecar](https://github.com/iotaledger/gas-station/tree/dev/sample_kms_sidecar). --- ## Gas Station Features ## Access Controller The **Access Controller** is a rule-based system that regulates access to the `/execute_tx` endpoint. Similar to firewalls, it enforces policies to determine which transactions are allowed or denied. ### Access Policy Modes The Access Controller operates in one of two modes: * **Deny All:** A strict security model where all transactions are blocked by default. Rules must be explicitly defined to allow specific transactions. * **Allow All:** A more permissive model where all transactions are allowed by default. Rules are used to selectively deny certain transactions. :::warning Important Note While the **`/reserve_gas`** endpoint remains unprotected by the ACL, potential abuse (such as multiple reservation attacks) is mitigated through configuration options like maximum reservation limits and expiration times. ::: ### Rule Processing The Access Controller processes rules sequentially, much like rule-based filtering systems such as firewalls. Each rule is evaluated in order, and once a matching rule is found, its action is applied, preventing further rules from being checked. This ensures predictable behavior where the most specific rules are placed earlier, while broader rules serve as fallbacks. When a transaction is submitted, the Access Controller starts by checking the first rule in the list. If the transaction matches the conditions set in that rule, the corresponding action is taken, either allowing or denying the transaction. If no match is found, the system proceeds to the next rule in sequence. This process continues until either a rule applies or no rules match, in which case the system falls back to the default policy (either **Allow All** or **Deny All**). #### Example For example, consider the following **Allow All** policy configuration with three rules: ```yaml # Rule 1 - sender-address: "0x0202020202020202020202020202020202020202020202020202020202020202" transaction-gas-budget: "<=500000" action: allow # Rule 2 - sender-address: "0x0101010101010101010101010101010101010101010101010101010101010101" transaction-gas-budget: "<=10000000" action: allow # Rule 3 - sender-address: "*" action: deny ``` Rule 1: Allow transactions from 0x0202... with a gas budget ≤ 500000\. Rule 2: Allow transactions from 0x0101... with a gas budget ≤ 1000000\. Rule 3: Deny all other transactions. #### Evaluation Example 1 A sender `0x0101...` submits a transaction with a `900000` gas budget. The system evaluates `Rule 1` first but skips it since the sender’s address does not match. `Rule 2` is then checked, and since both the sender and gas budget conditions are satisfied, the transaction is allowed. ```mermaid flowchart TD A["Start"] ==> B{"Match Rule 1?"} B == No ==> C{"Match Rule 2?"} C == Yes ==> D["Transaction Allowed"] C -. No .-> E{"Match Rule 3"} style A stroke-width:4px,stroke-dasharray: 0 style B stroke-width:4px,stroke-dasharray: 0 style C stroke:#00C853,stroke-width:4px,stroke-dasharray: 0 style D stroke:#00C853,stroke-width:4px,stroke-dasharray: 0 style E stroke-width:2px,stroke-dasharray: 2,stroke:#616161,fill:transparent ``` **Explanation:** | rule | explanation | |--------|-----------------------------------------------------------------------------| | Rule 1 | Checked first but skipped as the sender doesn’t match. | | Rule 2 | Evaluated, and since the sender and budget match, the transaction is **allowed**. | | Rule 3 | Never reached because Rule 2 applied. | #### Evaluation Example 2 A sender `0x0303...` submits a transaction with a `400000` gas budget. The system evaluates `Rule 1` and `Rule 2` but skips both as the sender’s address does not match. The final `Rule 3` is then evaluated, which applies to all remaining transactions, resulting in the transaction being denied. ```mermaid flowchart TD A["Start"] ==> B{"Match Rule 1?"} B == No ==> C{"Match Rule 2?"} C == No ==> E{"Match Rule 3?"} E == Yes ==> F["Transaction Denied"] E -. No .-> G["Policy: Accept All"] style A stroke-width:4px style B stroke-width:4px,stroke-dasharray: 0 style C stroke-width:4px,stroke-dasharray: 0 style E stroke-width:4px,stroke-dasharray: 0,stroke:#00C853 style F stroke:#00C853,stroke-width:4px,stroke-dasharray: 0 style G stroke-width:2px,stroke-dasharray: 2,fill:transparent ``` **Explanation:** | rule | explanation | |--------|------------------------------------------------------------------------| | Rule 1 | Checked but skipped. | | Rule 2 | Evaluated but skipped as well. | | Rule 3 | Applies to all other cases, so the transaction is **denied**. | --- ### Gas Usage Limiting One of the Access Controller's most useful capabilities is its built-in mechanism for tracking gas usage across all gas station instances. Redis-backed counters synchronize limits cluster-wide, making this possible. Let's explore this feature with a practical example. :::warning Gas Usage is a Matcher `gas-usage` is a matcher, not an action. It works with other matchers, and only when all matchers match is the rule's action applied. ::: #### Daily Gas Limit Per Account Suppose you want to ensure that no individual sender can consume more than 1,000,000 gas units daily. You can define this rule as follows: ```yaml access-controller: access-policy: deny-all rules: - sender-address: "*" gas-usage: value: "<1000000" window: 1day count-by: [ sender-address ] action: allow ``` Here's what happens: * The access policy is set to `deny-all`, meaning only explicitly allowed transactions are processed. * The rule matches any sender address. * The `gas-usage` condition specifies a 1-day rolling window and limits usage to below 1,000,000 gas units per address. * The `count-by: [ sender-address ]` clause ensures that each sender's gas usage is tracked and limited separately. ##### Why `count-by` is Important The `count-by` field is crucial for isolating gas usage limits per sender address. It ensures that each sender has an independent gas usage counter, preventing one user's activity from affecting another's access to the gas station. This is particularly important in scenarios where multiple users interact with the gas station simultaneously. If `count-by` were omitted, the gas usage would be aggregated across all senders, leading to potential abuse or unfair distribution of resources. This means that if one user consumes a significant portion of the gas limit, it does not impact other users' ability to access the gas station. Each sender's gas usage is tracked independently, allowing fair and equitable resource access. #### Combining Multiple Rules with Separate Counters You can define multiple rules with independent gas usage counters. Each rule is evaluated individually and maintains its gas usage state, enabling precise control over different conditions. ```yaml rules: - sender-address: "0x0202020202020202020202020202020202020202020202020202020202020202" move-call-package-address: "0x0101010101010101010101010101010101010101010101010101010101010101" gas-usage: value: "<1500000" window: 1day action: allow - sender-address: "*" gas-usage: value: "<1000000" window: 1day count-by: [ sender-address ] action: allow ``` Let’s walk through how the transaction processing works with this setup: Suppose an account with the address `0x0202...` initiates a transaction targeting the package `0x0101...`. If the transaction requires, for instance, 1,400,000 gas units, it will be accepted by the **first rule** because: * The sender's address matches exactly. * The package address matches. * The gas requirement is below the 1,500,000 limit. ```mermaid flowchart TD A["Transaction"] --> B{"Match Rule 1?"} B -- "Yes: sender + package + gas usage OK" --> C["Apply Rule 1: Allow"] style A stroke-width:3px style B stroke:#4CAF50,stroke-width:3px style C stroke:#4CAF50,stroke-width:3px ``` Now, imagine the same account initiates another transaction, this time requiring 1,000,000 gas units. The **first rule** does not apply because the `gas-limit` matcher would be above the 1,500,000. The transaction then proceeds to the **second rule**, which: * Matches any sender address (`"*"`). * Tracks gas usage individually using `count-by: [ sender-address ]`. * Confirms the daily usage is still under 1,000,000 for this sender. The second rule is applied in this case, and the transaction is accepted. ```mermaid flowchart TD A["Start Transaction"] --> B{"Match Rule 1?"} B -- "No: gas usage mismatch" --> D{"Match Rule 2?"} D -- "Yes: sender matched + gas usage OK" --> E["Apply Rule 2: Allow"] style A stroke-width:3px style B stroke:#2196F3,stroke-width:3px style D stroke:#4CAF50,stroke-width:3px style E stroke:#4CAF50,stroke-width:3px ``` Each rule with a gas-usage matcher defines its isolated gas usage counter. No gas usage data is shared between rules. Understanding this evaluation flow is critical when building multi-tiered access policies with fallback logic. #### Performance Considerations Every gas-usage rule results in two Redis operations per transaction, introducing potential I/O bottlenecks. To mitigate Redis pressure, evaluate the number and specificity of rules for high-throughput setups. Optimizations are planned for future releases. ### Rego Expressions From version `0.2`, the Access Controller in Gas Stations supports matching using the Rego language. This enables more complex filtering, especially when multiple nested transaction attributes must be inspected. #### Example Rego Predicate Let's extend the previous example with a Rego predicate that checks for a specific Move Call package address, function name, and argument. To do this, you can create a Rego script that defines a rule to match the Move Call transaction attributes. ```go package matchers default move_call_matches = false move_call_matches if { cmds := input.transaction_data.V1.kind.ProgrammableTransaction.commands count(cmds) == 1 mc := cmds[0].MoveCall mc["package"] == "0xb674e2ed79db3c25fa4c00d5c7d62a9c18089e1fc4c2de5b5ee8b2836a85ae26" mc.module == "allowed_module_name" mc.function == "allowed_function_name" argv_1 := input.transaction_data.V1.kind.ProgrammableTransaction.input.Pure bcs.decode_typed(argv_1, "string") == "allowed_argument_value" } ``` For more information on Rego syntax and capabilities, refer to the [Rego documentation](https://www.openpolicyagent.org/docs/policy-language/). Refer to [Gas Station Server repository](https://github.com/iotaledger/gas-station/blob/dev/docs/access-controller.md#rego-expression-sources) for detailed config syntax and more examples. #### Using Rego in Access Controller Rego can be used as the `rego-expression` matcher in the Access Controller. It allows you to define complex rules that evaluate transaction attributes, such as commands, package addresses, and function names. :::warning rego expressions are compiled at the start of the Gas Station Server. Therefore the changes in the source location (`file`, `redis`, or `http`) will not be applied until the server is restarted or the [`/v1/reload_access_controller`](../api-reference/api-reference.mdx#reload-access-controller) API endpoint is called. ::: ```yaml rules: - sender-address: "0x0202020202020202020202020202020202020202020202020202020202020202" rego-expression: location-type: redis url: "redis://localhost:6379" redis-key: "id:matcher" rego-rule-path: data.matchers.move_call_matches gas-usage: value: "<1500000" window: 1day action: allow ``` Supported `location-type` values: * `file` - local file * `redis` - Redis stored * `http` - download from remote URL Refer to [Gas Station Server repository](https://github.com/iotaledger/gas-station/blob/dev/docs/access-controller.md#rego-expression-sources) for detailed config syntax and more examples. ### External Filtering with Hooks Access Controller allows decision delegation via HTTP hooks if predicates and Rego rules are insufficient. A hook is a remote service that receives the transaction payload and returns a decision. The hook can be used to implement complex logic, such as checking external databases, performing additional validations, or integrating with third-party services. The hook is another type of `action` in the rule definition. It is executed when the rule matches and the hook's response determines the following action. #### Example Hook Usage ```yaml rules: - move-call-package-address: "0x0101010101010101010101010101010101010101010101010101010101010101" ptb-command-count: "<=1" action: "https://hook.addr" - sender-address: "*" gas-usage: value: "<1000000" window: 1day count-by: [ sender-address ] action: allow ``` This configuration sends matching transactions (package address `0x0101...`) to a webhook and falls back to the gas-usage rule if the hook returns `noDecision`. #### Decision Schema The hook must return a JSON response with the following structure: ```json { "decision": "allow|deny|noDecision" } ``` * `allow` or `deny`: immediately apply decision and stop processing further rules. * `noDecision`: skip this rule and evaluate the next one. :::info For up-to-date information on the hook response schema, refer to the OpenAPI specification at [Github](https://github.com/iotaledger/gas-station/blob/dev/docs/hook-openapi.json). ::: #### Hook Implementation Example To implement a custom hook, use the example in the [Gas Station Server repository](https://github.com/iotaledger/gas-station/tree/dev/examples/hook). ### Dynamic Reloading The Access Controller does not automatically reload external resources at runtime. To apply changes in Rego scripts, Redis-stored matchers, or hook logic, you must call the [`/v1/reload_access_controller`](../api-reference/api-reference.mdx#reload-access-controller) API endpoint. This endpoint triggers a reload of the Access Controller configuration, allowing you to update rules, matchers, and hooks without restarting the Gas Station Server. :::warning Please note that the Access Controller doesn't protect the endpoint, so it should be secured with proper authentication mechanisms to prevent unauthorized access. ::: --- ## Monitoring & Analytics ### Prometheus Metrics To ensure smooth operations, the **Gas Station Server** exposes a **Prometheus metrics** endpoint (`/metrics`). These metrics provide valuable insights into system performance, resource utilization, and transaction throughput. When running multiple Gas Station Servers, an external **Prometheus Server** is recommended for centralized data aggregation and visualization. ### In-depth Monitoring & Traffic Analysis For deeper insights, developers can enable **transaction effect logging** by setting `TRANSACTIONS_LOGGING=true`. This feature allows external analytics tools such as **Elasticsearch, Splunk, or Datadog** to parse and analyze transaction data. By logging every signed transaction, developers can: * Track transaction sources and patterns. * Analyze gas station usage over time. * Detect anomalies or suspicious activity in real time. ### Example of integration ```mermaid flowchart LR Gas_Station["Gas Station Server"] -- Logs Transaction Details --> Loki(("Loki")) Gas_Station -- "Metrics Endpoint /metrics" --> Prometheus_Server[("Prometheus Server")] Loki -- "Parsed Transaction Data" --> Prometheus_Server Prometheus_Server -- Aggregated Data --> Grafana((("Grafana Dashboard"))) ``` A typical scenario involves **analyzing transaction performance** to detect anomalies or improve resource allocation. The **Gas Station Server** continuously generates two key data sources: 1. **Logs** – Capturing transaction events and potential errors, which are forwarded to **Loki** for structured logging. 2. **Metrics** – Exposing system performance details via the **Prometheus metrics endpoint** (`/metrics`), which is periodically scraped by a **Prometheus Server**. Since Loki integrates with **Prometheus**, structured log data is made available alongside real-time metrics. **Grafana** then queries **Prometheus** to visualize both system performance trends and transaction behaviors. For example, if a sudden surge in transactions is detected, developers can **cross-reference logs with performance metrics** in Grafana to determine whether the increase is due to organic growth or potential spam activity. If anomalies are found, rate-limiting mechanisms or access control adjustments can be made accordingly. This architecture ensures that the **Gas Station remains scalable, efficient, and secure** while providing real-time insights for continuous improvement. --- ## Deployment Guide The IOTA Gas Station is distributed as two Docker images: 1. [iotaledger/gas-station-tool](https://hub.docker.com/r/iotaledger/gas-station-tool) – A utility tool for: * Generating configuration files * Performing health checks * Running benchmarks 2. [iotaledger/gas-station](https://hub.docker.com/r/iotaledger/gas-station) – The Gas Station server with the exposed **HTTP API**. --- ## **Test Deployment** :::warning Test Deployment This deployment guide is not intended for production environments. In production, it is essential to implement robust security practices. ::: :::info For updated instructions and additional configuration options, refer to the official [IOTA Gas Station Repository](https://github.com/iotaledger/gas-station) ::: For testing purposes, the easiest method is to use the provided **Docker Compose** setup. This allows you to quickly spin up a Gas Station instance using pre-configured Docker images. ### **Prerequisites** Ensure you have the following tools installed: * [Git](https://github.com/git-guides/install-git) * [Docker](https://docs.docker.com/engine/install/) * [Docker Compose](https://docs.docker.com/compose/install/) ### **Setup Steps** 1. **Clone the IOTA Gas Station Repository:** ``` git clone https://github.com/iotaledger/gas-station ``` 2. **Navigate to the Docker Directory and Generate the Config File:** ``` cd gas-station/docker ../utils/./gas-station-tool.sh generate-sample-config --config-path config.yaml --docker-compose ``` :::info Network You can use the `-n`/`--network` flag to specify the network. Possible options are: `local`, `devnet`, `testnet`, `mainnet`. `testnet` is default. ::: :::info Private Keys If the generated private key pair doesn’t meet your requirements, replace it with your own keys. ::: 3. **Set Up Authentication:** Define a bearer token for the Gas Station API using the `GAS_STATION_AUTH` environment variable. If set, this token must be provided in all requests to the Gas Station, except for the `/` and `/version` endpoints. It can also be omitted to disable default authentication, e.g. if one wants to add a custom authentication to the server. In this case, requests against the Gas Station can be made without an authentication token. #### 4. Start the Gas Station ``` GAS_STATION_AUTH=[bearer_token] docker compose up ``` Your Gas Station instance should now be running and accessible via its [HTTP API](../api-reference/api-reference.mdx). --- ## Production Deployment There is no single official method for deploying the IOTA Gas Station in production environments. Each organization has its own security standards, infrastructure preferences, and operational requirements. This flexibility allows teams to tailor their deployments according to their specific needs. However, there are important security considerations that should be taken into account to ensure a safe and resilient setup. Below are some security recommendations you should consider when planning your production deployment: ### Security Recommendations When deploying the IOTA Gas Station in a production environment, it's essential to adopt a multi-layered security approach. The system should be designed to minimize exposure to potential threats while maintaining high availability and performance. #### Deploy Behind a Security Gateway * Position the Gas Station behind a security gateway (e.g., **NGINX**, **HAProxy**, or a cloud-based API gateway) to act as the first line of defense. This gateway should enforce authentication protocols, rate-limiting, and IP whitelisting to prevent unauthorized access and mitigate DDoS attacks. * Configure the gateway to handle TLS/SSL termination, ensuring that all traffic between the client and the gateway is encrypted. * Implement firewall rules and intrusion detection systems (IDS) to monitor and block malicious activity. #### Use Kubernetes (K8s) for Scalability and High Availability * Utilize Kubernetes to orchestrate and manage a pool of Gas Station server instances. Kubernetes enables dynamic scaling, allowing the system to handle fluctuating traffic loads while maintaining service availability. * Employ Kubernetes **Namespaces** and **Role-Based Access Control (RBAC)** to isolate resources and control user permissions. * Set up **Kubernetes Network Policies** to manage internal traffic flow between pods and restrict unnecessary communications. * Implement auto-scaling for Gas Station instances based on resource utilization metrics, ensuring optimal performance under varying loads. #### Centralized Redis & KMS for Consistency and Security * Use a centralized **Redis** instance for maintaining the state of gas object reservations across all Gas Station servers. Ensure Redis is secured with proper authentication mechanisms and is accessible only from within the trusted network. * Integrate an external **Key Management System (KMS)** for handling private keys securely. All Gas Station servers within the same pool should interact with the shared KMS for signing transactions, maintaining consistency and simplifying key management. * Regularly rotate encryption keys and audit KMS access logs to detect potential misuse or unauthorized access. #### Access Controller Manager * Implement the **Access Controller Manager** to define and enforce access control rules that regulate transaction execution. This prevents unintentional or malicious usage of the Gas Station. * Regularly review and update rules to adapt to evolving usage patterns and security requirements. #### Additional Security Best Practices * **Implement Logging and Monitoring:** Use tools like **Prometheus**, **Grafana**, and **Loki** to monitor system health, resource usage, and transaction flows. Set up alerts to detect unusual activities or performance degradation. * **Enable Secure API Access:** Protect the Gas Station’s API with authentication tokens (e.g., bearer tokens) and ensure that sensitive endpoints are not exposed publicly. * **Conduct Regular Security Audits:** Perform periodic vulnerability assessments and penetration testing to identify and address potential security gaps. --- The **IOTA Gas Station** offers flexible deployment options to suit both testing and production environments. While test deployments can be quickly set up using Docker Compose, production deployments require stricter security measures and proper infrastructure scaling. By leveraging tools like **Redis**, **KMS**, and **Kubernetes**, along with robust monitoring and security practices, you can deploy a resilient and efficient Gas Station infrastructure suitable for real-world usage. --- ## Gas Station ![IOTA Gas Station](/img/banner/banner_gas_station.png) :::note **Important: Self-Hosted Component** The IOTA Gas Station is a self-hosted building block that allows application providers to sponsor IOTA gas fees of their customers. You deploy, run, and tune your own instance, deciding which transactions you cover for your users, platform, or ecosystem. The IOTA Foundation does not run a network-wide or app-specific Gas Station service. We may launch short-lived instances for community campaigns, demos, or testing, but all production or commercial sponsorship sits with the application provider. Need help getting started? We’re happy to guide you through the setup. ::: The IOTA Gas Station is a containerized component designed to simplify user onboarding and enhance engagement by enabling businesses and developers to sponsor transaction fees on the IOTA network. Many users hesitate to interact with Web3 applications due to the complexity of wallets and gas fees. The Gas Station eliminates this friction by allowing seamless interactions without requiring users to acquire tokens. Developers can configure it to cover transaction costs, ensuring a smooth and accessible experience. With features like access control, usage metrics, and customizable limits, it provides a flexible solution for enterprises, dApp builders, and Web3 projects looking to optimize gas management. --- ## Node Operator Overview Operator guides demonstrate how to run full nodes and validator nodes on IOTA. ## Full Node Operator Guides IOTA full nodes validate blockchain activities, including transactions, checkpoints, and epoch changes. Each full node stores and services the queries for the blockchain state and history. - [Full Node Overview](full-node/overview.mdx) - [Full Node Configuration](full-node/configuration.mdx) ## Validator Node Operator Guides IOTA validators focus on servicing and processing transactions. When a validator commits a new set of transactions (or a block of transactions), the validator pushes that block to all connected full nodes that then service the queries from clients. - [Validator Node Overview](validator-node/overview.mdx) - [Validator Node Configuration](validator-node/configuration.mdx) --- ## Security Releases There may be instances where urgent security fixes need to be rolled out before publicly announcing their presence (issues affecting liveness, invariants such as IOTA supply, governance, etc.). To avoid active exploitation, the IOTA Foundation will release signed security binaries incorporating such fixes with a delay in publishing the source code until a large percentage of our validators have patched the vulnerability. This release process will be different, and we expect to announce the directory for such binaries out of band. You can download all the necessary signed binaries and Docker artifacts incorporating the security fixes by using the [download_private.sh](https://github.com/iotaledger/iota/blob/develop/setups/download_private.sh) script. Usage: ```shell ./download_private.sh ``` You can also download and verify specific binaries that may not be included by the above script using the [download_and_verify_private_binary.sh](https://github.com/iotaledger/iota/blob/develop/setups/download_and_verify_private_binary.sh) script. Usage: ```shell ./download_and_verify_private_binary.sh ``` --- ## State Sync Full Node (SSFN) Guide ## Hardware requirements ## What is a State Sync Full Node? A State Sync Full Node (SSFN) is a specialized IOTA full node that acts as a bridge between validators and the broader IOTA network. While it shares most characteristics with regular full nodes, it is specifically configured to optimize state synchronization for validators. ## Why are SSFNs Important for Validators? SSFNs play a crucial role in a validator's infrastructure for several reasons: 1. **Network Isolation**: SSFNs provide a security buffer between validators and the public network, helping protect validators from potential attacks or network issues. 2. **Optimized State Sync**: They are specifically configured to handle state synchronization efficiently, ensuring validators maintain accurate and up-to-date network state. 3. **Resource Optimization**: By offloading some network communication tasks to SSFNs, validators can focus their resources on their primary validation duties. 4. **Redundancy**: Running multiple SSFNs provides failover capability and ensures continuous operation even if one SSFN experiences issues. ## Recommended Setup For optimal validator operation, we recommend: - Running at least 1 SSFN per validator - Deploying SSFNs on separate machines from the validator - Ensuring high-bandwidth, low-latency connections between SSFNs and their validator ![SSFN Architecture](ssfn-diagram.svg) ## How SSFNs Differ from Regular Full Nodes SSFNs have several key differences from standard full nodes: 1. **Peering Configuration**: - SSFNs are directly peered with their validator - They are the only nodes that explicitly set validators as their peers 2. **Resource Optimization**: - Indexing is disabled to reduce resource usage - Aggressive pruning is enabled to maintain a smaller database footprint - Metrics are pushed to IOTA's metric proxy for monitoring ## Configuration Guide ### 1. Generate Network Keys First, generate a network key for your SSFN: ```bash iota keytool generate ed25519 ``` Example output: ``` ╭─────────────────────────┬──────────────────────────────────────────────────────────────────────────────╮ │ alias │ │ │ iotaAddress │ 0x41a6f4b3985d32260c3b616e61eab4ad3d3d30dff10d9f5c55094f7fbf6446d5 │ │ publicBase64Key │ WGf/j8NGru668qzxydByVAkosLdh7KbwJ/I7o8qwtfU= │ │ publicBase64KeyWithFlag │ AFhn/4/DRq7uuvKs8cnQclQJKLC3Yeym8CfyO6PKsLX1 │ │ keyScheme │ ed25519 │ │ flag │ 0 │ │ mnemonic │ chalk practice cloud wheat skull stay leader next lesson ceiling cake pink │ │ peerId │ 5867ff8fc346aeeebaf2acf1c9d072540928b0b761eca6f027f23ba3cab0b5f5 │ ╰─────────────────────────┴──────────────────────────────────────────────────────────────────────────────╯ ``` This will create a file like `0x41a6f4b3985d32260c3b616e61eab4ad3d3d30dff10d9f5c55094f7fbf6446d5.key`. You can rename the file to `network.key` and make a note of the `peerId` (`5867ff8fc346aeeebaf2acf1c9d072540928b0b761eca6f027f23ba3cab0b5f5`) in the output. ### 2. SSFN Configuration :::info Default Firewall Ports Make sure to open these ports in your firewall configuration. (These are the defaults, but may vary for a custom setup) | Protocol/Port | Reachability | Purpose | Description | | ------------- | ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------- | | TCP/8080 | Outbound | Validator/transaction interface | full nodes connect to validators to collect partial signatures for transactions | | UDP/8084 | Inbound/Outbound | Peer-to-peer state sync interface | nodes sync their state with each other | | TCP/9000 | Inbound | JSON RPC endpoint | API for interaction with the network | | TCP/9184 | Localhost | Prometheus metrics scraping | for local monitoring only | ::: Create or modify your SSFN's configuration file with these specific settings: ```yaml file=/setups/validator/ssfn-mainnet.yaml ``` ```yaml file=/setups/validator/ssfn-testnet.yaml ``` :::info You can extract the peer-id from your validator `network.key` by running: `iota keytool show network.key` and copying the `peerId` from the output. ::: ### 3. Metrics Whitelisting To successfully push metrics from your SSFN to IOTA's metrics proxy, you need to be whitelisted: 1. **Provide the following information to the IOTA team**: - Your SSFN's p2p-address (e.g., `/dns/your-ssfn-hostname/udp/8084`) - Your SSFN's peer-id (generated in step 1) 2. **Whitelisting Process**: - Submit this information through the official IOTA Discord or by IOTA Slack - Wait for confirmation that your SSFN has been added to the whitelist - Without whitelisting, your metrics will be rejected by the proxy :::note Whitelisting Whitelisting is required for security reasons and helps prevent unauthorized access to the metrics infrastructure. ::: ### 4. Validator Configuration Update your validator's configuration to connect to your SSFNs: ```yaml p2p-config: seed-peers: - address: /dns/ssfn1-hostname/udp/8084 peer-id: - address: /dns/ssfn2-hostname/udp/8084 peer-id: ``` ## Best Practices 1. **Monitoring**: - Regularly monitor SSFN metrics through IOTA's metric proxy - Set up alerts for connectivity issues or resource constraints 2. **Maintenance**: - Keep SSFNs updated with the latest IOTA software - Regularly verify the connection between SSFNs and validator - Monitor disk usage as even with pruning, storage needs will grow over time 3. **Security**: - Place SSFNs behind appropriate firewalls - Restrict network access to only necessary ports - Regularly update the underlying operating system and dependencies ## Troubleshooting If you experience issues with your SSFNs: 1. Verify network connectivity between SSFNs and validator 2. Check logs for any error messages 3. Ensure all peer IDs are correctly configured 4. Verify that pruning is working as expected by monitoring database size 5. Confirm metrics are being successfully pushed to the metrics proxy --- ## Validator Commands ## Overview This guide focuses on using the [IOTA CLI `validator` commands](../../developer/references/cli/validator.mdx). :::info This tool only supports pending validators and active validators at the moment. ::: ## Preparation 1. Make sure you have completed all the [prerequisites](../../developer/getting-started/install-iota.mdx#prerequisites). 2. Build the `iota` binary, which you will need for the genesis ceremony. This step can be done on any machine you like. It does not have to be done on the machine on which you will run the validator. 1. Clone the git repo: ```sh git clone git@github.com:iotaledger/iota.git && cd iota ``` 2. Check out the commit we will be using for the testnet: ```sh git checkout testnet ``` 3. Build the `iota` binary: ```sh cargo build --bin iota ``` 4. Remember the path to your binary: ```sh export IOTA_BINARY="$(pwd)/target/debug/iota" ``` 3. Run the following command to set up your IOTA account and CLI environment. 1. If this is the first time running this program, it will ask you to provide an IOTA Fullnode Server URL and a meaningful environment alias. It will also generate a random key pair in `iota.keystore` and a config `client.yaml`. Swap in your validator account key if you already have one. 2. If you already set it up, simply make sure: - `rpc` is correct in `client.yaml`. - `active_address` is correct in `client.yaml`. - `iota.keystore` contains your account key pair. If at this point you can't find where `client.yaml` or `iota.keystore` is or have other questions, read the [IOTA Client CLI tutorial](../../developer/references/cli/client.mdx). ```sh $IOTA_BINARY client ``` 4. To test that you are connected to the network and have configured your config correctly, run the following command to display your validator info. ```sh $IOTA_BINARY validator display-metadata ``` ## Using IOTA CLI Validator Commands ### Print Help Info ```sh $IOTA_BINARY validator --help ``` ### Display Validator Metadata ```sh $IOTA_BINARY validator display-metadata ``` or ```sh $IOTA_BINARY validator display-metadata ``` to print another validator's information. ### Update Validator Metadata Run the following to see how to update validator metadata. Read the description carefully about when the change will take effect. ```sh $IOTA_BINARY validator update-metadata --help ``` You can update the following on-chain metadata: 1. Name 2. Description 3. Image URL 4. Project URL 5. Network address 6. P2P address 7. Primary address 8. Authority public key 9. Network public key 10. Protocol public key Notably, only the first four metadata listed above take effect immediately. If you change any metadata from points 5 to 10, they will be changed only after the next epoch. **For these, you'll want to restart the validator program immediately after the next epoch, with the new key files and/or updated `validator.yaml` config. Particularly, make sure the new address is not behind a firewall.** Run the following to see how to update each metadata. ```sh $IOTA_BINARY validator update-metadata --help ``` ### Operation Cap Operation Cap allows a validator to authorize another account to perform certain actions on behalf of this validator. To avoid touching account keys too often and allowing them to be stored offline, validators can delegate the operation ability to another address. This address can then update the reference gas price and tallying rule on behalf of the validator. Upon creating a `Validator`, an `UnverifiedValidatorOperationCap` is created as well and transferred to the validator address. The holder of this `Cap` object (short for "Capability") can therefore perform operational actions for this validator. To authorize another address to conduct these operations, a validator transfers the object to another address that they control. The transfer can be done by using the IOTA client CLI: `iota client transfer`. To rotate the delegatee address or revoke the authorization, the current holder of `Cap` transfers it to another address. In the event of compromised or lost keys, the validator can create a new `Cap` object to invalidate the incumbent one. This is done by calling `iota_system::rotate_operation_cap`: ```sh $ iota client call --package 0x3 --module iota_system --function rotate_operation_cap --args 0x5 --gas-budget 10000 ``` By default, the new `Cap` object is transferred to the validator address, which can then be transferred to the new delegatee address. At this point, the old `Cap` becomes invalid and no longer represents eligibility. To get the current valid `Cap` object's ID of a validator, use the IOTA Client CLI `iota client objects` command after setting the holder as the active address. The Operation Cap holder (either the validator itself or the delegatee) updates its Gas Price and reports validator peers with the Operation Cap. ### Report Validators To report a validator or undo an existing report, the sender needs to hold a valid [`UnverifiedValidatorOperationCap`](#operation-cap). The sender could be the validator itself or a trusted delegatee. To report validator peers, run: ```sh $IOTA_BINARY validator report-validator ``` Add `--undo-report true` if it intends to undo an existing report. Similarly, if the account is a delegatee, add the `--operation-cap-id ` option to the command. ### Become a Validator To become a validator candidate, first run: ```sh $IOTA_BINARY validator make-validator-info ``` This will generate a `validator.info` file and key pair files. The output of this command includes: 1. Four key pair files. ==Set their permissions with minimal visibility (chmod 600, for example) and store them securely==. They are needed when running the validator node as covered below. - If you follow this guide thoroughly, this key pair is actually copied from your `iota.keystore` file. 2. `validator.info` file that contains your validator info. **Double-check all information is correct**. Then run the following command: ```sh $IOTA_BINARY validator become-candidate {path-to}validator.info ``` to submit an on-chain transaction to become a validator candidate. The parameter is the file path to the `validator.info` generated in the previous step. **Make sure the transaction succeeds (printed in the output).** At this point, you are a validator candidate and can start to accept self-staking and delegated staking. **If you haven't already, start a fullnode now to catch up with the network. When you officially join the committee but are not fully up-to-date, you cannot make meaningful contributions to the network and may be subject to peer reporting, hence face the risk of reduced staking rewards for you and your delegators.** Add stake to a validator's staking pool: https://docs.iota.org/developer/references/framework/iota_system/iota_system#iota_system_iota_system_request_add_stake Once you collect enough staking amount, run: ```sh $IOTA_BINARY validator join-validators ``` to become a pending validator. A pending validator will become active and be eligible to join the committee starting from the next epoch. ### Leave Validators To leave the validators, run: ```sh $IOTA_BINARY validator leave-validators ``` Afterwards, you will be removed from the validators (and potentially the committee) starting from the next epoch. ### Generate the Payload to Create PoP Serialize the payload that is used to generate a Proof of Possession. This allows the signer to take the payload offline for an Authority protocol BLS keypair to sign. ```sh $IOTA_BINARY validator serialize-payload-pop --account-address $ACCOUNT_ADDRESS --authority-public-key $BLS_PUBKEY Serialized payload: $PAYLOAD_TO_SIGN ``` ## Test Becoming a Validator in a Local Network ### 1. Start a Local Network with a Larger Faucet Amount Set a larger faucet amount and two coins per request: one coin to stake, above the minimum validator joining stake, and one to pay the gas fees. ```bash iota-localnet start --force-regenesis --with-faucet --faucet-amount 6600000000000000 --faucet-coin-count 2 ``` ### 2. Request Coins from the Faucet ```bash iota client switch --env localnet ``` Then request funds: ```bash iota client faucet --url http://127.0.0.1:9123/gas ``` ### 3. Make Validator Info ```bash iota validator make-validator-info validator0 description https://iota.org/logo.png https://www.iota.org 127.0.0.1 ``` ### 4. Become a Validator ```bash iota validator become-candidate validator.info ``` ### 5. Stake Funds Get an object ID for a gas coin to stake and the address to stake to: ```bash iota client gas && iota client active-address ``` Stake the coin object: ```bash iota client call --package 0x3 --module iota_system --function request_add_stake --args 0x5 --gas-budget 10000000 ``` Example where `0xfff7...` is a coin object ID, `0x111...` is the validator address: ```bash iota client call --package 0x3 --module iota_system --function request_add_stake --args 0x5 0xfff7d5a924a599e811e307c3eeb65d69906054466ac098a2715a19ab802ddf15 0x111111111504e9350e635d65cd38ccd2c029434c6a3a480d8947a9ba6a15b215 --gas-budget 10000000 ``` All in one: ```bash coinObjectId=$(iota client gas --json | jq '.[0].gasCoinId') validatorAddress=$(iota client active-address) iota client call --package 0x3 --module iota_system --function request_add_stake --args 0x5 $coinObjectId $validatorAddress --gas-budget 10000000 ``` ### 6. Finally, Join the Validators ```bash iota validator join-validators ``` ### Combined First terminal: ```bash iota-localnet start --force-regenesis --with-faucet --faucet-amount 6600000000000000 --faucet-coin-count 2 ``` Second terminal after the faucet is up: ```bash iota client switch --env localnet iota client faucet --url http://127.0.0.1:9123/gas sleep 2 iota validator make-validator-info validator0 description https://iota.org/logo.png https://www.iota.org 127.0.0.1 iota validator become-candidate validator.info sleep 2 coinObjectId=$(iota client gas --json | jq '.[0].gasCoinId') validatorAddress=$(iota client active-address) iota client call --package 0x3 --module iota_system --function request_add_stake --args 0x5 $coinObjectId $validatorAddress --gas-budget 10000000 sleep 2 iota validator join-validators sleep 2 iota validator display-metadata ``` --- ## Archives(3) import ArchivesContent from '../../common/archives.mdx'; --- ## Genesis(3) import GenesisText from '../../common/genesis.mdx'; --- ## Network(Configs) ## Connectivity To synchronize the state, the node needs to establish connections with other nodes that have already joined the network. Ensure that port `8084` is open and set up an external address for node discovery. ## Required Network Ports Ensure the following ports are open in your firewall: | Protocol/Port | Reachability | Purpose | Description | | ------------- | ---------------- | --------------------------------- | ------------------------------------------------------------------------------- | | TCP/8080 | Inbound | Validator/transaction interface | full nodes connect to validators to collect partial signatures for transactions | | TCP/8081 | Inbound/Outbound | Consensus interface | validator nodes send consensus blocks to each other | | UDP/8084 | Inbound/Outbound | Peer-to-peer state sync interface | nodes sync their state with each other | | TCP/8443 | Outbound | Metrics pushing | validator metrics get pushed to IOTA Foundation monitoring | | TCP/9184 | Localhost | Metrics scraping | for local monitoring only | To run a validator successfully, it is critical that ports are open as outlined, including the specific protocol (TCP/UDP). ## Network Buffer From load testing IOTA validator networks, it has been determined that the default Linux network buffer sizes are too small. We recommend increasing them using one of the following two methods: ### Option 1: With /etc/sysctl.d/ These settings can be added to a new sysctl file specifically for the `iota-node`, or appended to an existing file. Modifications made in this way will persist across system restarts. Create a new sysctl file for the `iota-node`: ```sh $ sudo nano /etc/sysctl.d/100-iota-node.conf ``` Add these lines to the file, overwriting existing settings if necessary: ```sh net.core.rmem_max = 104857600 net.core.wmem_max = 104857600 net.ipv4.tcp_rmem = 8192 262144 104857600 net.ipv4.tcp_wmem = 8192 262144 104857600 ``` Apply the settings immediately, before the next restart: ```sh $ sudo sysctl --system ``` ### Option 2: With sysctl Command These modifications do not persist across system restarts. Therefore, the commands should be run each time the host restarts. ```sh $ sudo sysctl -w net.core.rmem_max=104857600 $ sudo sysctl -w net.core.wmem_max=104857600 $ sudo sysctl -w net.ipv4.tcp_rmem="8192 262144 104857600" $ sudo sysctl -w net.ipv4.tcp_wmem="8192 262144 104857600" ``` ### Verification To verify that the system settings have indeed been updated, check the output of the following command: ```sh $ sudo sysctl -a | egrep [rw]mem ``` --- ## Pruning(Configs) import PruningContent from '../../common/pruning.mdx'; --- ## Snapshots(3) import SnapshotsContent from '../../common/snapshots.mdx'; --- ## Validator Node Configuration The IOTA validator node configuration file is a YAML file (usually named `validator.yaml`) that you can use to configure your `iota-node`. To simplify the setup process, you can use the default validator node configuration file: ```yaml file=/setups/validator/validator-mainnet.yaml ``` ```yaml file=/setups/validator/validator-testnet.yaml ``` This default configuration uses default ports, archive and pruning settings, and also covers peering configuration for the selected network. If you're fine with the default settings, you can use the template as is and continue with the node setup either [using Docker](docker.mdx) or [using systemd](systemd.mdx). For more specific guidance on how to configure your node to suit your needs, e.g., configure snapshot, refer to the following sections: - [Network](configs/network.mdx) - Peering configuration for the node. - [Pruning](configs/pruning.mdx) - How to prune your validator node to leverage storage space. - [Snapshots](configs/snapshots.mdx) - How to configure formal snapshots for your validator node. - [Archives](configs/archives.mdx) - How to configure archives for your validator node. --- ## Docker Setup(Validator-node) ## Required Network Ports Ensure the following ports are open in your firewall: | Protocol/Port | Reachability | Purpose | Description | | ------------- | ---------------- | --------------------------------- | ------------------------------------------------------------------------------- | | TCP/8080 | Inbound | Validator/transaction interface | full nodes connect to validators to collect partial signatures for transactions | | TCP/8081 | Inbound/Outbound | Consensus interface | validator nodes send consensus blocks to each other | | UDP/8084 | Inbound/Outbound | Peer-to-peer state sync interface | nodes sync their state with each other | | TCP/8443 | Outbound | Metrics pushing | validator metrics get pushed to IOTA Foundation monitoring | | TCP/9184 | Localhost | Metrics scraping | for local monitoring only | ## 1. Generate Validator Information :::info Key Security It is recommended to run these steps in a secure local environment to avoid storing the private key controlling the validator rewards on the node. ::: ### How to Generate a New Set of Validator Information #### 1.1. Install and configure the IOTA CLI See [Install IOTA](https://docs.iota.org/developer/getting-started/install-iota) on how to install the `iota` CLI and [Connect to an IOTA Network](https://docs.iota.org/developer/getting-started/connect) on how to select the correct environment (Mainnet or Testnet). #### 1.2. Generate a Validator Account Validators are owned and controlled by an Ed25519 address. You can create a new address, import an existing one or use your Ledger hardware wallet to control a Validator. You can skip this step if you already have an account configured that you want to use for the Validator. ```bash iota client new-address --alias ``` This generates a new key pair and stores the files within the standard working config folder, `~/.iota/iota_config/` (in Unix). Executing the above command provides the key pair info as output, e.g.: ```bash ╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Created new keypair and saved it to keystore. │ ├─────────────────────────┬──────────────────────────────────────────────────────────────────────────────┤ │ alias │ │ │ address │ 0x3607bc87821a13d861e1e09cd5a2525b272f8dbc3e00556a8baf940a39120f32 │ │ publicBase64Key │ m6mBnT0tq9CA1zgdwy9y6ag2e8sWyzHVAoca+zLKJq8= │ │ publicBase64KeyWithFlag │ AJupgZ09LavQgNc4HcMvcumoNnvLFssx1QKHGvsyyiav │ │ keyScheme │ ed25519 │ │ recoveryPhrase │ glimpse hard relax method kid deliver nominee wait brief surprise speed pond │ ╰─────────────────────────┴──────────────────────────────────────────────────────────────────────────────╯ ``` ```bash iota keytool import "glimpse hard relax method kid deliver nominee wait brief surprise speed pond" ed25519 --alias ``` This imports an existing key pair using the given mnemonic and stores the files within the standard working config folder, `~/.iota/iota_config/` (in Unix). Executing the above command provides the key pair info as output, e.g.: ```bash ╭─────────────────────────┬──────────────────────────────────────────────────────────────────────╮ │ alias │ │ │ iotaAddress │ 0x3607bc87821a13d861e1e09cd5a2525b272f8dbc3e00556a8baf940a39120f32 │ │ publicBase64Key │ m6mBnT0tq9CA1zgdwy9y6ag2e8sWyzHVAoca+zLKJq8= │ │ publicBase64KeyWithFlag │ AJupgZ09LavQgNc4HcMvcumoNnvLFssx1QKHGvsyyiav │ │ keyScheme │ ed25519 │ │ flag │ 0 │ │ peerId │ 9ba9819d3d2dabd080d7381dc32f72e9a8367bcb16cb31d502871afb32ca26af │ ╰─────────────────────────┴──────────────────────────────────────────────────────────────────────╯ ``` ```bash iota keytool import-ledger --alias ``` This imports an existing public key from the connected Ledger hardware wallet and stores the reference to it within the standard working config folder, `~/.iota/iota_config/` (in Unix). Executing the above command provides the key pair info as output, e.g.: ```bash ╭─────────────────────────┬──────────────────────────────────────────────────────────────────────╮ │ alias │ │ │ iotaAddress │ 0x3607bc87821a13d861e1e09cd5a2525b272f8dbc3e00556a8baf940a39120f32 │ │ publicBase64Key │ m6mBnT0tq9CA1zgdwy9y6ag2e8sWyzHVAoca+zLKJq8= │ │ publicBase64KeyWithFlag │ AJupgZ09LavQgNc4HcMvcumoNnvLFssx1QKHGvsyyiav │ │ keyScheme │ ed25519 │ │ flag │ 0 │ │ peerId │ 9ba9819d3d2dabd080d7381dc32f72e9a8367bcb16cb31d502871afb32ca26af │ │ externalSource │ ledger │ │ derivationPath │ m/44'/4218'/0'/0'/0' │ ╰─────────────────────────┴──────────────────────────────────────────────────────────────────────╯ ``` - `alias`: A human-readable identifier to use within the CLI scope to refer to a key pair. - `address`: The public address representing the key pair. - `publicBase64Key`: The public key, base64 encoded. - `publicBase64KeyWithFlag`: The public key prefixed with the scheme flag byte, base64 encoded. - `keyScheme`: The cryptographic scheme used to derive the keypair; the `ed25519` is the standard scheme, used most of the time, while the `BLS12381` scheme is used for the `authority_key`. :::info From now on, this key pair will be referred to as `account-key`. ::: #### 1.3. Switch to the validator account key This makes that key pair active in the CLI. ```bash iota client switch --address ``` #### 1.4. Generate the validator data Finally, all validator's remaining key pairs and data can be generated using the following command: ```bash iota validator make-validator-info \ \ \ \ \ \ ``` - `name`: A human-readable validator name, e.g., `validator1`. - `description`: A human-readable validator description, e.g., `"this is a validator"`. - `image_url`: The validator image URL, e.g., `https://www.iota.org/favicon.png`. - `project_url`: The validator project URL, e.g., `https://www.iota.org`. - `host_name`: The host name that is used to generate the validator `network_address`, `p2p-address` and `primary_address`, e.g., `validator1.iota.org`. This command generates a `validator.info` file and up to 4 key pair files in the same directory where the command was executed. All keys but the `account.key` will need to be copied over to the validator node and included in the YAML configuration of the node. See [Validator Node Configuration](https://docs.iota.org/operator/validator-node/configuration). - `account.key` contains an ed25519 private key _(keep this one private)_. This file will be skipped if you used a Ledger hardware wallet. - `network.key` contains an ed25519 private key _(copy over to the validator node)_. - `authority.key` contains a BLS12381 private key _(copy over to the validator node)_. - `protocol.key`contains an ed25519 private key _(copy over to the validator node)_. - `validator.info` contains the validator information, e.g.: ```yaml --- info: name: validator1 account-address: "0x547b20ffca39cf1c9f57e7d1ff946d4720df48bb582e89b763b5d488ec23f5fa" authority-key: h93SKC2tFXMDt+lu4SRb3KA668/lJCPREHnzYZelaQ3iAu0RuiHqETBg/1jkV9HFCECvBCrzKjTuVD/bt5yUDon5nPKQmCyYEmx0NRQmxfP7Szpg17YZ388eT+aTnGEK protocol-key: Lm1Iy5KDV0qlMcGVnQNatAMLxhg8FOxE2q/QUkgLAYA= network-key: ADBhWCBOzqIvsDa9cowpSQ4t1nz+ZQYeRLBGQYe1Dy8= gas-price: 1000 commission-rate: 200 network-address: /dns/validator1.iota.org/tcp/8080/http p2p-address: /dns/validator1.iota.org/udp/8084 primary-address: /dns/validator1.iota.org/udp/8081 description: validator-description1 image-url: "https://www.iota.org/favicon.png" project-url: "https://www.iota.org" proof_of_possession: hpep8yY/JCj/zTEv9Ws7Qow3KO+2jrsX/yAUAykfxzS5vsg7vlwsscJpspf4XF/u ``` Where: - `name`: human-readable validator name, defined above; - `account-address`: The address derived from the `account key` public key. - `authority-key`: The public key derived from the `authority key` private key. - `protocol-key`: The public key derived from the `protocol key` private key. - `network-key`: The public key derived from the `network key` private key. - `gas-price`: An unsigned integer value indicating the gas price proposed for the first epoch. This is unused as of protocol version v5. - `commission-rate`: The validator commission rate, i.e., the fee charged by the validator for staking services; the default set at 2% (200). - `network-address`: The network address of the validator, defined using the `host_name`. - `p2p-address`: The address of the validator used for p2p activities such as state sync, defined using the `host_name`. - `primary-address`: The primary address of the validator used for the consensus, defined using the `host_name`. - `description`: A human-readable validator description, defined above. - `image-url`: The validator image URL, defined above. - `project-url`: The validator project URL, defined above. - `proof_of_possession`: A BLS signature created using the `authority-key`, committed over the following message: `intent || pubkey || address || epoch`. `intent` is serialized to `[5, 0, 0]` representing an intent with scope as "Proof of Possession", `version` as "V0" and `app_id` as "Iota". `pubkey` is the serialized public key bytes of the `authority-key`. `address` is the `account-address`. `epoch` is serialized to `[0, 0, 0, 0, 0, 0, 0, 0]`. :::note You may have noticed that port 8081 is using UDP, which conflicts with the firewall rules. This is a known bug, but the node is actually communicating via TCP. ::: #### 1.5. Register as a Validator Candidate Submit an on-chain transaction to become a validator candidate. The parameter is the file path to the `validator.info` generated by [Make Validator Info](#1-generate-validator-information). Make sure your validator account has enough IOTA to cover the gas costs. On the Testnet you can use the Faucet to receive the necessary funds. :::note Environments Make sure your client is [configured to use the correct environment](https://docs.iota.org/developer/getting-started/connect#connect-your-cli-to-a-network). ::: ```bash iota validator become-candidate ``` You can verify the status by running: ```bash iota validator display-metadata --json true ``` You should see your validator's status is `Candidate`. ## 2. Setup Validator Node :::info Note Perform these steps on the server that will be running the validator instance. ::: ### 2.1. Download Docker Compose configuration Download the docker compose file: ```bash curl -fLJ -o docker-compose.yml https://github.com/iotaledger/iota/raw/refs/heads/develop/setups/validator/docker-compose-mainnet.yml ``` ```bash curl -fLJ -o docker-compose.yml https://github.com/iotaledger/iota/raw/refs/heads/develop/setups/validator/docker-compose-testnet.yml ``` ### 2.2. Download Configuration Template Download the validator configuration file template: ```bash curl -fLJ -o validator.yaml https://github.com/iotaledger/iota/raw/refs/heads/develop/setups/validator/validator-mainnet.yaml ``` ```bash curl -fLJ -o validator.yaml https://github.com/iotaledger/iota/raw/refs/heads/develop/setups/validator/validator-testnet.yaml ``` ### 2.3. Configure P2P Settings Configure the P2P settings in your `validator.yaml` file by following these steps: 1. Update the external address: ```yaml p2p-config: external-address: /dns//udp/8084 ``` :::note Replace `` with the hostname you used while generating the validator info. This address must be accessible from the internet. ::: :::note For better flexibility and faster disaster recovery, we suggest using a hostname with a very low TTL. This allows you to quickly relocate your validator to a new server if hardware fails. Direct IP addressing significantly complicates server migration. ::: ### 2.4. Download Genesis Blobs ```bash curl -fLJO https://dbfiles.mainnet.iota.cafe/genesis.blob curl -fLJO https://dbfiles.mainnet.iota.cafe/migration.blob ``` ```bash curl -fLJO https://dbfiles.testnet.iota.cafe/genesis.blob ``` ### 2.5. Copy Validator Keys Copy the `protocol.key`, `network.key` and `authority.key` you created previously to the server and place them in a folder named `key-pairs`. ### 2.6. Download the latest Formal Snapshot ```bash docker run --rm -v ./iotadb:/opt/iota/db -v ./genesis.blob:/opt/iota/config/genesis.blob iotaledger/iota-node:mainnet /bin/sh -c "/usr/local/bin/iota-tool download-formal-snapshot --latest --genesis /opt/iota/config/genesis.blob --path /opt/iota/db/authorities_db --no-sign-request --verify normal --network mainnet --verbose" ``` ```bash docker run --rm -v ./iotadb:/opt/iota/db -v ./genesis.blob:/opt/iota/config/genesis.blob iotaledger/iota-node:testnet /bin/sh -c "/usr/local/bin/iota-tool download-formal-snapshot --latest --genesis /opt/iota/config/genesis.blob --path /opt/iota/db/authorities_db --no-sign-request --verify normal --network testnet --verbose" ``` ### 2.7. Start Your Validator Node ```bash docker compose up -d docker compose logs -f ``` ## 3. Join the Validators :::info Key Security The following steps should be executed in the same environment you used while creating the validator keys. ::: Make sure your validator account created in [Generate Validator Information](https://docs.iota.org/operator/validator-node/systemd#1-generate-validator-information) has enough IOTA to cover the gas costs. :::note Tesnet Faucet On the Testnet you can use the Faucet to receive the necessary funds. ::: Before joining the validators, ensure you have enough delegated stake by running the following command: ```bash iota validator display-metadata --json true | grep stakingPoolIotaBalance ``` You need [at least 6M IOTA to join the validators](https://docs.iota.org/about-iota/tokenomics/validators-staking#validator-staking-pool-requirements). So make sure the `stakingPoolIotaBalance` is at least `6_000_000_000_000_000`. Once your have enough stake, submit your request to join the validators: ```bash iota validator join-validators ``` ## 4. Monitor Validator Status ```bash iota validator display-metadata --json true ``` You should see your validator's status is `Pending` now, it will become active and join the validators starting from next epoch. ``` 's validator status: Pending ``` You can also check the [Monitoring](./monitoring.mdx) about metrics endpoint. ## 5. State Sync Full Node (SSFN) Setup Follow the [State Sync Full Node (SSFN) Guide](https://docs.iota.org/operator/ssfn_guide). --- ## Monitoring(Validator-node) After a successful setup of a validator node, the next important step is to set up monitoring. Monitoring is important to keep track of the node's health and performance. ## Node Health Metrics IOTA nodes expose a wide range of metrics to be scraped by [Prometheus](https://prometheus.io/). By default, metrics are available at the `http://localhost:9184/metrics` endpoint. The best way to visualize these metrics is to use [Grafana](https://grafana.com/). Additionally, a common approach is to use [node exporter](https://github.com/prometheus/node_exporter) to scrape performance metrics from the node and push them to Prometheus. ### Fetch key health metrics Key health metrics via the `/metrics` HTTP endpoint: ```sh curl -s localhost:9184/metrics | grep -E "^last_executed_checkpoint|^highest_synced_checkpoint|^highest_known_checkpoint|^last_committed_round|^consensus_threshold_clock_round|^highest_received_round|^consensus_proposed_blocks|^uptime" ``` For instance, for a validator node, the output would be: ```sh consensus_proposed_blocks{force="false"} 247 consensus_proposed_blocks{force="true"} 1 consensus_threshold_clock_round 257 highest_known_checkpoint 555 highest_synced_checkpoint 886 last_executed_checkpoint 890 last_executed_checkpoint_age_bucket{le="0.001"} 0 last_executed_checkpoint_age_bucket{le="0.005"} 0 last_executed_checkpoint_age_bucket{le="0.01"} 0 ... last_executed_checkpoint_age_bucket{le="60"} 891 last_executed_checkpoint_age_bucket{le="90"} 891 last_executed_checkpoint_age_bucket{le="+Inf"} 891 last_executed_checkpoint_age_sum 156.52341099999992 last_executed_checkpoint_age_count 891 last_executed_checkpoint_timestamp_ms 1748335503888 uptime{chain_identifier="b5d7e5c8",is_docker="false",os_version="macOS 15.5 Sequoia",process="validator",version="1.1.0"} 196 ``` ### Ensure node health using last checkpoint timestamp To make sure your node runs properly, we check that the last processed checkpoint is recent enough: - 10 seconds is typical - 30 seconds is still fine - You want to check that the timestamp difference stays under 1 minute You can check that from the previous metric `last_executed_checkpoint_timestamp_ms`, and compare timestamps with now using this command: ```shell last_executed_checkpoint_timestamp_ms="$(curl -s localhost:9184/metrics | grep ^last_executed_checkpoint_timestamp_ms | awk '{print $2}')" now_timestamp="$(date +%s%3N)" if (( now_timestamp - last_executed_checkpoint_timestamp_ms < 60000 )); then echo "[OK] healthy & in sync" else echo "[ERROR] Node unhealthy. Last known checkpoint is too old." fi ``` ### Monitor consensus sync status To ensure your node's consensus module is properly synced with the network, monitor the difference between `consensus_commit_sync_local_index` and `consensus_commit_sync_quorum_index`: ```shell metrics="$(curl -s localhost:9184/metrics)" local_index="$(echo "$metrics" | grep ^consensus_commit_sync_local_index | awk '{print $2}')" quorum_index="$(echo "$metrics" | grep ^consensus_commit_sync_quorum_index | awk '{print $2}')" difference=$((quorum_index - local_index)) if (( difference > 100 )); then echo "[WARNING] Consensus module not in sync. Difference: $difference" echo "[INFO] Monitor this difference over time:" echo " - If growing: Node falling behind network" echo " - If shrinking: Node catching up to network" else echo "[OK] Consensus module in sync. Difference: $difference" fi ``` **Key indicators:** - **Difference > 100**: Node's consensus module is not in sync - **Growing difference**: Network is advancing faster than node is syncing - **Shrinking difference**: Node is correctly syncing and catching up ### Monitor skipped proposals Check the `consensus_core_skipped_proposals` metric to detect issues with block production: ```shell metrics="$(curl -s curl -s localhost:9184/metrics)" skipped_proposals="$(echo "$metrics" | grep ^consensus_core_skipped_proposals)" if [[ -z "$skipped_proposals" ]]; then echo "[OK] No skipped proposals metric found" else echo "Skipped proposals by reason:" echo "$skipped_proposals" echo "" echo "Monitor these values over time. Growing counts indicate issues:" echo " - no_quorum_subscriber: Insufficient connections (< 2f+1 stake) to produce blocks" echo " - Other reasons: Check the specific label for the cause" fi ``` **Key indicators:** - **Growing `no_quorum_subscriber`**: Insufficient validator connections (less than 2f+1 stake connected). Node cannot safely produce new blocks - **Growing other labels**: The corresponding label value indicates the specific cause of skipped proposals - Run the script periodically to compare values and identify trends ### Overall node health When all the above healthchecks pass, your node should be considered healthy: - ✅ Last checkpoint timestamp is recent (< 60 seconds) - ✅ Consensus sync difference is acceptable (< 100) - ✅ Validator connections are sufficient (≥ 80% of committee) - ✅ Connection symmetry is maintained ## Logs #### Configuring Logs Log level (error, warn, info, trace) is controlled using the `RUST_LOG` environment variable. The `RUST_LOG_JSON=1` environment variable can optionally be set to enable logging in JSON structured format. Depending on your deployment method, these are configured in the following places: ``` [Service] ... Environment=RUST_BACKTRACE=1 Environment=RUST_LOG=info,iota_core=debug,consensus=debug,jsonrpsee=error ``` Add the following to the node container settings: ```yaml environment: - RUST_BACKTRACE=1 - RUST_LOG=info,iota_core=debug,consensus=debug,jsonrpsee=error ``` It is possible to change the logging configuration while a node is running using the admin interface. #### Verify Configured Logging Values To view the currently configured logging values: ```sh curl -w "\n" localhost:1337/logging ``` To change the currently configured logging values: ```sh curl localhost:1337/logging -d "info" ``` Note that the admin port (`1337`) is only exposed to `localhost` by default, so the commands must be executed inside the container. To view the currently configured logging values: ```sh docker exec curl -w "\n" localhost:1337/logging ``` To change the currently configured logging values: ```sh docker exec curl localhost:1337/logging -d "info" ``` Replace `` with your actual container name, such as `iota-fullnode-docker-setup-fullnode-1`. #### Viewing Logs To view and follow the IOTA node logs: ```sh journalctl -u iota-node -f ``` To search for a particular match: ```sh $ journalctl -u iota-node -g ``` View and follow: ```sh sudo docker compose logs -f [node_container_name] ``` By default, all logs are output. Limit this using `--since`: ```sh sudo docker logs --since 10m -f [node_container_name] ``` ## Monitoring Services Implementing monitoring services is essential to ensure the reliability, security, and performance of the blockchain network by providing real-time insights, detecting anomalies, enabling proactive issue resolution, and receiving automatic alerts. **Prometheus and Grafana** (recommended) Example pre-made dashboards you can use: - Community-owned grafana dashboard: https://github.com/stakeme-team/grafana-iota. - Grafana setup for the [local private network](https://github.com/iotaledger/iota/tree/develop/dev-tools/grafana-local), which might be a good example of how to build your own setup. - Officially supported dashboards can be found [here](https://github.com/iotaledger/iota/tree/develop/setups/grafana) in the IOTA repository. **Dolphin** Dolphin is a CLI tool that provides high-level features for validator and fullnode monitoring. Under the hood, it uses the IOTA node Prometheus metric exporter to check the health of the node. More info: https://gitlab.com/blockscope-net/dolphin-v2. --- ## Validator Node Overview Validators on the IOTA network run special nodes and have additional tasks and responsibilities beyond those of full node operators. ## Requirements to Run a Validator Node on IOTA To run an IOTA validator node, you must: 1. Set up and run a validator node. Follow the setup guides for [Docker](docker.mdx) or [Systemd](systemd.mdx). 2. Ensure your staking pool has at least **6 million IOTA**. This is the minimum required to join the validator set. Other users can delegate IOTA to your pool to help reach this amount. 3. Complete the necessary tasks to join the validator set and maintain the validator status. See the relevant guides for [Docker](docker.mdx) or [Systemd](systemd.mdx) for detailed instructions. Specific steps you must take include: - Install and configure IOTA validator node - Key management - Software updates - On-chain commands - Update the Gas Price Survey - Reporting other validators ## Validator Staking Pool Requirements There are minimum staking requirements a validator must satisfy to become active and to stay in the active validator set. More precisely: - A validator candidate must accrue at least 6M IOTA of stake before they can request to join the validator set. - If an active validator’s stake falls below 5.5M IOTA, the validator has seven epochs of grace period to gain back the stake before being removed from the validator set. - If an active validator’s stake falls below 5M IOTA, the validator is removed from the validator set at the end of the current epoch boundary. IOTA uses 24-hour epochs. ## Hardware Requirements - **RAM:** 128 GB - **CPU:** 24-core processor / 48 vCPUs - **Storage:** 4 TB NVMe SSD - **Network Uplink:** 1 Gbps - **Minimum Stake:** 6 million IOTAs (can be delegated to your validator by someone else) - **RAM:** 64 GB - **CPU:** 8-core processor / 16 vCPUs - **Storage:** 2 TB NVMe SSD - **Network Uplink:** 1 Gbps - **Minimum Stake:** 6 million IOTAs (can be delegated to your validator by someone else) ## Validator Tokenomics - [Validator Tokenomics](./validator-tokenomics.mdx) --- ## Systemd Setup ## Hardware Requirements ## Required Network Ports Ensure the following ports are open in your firewall: | Protocol/Port | Reachability | Purpose | Description | | ------------- | ---------------- | --------------------------------- | ------------------------------------------------------------------------------- | | TCP/8080 | Inbound | Validator/transaction interface | full nodes connect to validators to collect partial signatures for transactions | | TCP/8081 | Inbound/Outbound | Consensus interface | validator nodes send consensus blocks to each other | | UDP/8084 | Inbound/Outbound | Peer-to-peer state sync interface | nodes sync their state with each other | | TCP/8443 | Outbound | Metrics pushing | validator metrics get pushed to IOTA Foundation monitoring | | TCP/9184 | Localhost | Metrics scraping | for local monitoring only | ## 1. Generate Validator Information :::info Key Security It is recommended to run these steps in a secure local environment to avoid storing the private key controlling the validator rewards on the node. ::: ### How to Generate a New Set of Validator Information #### 1.1. Install and configure the IOTA CLI See [Install IOTA](https://docs.iota.org/developer/getting-started/install-iota) on how to install the `iota` CLI and [Connect to an IOTA Network](https://docs.iota.org/developer/getting-started/connect) on how to select the correct environment (Mainnet or Testnet). #### 1.2. Generate a Validator Account Validators are owned and controlled by an Ed25519 address. You can create a new address, import an existing one or use your Ledger hardware wallet to control a Validator. You can skip this step if you already have an account configured that you want to use for the Validator. ```bash iota client new-address --alias ``` This generates a new key pair and stores the files within the standard working config folder, `~/.iota/iota_config/` (in Unix). Executing the above command provides the key pair info as output, e.g.: ```bash ╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Created new keypair and saved it to keystore. │ ├─────────────────────────┬──────────────────────────────────────────────────────────────────────────────┤ │ alias │ │ │ address │ 0x3607bc87821a13d861e1e09cd5a2525b272f8dbc3e00556a8baf940a39120f32 │ │ publicBase64Key │ m6mBnT0tq9CA1zgdwy9y6ag2e8sWyzHVAoca+zLKJq8= │ │ publicBase64KeyWithFlag │ AJupgZ09LavQgNc4HcMvcumoNnvLFssx1QKHGvsyyiav │ │ keyScheme │ ed25519 │ │ recoveryPhrase │ glimpse hard relax method kid deliver nominee wait brief surprise speed pond │ ╰─────────────────────────┴──────────────────────────────────────────────────────────────────────────────╯ ``` ```bash iota keytool import "glimpse hard relax method kid deliver nominee wait brief surprise speed pond" ed25519 --alias ``` This imports an existing key pair using the given mnemonic and stores the files within the standard working config folder, `~/.iota/iota_config/` (in Unix). Executing the above command provides the key pair info as output, e.g.: ```bash ╭─────────────────────────┬──────────────────────────────────────────────────────────────────────╮ │ alias │ │ │ iotaAddress │ 0x3607bc87821a13d861e1e09cd5a2525b272f8dbc3e00556a8baf940a39120f32 │ │ publicBase64Key │ m6mBnT0tq9CA1zgdwy9y6ag2e8sWyzHVAoca+zLKJq8= │ │ publicBase64KeyWithFlag │ AJupgZ09LavQgNc4HcMvcumoNnvLFssx1QKHGvsyyiav │ │ keyScheme │ ed25519 │ │ flag │ 0 │ │ peerId │ 9ba9819d3d2dabd080d7381dc32f72e9a8367bcb16cb31d502871afb32ca26af │ ╰─────────────────────────┴──────────────────────────────────────────────────────────────────────╯ ``` ```bash iota keytool import-ledger --alias ``` This imports an existing public key from the connected Ledger hardware wallet and stores the reference to it within the standard working config folder, `~/.iota/iota_config/` (in Unix). Executing the above command provides the key pair info as output, e.g.: ```bash ╭─────────────────────────┬──────────────────────────────────────────────────────────────────────╮ │ alias │ │ │ iotaAddress │ 0x3607bc87821a13d861e1e09cd5a2525b272f8dbc3e00556a8baf940a39120f32 │ │ publicBase64Key │ m6mBnT0tq9CA1zgdwy9y6ag2e8sWyzHVAoca+zLKJq8= │ │ publicBase64KeyWithFlag │ AJupgZ09LavQgNc4HcMvcumoNnvLFssx1QKHGvsyyiav │ │ keyScheme │ ed25519 │ │ flag │ 0 │ │ peerId │ 9ba9819d3d2dabd080d7381dc32f72e9a8367bcb16cb31d502871afb32ca26af │ │ externalSource │ ledger │ │ derivationPath │ m/44'/4218'/0'/0'/0' │ ╰─────────────────────────┴──────────────────────────────────────────────────────────────────────╯ ``` - `alias`: A human-readable identifier to use within the CLI scope to refer to a key pair. - `address`: The public address representing the key pair. - `publicBase64Key`: The public key, base64 encoded. - `publicBase64KeyWithFlag`: The public key prefixed with the scheme flag byte, base64 encoded. - `keyScheme`: The cryptographic scheme used to derive the keypair; the `ed25519` is the standard scheme, used most of the time, while the `BLS12381` scheme is used for the `authority_key`. :::info From now on, this key pair will be referred to as `account-key`. ::: #### 1.3. Switch to the validator account key This makes that key pair active in the CLI. ```bash iota client switch --address ``` #### 1.4. Generate the validator data Finally, all validator's remaining key pairs and data can be generated using the following command: ```bash iota validator make-validator-info \ \ \ \ \ \ ``` - `name`: A human-readable validator name, e.g., `validator1`. - `description`: A human-readable validator description, e.g., `"this is a validator"`. - `image_url`: The validator image URL, e.g., `https://www.iota.org/favicon.png`. - `project_url`: The validator project URL, e.g., `https://www.iota.org`. - `host_name`: The host name that is used to generate the validator `network_address`, `p2p-address` and `primary_address`, e.g., `validator1.iota.org`. This command generates a `validator.info` file and up to 4 key pair files in the same directory where the command was executed. All keys but the `account.key` will need to be copied over to the validator node and included in the YAML configuration of the node. See [Validator Node Configuration](https://docs.iota.org/operator/validator-node/configuration). - `account.key` contains an ed25519 private key _(keep this one private)_. This file will be skipped if you used a Ledger hardware wallet. - `network.key` contains an ed25519 private key _(copy over to the validator node)_. - `authority.key` contains a BLS12381 private key _(copy over to the validator node)_. - `protocol.key`contains an ed25519 private key _(copy over to the validator node)_. - `validator.info` contains the validator information, e.g.: ```yaml --- info: name: validator1 account-address: "0x547b20ffca39cf1c9f57e7d1ff946d4720df48bb582e89b763b5d488ec23f5fa" authority-key: h93SKC2tFXMDt+lu4SRb3KA668/lJCPREHnzYZelaQ3iAu0RuiHqETBg/1jkV9HFCECvBCrzKjTuVD/bt5yUDon5nPKQmCyYEmx0NRQmxfP7Szpg17YZ388eT+aTnGEK protocol-key: Lm1Iy5KDV0qlMcGVnQNatAMLxhg8FOxE2q/QUkgLAYA= network-key: ADBhWCBOzqIvsDa9cowpSQ4t1nz+ZQYeRLBGQYe1Dy8= gas-price: 1000 commission-rate: 200 network-address: /dns/validator1.iota.org/tcp/8080/http p2p-address: /dns/validator1.iota.org/udp/8084 primary-address: /dns/validator1.iota.org/udp/8081 description: validator-description1 image-url: "https://www.iota.org/favicon.png" project-url: "https://www.iota.org" proof_of_possession: hpep8yY/JCj/zTEv9Ws7Qow3KO+2jrsX/yAUAykfxzS5vsg7vlwsscJpspf4XF/u ``` Where: - `name`: human-readable validator name, defined above; - `account-address`: The address derived from the `account key` public key. - `authority-key`: The public key derived from the `authority key` private key. - `protocol-key`: The public key derived from the `protocol key` private key. - `network-key`: The public key derived from the `network key` private key. - `gas-price`: An unsigned integer value indicating the gas price proposed for the first epoch. This is unused as of protocol version v5. - `commission-rate`: The validator commission rate, i.e., the fee charged by the validator for staking services; the default set at 2% (200). - `network-address`: The network address of the validator, defined using the `host_name`. - `p2p-address`: The address of the validator used for p2p activities such as state sync, defined using the `host_name`. - `primary-address`: The primary address of the validator used for the consensus, defined using the `host_name`. - `description`: A human-readable validator description, defined above. - `image-url`: The validator image URL, defined above. - `project-url`: The validator project URL, defined above. - `proof_of_possession`: A BLS signature created using the `authority-key`, committed over the following message: `intent || pubkey || address || epoch`. `intent` is serialized to `[5, 0, 0]` representing an intent with scope as "Proof of Possession", `version` as "V0" and `app_id` as "Iota". `pubkey` is the serialized public key bytes of the `authority-key`. `address` is the `account-address`. `epoch` is serialized to `[0, 0, 0, 0, 0, 0, 0, 0]`. :::note You may have noticed that port 8081 is using UDP, which conflicts with the firewall rules. This is a known bug, but the node is actually communicating via TCP. ::: #### 1.5. Register as a Validator Candidate Submit an on-chain transaction to become a validator candidate. The parameter is the file path to the `validator.info` generated by [Make Validator Info](#1-generate-validator-information). Make sure your validator account has enough IOTA to cover the gas costs. On the Testnet you can use the Faucet to receive the necessary funds. :::note Environments Make sure your client is [configured to use the correct environment](https://docs.iota.org/developer/getting-started/connect#connect-your-cli-to-a-network). ::: ```bash iota validator become-candidate ``` You can verify the status by running: ```bash iota validator display-metadata --json true ``` You should see your validator's status is `Candidate`. ## 2. Setup Validator Node :::info Note Perform these steps on the server that will be running the validator instance. ::: ### 2.1. Prepare the Environment ```bash sudo useradd iota sudo mkdir -p /opt/iota/bin sudo mkdir -p /opt/iota/config sudo mkdir -p /opt/iota/db sudo mkdir -p /opt/iota/key-pairs sudo chown -R iota:iota /opt/iota ``` ### 2.2. Build From Source First make sure to install all pre-requisites: ```bash sudo apt update && sudo apt install -y curl git-all cmake gcc libssl-dev pkg-config libclang-dev build-essential ``` Build and install the `iota-node` and `iota-tool` binaries from source. ```bash git clone https://github.com/iotaledger/iota.git && cd iota git checkout mainnet cargo build --release --bin iota-node --bin iota-tool mv ./target/release/iota-node /opt/iota/bin/iota-node mv ./target/release/iota-tool /opt/iota/bin/iota-tool ``` ```bash git clone https://github.com/iotaledger/iota.git && cd iota git checkout testnet cargo build --release --bin iota-node --bin iota-tool mv ./target/release/iota-node /opt/iota/bin/iota-node mv ./target/release/iota-tool /opt/iota/bin/iota-tool ``` ### 2.3. Create a `systemd` Service ```bash sudo tee /etc/systemd/system/iota-node.service > /dev/null < ```bash curl -fLJ -o validator.yaml https://github.com/iotaledger/iota/raw/refs/heads/develop/setups/validator/validator-mainnet.yaml ``` ```bash curl -fLJ -o validator.yaml https://github.com/iotaledger/iota/raw/refs/heads/develop/setups/validator/validator-testnet.yaml ``` ### 2.5. Configure the P2P Settings Configure the P2P settings in your `validator.yaml` file by following these steps: 1. Update the external address: ```yaml p2p-config: external-address: /dns//udp/8084 ``` :::note Replace `` with the hostname you used while generating the validator info. This address must be accessible from the internet. ::: :::note For better flexibility and faster disaster recovery, we suggest using a hostname with a very low TTL. This allows you to quickly relocate your validator to a new server if hardware fails. Direct IP addressing significantly complicates server migration. ::: ### 2.6. Download the Genesis Blobs ```bash cd /opt/iota/config ``` ```bash curl -fLJO https://dbfiles.mainnet.iota.cafe/genesis.blob curl -fLJO https://dbfiles.mainnet.iota.cafe/migration.blob ``` ```bash curl -fLJO https://dbfiles.testnet.iota.cafe/genesis.blob ``` ### 2.7. Copy the Validator Keys Copy the `protocol.key`, `network.key` and `authority.key` you created previously to the server and place them in `/opt/iota/key-pairs`. ### 2.8. Download the Latest Formal Snapshot ```bash cd /opt/iota ``` ```bash sudo -u iota /opt/iota/bin/iota-tool download-formal-snapshot --latest --genesis /opt/iota/config/genesis.blob --path /opt/iota/db/authorities_db --no-sign-request --verify normal --network mainnet --verbose ``` ```bash sudo -u iota /opt/iota/bin/iota-tool download-formal-snapshot --latest --genesis /opt/iota/config/genesis.blob --path /opt/iota/db/authorities_db --no-sign-request --verify normal --network testnet --verbose ``` ### 2.9. Start Your Validator Node ```bash sudo systemctl start iota-node ``` You can follow the node logs with: ```bash sudo journalctl -u iota-node -f ``` ## 3. Join the Validators :::info Key Security The following steps should be executed in the same environment you used while creating the validator keys. ::: Make sure your validator account created in [Generate Validator Information](https://docs.iota.org/operator/validator-node/systemd#1-generate-validator-information) has enough IOTA to cover the gas costs. :::note Tesnet Faucet On the Testnet you can use the Faucet to receive the necessary funds. ::: Before joining the validators, ensure you have enough delegated stake by running the following command: ```bash iota validator display-metadata --json true | grep stakingPoolIotaBalance ``` You need [at least 6M IOTA to join the validators](https://docs.iota.org/about-iota/tokenomics/validators-staking#validator-staking-pool-requirements). So make sure the `stakingPoolIotaBalance` is at least `6_000_000_000_000_000`. Once your have enough stake, submit your request to join the validators: ```bash iota validator join-validators ``` ## 4. Monitor Validator Status ```bash iota validator display-metadata --json true ``` You should see your validator's status is `Pending` now, it will become active and join the validators starting from next epoch. ``` 's validator status: Pending ``` You can also check the [Monitoring](./monitoring.mdx) about metrics endpoint. ## 5. State Sync Full Node (SSFN) Setup Follow the [State Sync Full Node (SSFN) Guide](https://docs.iota.org/operator/ssfn_guide). --- ## Troubleshooting(Validator-node) ## Validator Recovery In Case Of Database Corruption: By following these steps, you can quickly recover your validator node, minimizing downtime and maintaining your participation in the consensus. ### Stop the failed validator node ```sh sudo systemctl stop iota-node ``` ```sh docker compose stop ``` ### Delete the database folders There are two database folders to delete, specified in your `validator.yaml` under the keys `db-path:` and `consensus-config -> db-path:`. If you used default values for these folders, run this command to delete both folders at once: ```sh rm -rf /opt/iota/db/* ``` ```sh rm -rf /iotadb/* ``` If you customized them at setup, replace the previous paths with the ones you specified in `validator.yaml` under `db-path:` and `consensus-config -> db-path:`. ### Restore from formal snapshot Follow the steps in this document: https://docs.iota.org/operator/validator-node/configs/snapshots#restoring-a-full-node-from-snapshots ### Start the validator node ```sh sudo systemctl start iota-node ``` ```sh docker compose up -d ``` ### Let the validator sync all databases (including `consensus_db`) and monitor recovery progress: ```sh sudo journalctl -fu iota-node ``` ```sh docker compose logs -f ``` See also https://docs.iota.org/operator/validator-node/monitoring. --- ## Validator Tokenomics ## Validator Consensus and Voting Power The total voting power on IOTA is always 10,000, regardless of the amount staked. Therefore, the quorum threshold is 6,667. There is no limit to the amount of IOTA users can stake with a validator. Each validator has consensus voting power proportional to IOTA in its staking pool, with one exception: the voting power of an individual validator is capped at 1,000 (10% of the total). If a validator accumulates more than 10% of total stake, the validator's voting power remains fixed at 10%, and the remaining voting power is spread across the rest of the validator set. ### User Staking and Withdrawals When users stake IOTA tokens, these IOTA objects are wrapped into `StakedIOTA` objects. The calculation to determine each user's relative ownership of the staking pool is done directly with the timestamp of the `StakedIOTA` object (which determines the moment at which the deposit took place) and the change in the exchange rates between the deposit epoch and the withdrawal epoch. Each staking pool's data structure contains a time series with that pool's exchange rates. These exchange rates can be used to determine the withdrawals of any of the pool's stakers. Stake withdrawals are processed immediately with the exchange rate prevailing at the previous epoch's exchange rate. Withdrawals do not have to wait for the current epoch to close. Withdrawals include both the original stake the user deposited and all the stake rewards accumulated up to the previous epoch. Stakers do not earn the rewards accruing to their stake during the epoch at which they withdraw. Since there is no way to know how many stake rewards will be accumulated during the current epoch until the epoch closes, these cannot be included in the withdrawal. Hence, any user can withdraw their stake immediately and receive: IOTA withdrawn at E' = ( IOTA deposited at E ) * ( Exchange Rate at E'-1 / Exchange Rate at E ) ### Find the Exchange Rate Each epoch change emits a `0x3::validator_set::ValidatorEpochInfoEventV1` event per validator with the exchange rate information. You can use the Events API to query events. ## Staking Rewards Each epoch, within a given validator staking pool, all stakers are assigned the same proportion of rewards through the pool's exchange rate appreciation. Additionally, as validators earn commissions over the stake they manage, validators receive additional `StakedIOTA` objects at the end of each epoch in proportion to the amount of commissions their staking pool earns. Staking rewards are funded by stake subsidies released at the end of the epoch. The total stake subsidy per epoch is 767k IOTA, which is distributed among pools according to their voting power and the tallying rule. ### User Staking and Rewards A stake deposit request goes into a pending state immediately in the staking pool as soon as it is made. IOTA Wallet reflects any pending stake deposit requests for the user's account. However, pending stake deposit requests do not take effect until the end of the epoch during which the request is made. A withdrawal (un-stake) request is processed immediately as soon as it is received. The staker obtains the originally deposited IOTA together with all accrued stake rewards up to the previous epoch boundary – in other words, they do not include stake rewards for the current epoch. Users can't withdraw a portion of their active stake. They must withdraw all staked IOTA at the same time. Users can, however, stake using multiple `StakedIOTA` objects by splitting their IOTA into multiple coins. They can then perform a partial withdrawal from a validator by un-staking only some of the `StakedIOTA` objects. ## Reference Gas Price IOTA is designed such that end-users can expect the gas price to be stable and predictable during regular network operations. This is achieved by having validators set the network's reference gas price at the beginning of each epoch. Operationally, this is achieved through a gas price survey that occurs as follows: - During each epoch E, each validator submits what they think the optimal reference gas price should be for the next epoch E+1. - At the epoch boundary, when IOTA transitions from epoch E to epoch E+1, the network observes the gas price quotes across the validator set and sets the 2/3 percentile weighted by stake as the epoch's reference gas price. Hence, the reference gas price is constant throughout each epoch and is only updated when the epoch changes. For example, assume that there are seven validators with equal stake, and the price quotes they submit are `{15, 1, 4, 2, 8, 3, 23}`. The protocol sets the reference gas price at 8. In practice, the process for submitting a gas price quote for the Gas Price Survey is a straightforward one. Each validator owns an object that contains their quote for the reference gas price. To change their response, they must update the value in that object. For example, to set the price quote for the next epoch to 42, run: ```shell iota client call --package --module iota_system --function request_set_gas_price --args 0x5 "42" --gas-budget ``` Importantly, the gas object's value persists across epochs so that a validator who does not update and submit a new quote uses the same quote from the previous epoch. Hence, a validator seeking to optimize its own operations should update its quote every epoch in response to changes in network operations and market conditions. ## Validator Slashing and Tallying Rule IOTA is designed to encourage and enforce community monitoring of the validator set. This is done through the Tallying Rule by which each validator monitors and scores every other validator in order to ensure that everyone is operating efficiently and in the network's best interest. Validators that receive a low score can be penalized with slashed stake rewards. The protocol only computes the global Tallying Rule score at the epoch boundary and so relies on validators monitoring actively and changing their individual scores whenever they detect changes in other validator behavior. In general, the Tallying Rule default option should always be a score of one for all validators and only be changed to zero upon determining bad operations. In practice, the Tallying Rule consists of a set of objects each validator owns that default to scores of one and thus a validator will generally be passive and only update the object corresponding to another validator's score whenever needed. For example, to report a validator whose IOTA address is `0x44840a79dd5cf1f5efeff1379f5eece04c72db13512a2e31e8750f5176285446` as bad or non-performant, run: ```shell iota client call --package --module iota_system --function report_validator --args 0x5 0x44840a79dd5cf1f5efeff1379f5eece04c72db13512a2e31e8750f5176285446 --gas-budget ``` The Tallying Rule should be implemented through a social equilibrium. The validator set should actively monitor itself and if one validator is clearly non-performant, then the other validators should score that validator with a 0 and slash its rewards. Community members can launch public dashboards tracking validator performance and that can be used as further signal into a validator's operations. There is no limit on the number of validators that can receive a 0 tallying score in an epoch.