Joshua Sells

Crypto arbitrage system

Ended Jun 2023
When
Dec 2021 to Jun 2023
My role
Co-built with Max
Built with
JavaTypeScriptSoliditySQLNode.jsSpringweb3jweb3.jsethers.jsApache BeamDataflowBigQueryCloud RunCloud FunctionsCloud WorkflowsCloud SchedulerCloud BuildPub/SubSecret ManagerMongoDBPostgreSQLRabbitMQKubernetesDockerGitHub ActionsHardhatTruffleEthereumPolygonBNB ChainGethBesuTekuFlashbots
Links
GitHub orgLedgerContractsMath and papers

December 2021 to June 2023. A two-person project, built by my friend Max and me in the evenings and weekends alongside full-time jobs. The source is public at github.com/crypto-jambot.

The short version

Between December 2021 and June 2023 we built a system that watches decentralized exchanges on Ethereum, BNB Chain and Polygon, works out which cycle of token swaps would earn a profit after a given swap changes a pool's price, and executes it on chain. The system ran live. It found real mispricings and captured two of them as profitable trades, one on Polygon in July 2022 and one on Ethereum in April 2023. Most candidates were worth about what they cost in gas, which makes sense for an optimized system. This page is the full record: the story in order, each repository that came out of each step, every deployed contract, the research behind the math, and every trade we ever sent.

214,061 → 2,749

liquidity pools ingested, then kept after rule-based filtering (1.3%)

224,124

cyclic trade paths enumerated from 16 start tokens, up to four hops

2

profitable automated trades: Polygon (2022) and Ethereum (2023)

Three companion pages hold the material that would otherwise bury the story: the ledger lists every automated trade, the contracts page lists every deployment with its source, and the math page has the papers and the derivation.

Background

Everything below depends on three ideas: how a decentralized exchange sets prices, what a pool is, and what arbitrage means.

How an automated market maker works

A traditional exchange keeps an order book: a list of people who want to buy and sell at stated prices, matched by the exchange. A decentralized exchange on a blockchain, such as Uniswap, has no order book and no operator. It is a smart contract that holds two tokens and follows one rule, and anyone can trade against it at any time. That kind of exchange is called an automated market maker, or AMM.

The rule used by Uniswap V2 and the many exchanges copied from it is the constant-product rule. Say a contract holds 100 ETH and 200,000 USDC. Multiply the two quantities: 100 × 200,000 = 20,000,000. Every trade must leave that product unchanged (a small fee aside). The price comes from the ratio, 200,000 / 100, or 2,000 USDC per ETH.

Lets say someone buys 1 ETH. Afterwards, the pool is left with 99 ETH, so to keep the product at 20,000,000 it must hold 202,020.20 USDC (20,000,000 = 99 * x), which means the buyer pays 2,020.20 USDC. That is more than 2,000. Buy more and the price gets worse, because every ETH removed makes the next one scarcer. After that trade the pool prices ETH at 202,020.20 / 99, about 2,040.61 USDC. Two things follow. A bigger trade moves the price more (this is called price impact), and a trade always leaves the pool at a new price.

Why it is called a pool

The tokens in the contract come from people called liquidity providers. Each one deposits equal values of both tokens into a shared pot, the pool, and receives a claim on a share of it. Every trade pays a fee to the pool (0.3% on Uniswap V2), and the providers split those fees in proportion to their shares. Traders get a market that is always open. Providers earn a return for taking the risk of holding both tokens. "Liquidity" just means how much there is to trade against: a deep pool barely moves when you trade, and a shallow one moves a lot.

Anyone can create a pool for any pair of tokens. That is why there were 214,061 of them in this project's data, many holding almost nothing.

What arbitrage is

Arbitrage is buying low in one place and selling high in another, and keeping the difference. It exists wherever one thing has two prices.

On a blockchain, the same pair of tokens can trade in many pools at once, on different exchanges, and each pool prices them only from its own two balances (reserves). When someone makes a big trade in one pool, its price moves, but the others do not know. For a moment ETH might cost 2,040 USDC in one pool and 2,000 USDC in another. Anyone who buys in the low-priced pool and sells in the high-priced one profits, and both trades push the prices toward each other. The gap closes when you trade on the involved pools until they reach the same price again (And yes, there is an optimal amount you can calculate to bring the pools into perfect balance).

That is how AMM prices stay in line with the rest of the market. There is no central price feed. Pools are kept in balance by people who profit from the gaps. Arbitrageurs are paid for the correction, and if they weren't there, a pool's price would drift away from everyone else's.

Cycles, flash swaps and gas

The trade does not have to be between two pools. A cycle of three or four pools can also leave you with more than you started with: start with token A, trade it for B in one pool, B for C in another, C back to A in a third, and end with more A than you began with. Finding which cycles are profitable, and how much to put in, is the main problem this project tried to solve.

Two more pieces of vocabulary appear later. A flash swap (and its cousin the flash loan) lets a contract borrow tokens with no collateral, as long as it repays within the same transaction. If the trade turns out unprofitable, the whole transaction reverts as if it never happened, so the borrower needs no capital of their own. Gas is the fee paid to the network to run a transaction, and it is the cost that decides whether a small gap is worth trading at all.

Next ETH costs in A

$2,006

Next ETH sells for in B

$2,393

Gap

19.3%

Profit

$0

15002000250030003500050100150200250ETH tradedUSDC per ETHbuy in pool Asell in pool Bbest size
0.0 ETH
Figure B1. Two pools of ETH/USDC with 1,000 ETH each and 0.3% fees, priced $2,000 and $2,400. The gap is exaggerated so the curves show. The blue line is what the next ETH costs in the low-priced pool, and it rises faster the more you buy, because each purchase moves the pool's ratio. The amber line is what the next ETH sells for in the high-priced pool, and it falls the same way. The shaded area between them is your profit. It grows until the lines cross, and past that point each extra ETH loses money. Prices include the fee.

If you would like to see these ideas explained visually, Whiteboard Crypto has two short videos: What is an Automated Market Maker? and What is a Liquidity Pool in Crypto?. Both were part of how I learned this.

0. The trade

On April 1st, 2023 at 18:37 UTC, a transaction landed in block 16,956,123 at position six: 0xf20fe22d…49ac8a. It was the best trade the system ever made.

The transaction borrowed 0.010847 WETH from one Uniswap V2 pool without paying for it yet, which is what a flash swap is. It pushed 0.00000089 WETH, about a sixth of a cent, into a second pool. That pool returned a token called HOE, which went to a third pool, which returned a token called H, which went to a fourth pool. The fourth pool paid out 19.764018 USDT, which went back to the first pool as repayment. The first pool checked its books, found them balanced, and let the transaction finish. We kept 0.010846 WETH, worth $19.68. The wallet paid $17.55 in gas. Net: $2.13.

Pool AWETH / USDTJamBot V2our contractPool BWETH / HOEPool CHOE / HPool DH / USDT

Kept, 0.010846 WETH

$19.68

Paid in gas

−$17.55

Net

$2.13

Figure 1. The transaction behind the $2.13, decoded from its logs: 0xf20fe22d…49ac8a. All four pools are Uniswap V2. Click a step to jump to it.

The reason it worked is less flattering than the trade looks. The fourth pool held 19.7656 USDT and 0.00000023 of its other token. At that ratio the pool priced its token at roughly 87 million USDT, and a tiny input drained 99.99% of its USDT. The size of the profit was set by what a dust pool happened to hold, not by the depth of any market. Gas took 89% of the gross. We found a real mispricing, captured it, and kept 11% of it.

The rest of this page is how the system that found it came to exist.

1. The spark

My high school friends introduced me to what they called DeFi Summer 2.0, in the summer of 2021. It was a period when some outrageous "game theory" projects in crypto rose and fell. I was a programmer, and this was my first exposure to the blockchain world. It was the wild west, full of some pretty silly projects. I made some money during that time, by pure luck, and eventually came across the idea of using my programming skills to arbitrage real-time price discrepancies in the market. DeFi Summer got me in, but arbitrage is where I saw potential. I found a tutorial about it from an online crypto creator.

The first code I played with was a modified script from a Dapp University course. It compared prices for a token pair on 0x and 1inch and, if a profit looked possible, borrowed from dYdX and traded. It was a good way to learn how flash loans work, and it had one obvious flaw: everyone who took the course could run the same code and see the same trade at the same moment. I remember thinking that if I widened the search space, I could find opportunities nobody else was looking at. Instead of two tokens and two venues, why not look at every pool on a chain? I was able to bring the systems thinking from my work at Best Buy to this problem.

  • zzz-arb-bot-dydx-1inch-0xNot publishedThe first script: a modified course bot that compared 0x with 1inch and borrowed from dYdX. Based on paid course code, so it stays private.

2. Iteration 1: look at everything (May to November 2022)

The first version of "everything" was all about gathering data. A BigQuery collector pulled every pair from every V2-style exchange. A second job added token names and decimals. A listener watched swaps, and a depth-first search tried to build paths across all tokens. Two bots sat on top of it: one that traded a pair of pools, and one that evaluated paths of any length.

The first lesson arrived quickly: an unfiltered search space is the enemy. BNB Chain alone had about a million pairs, and one start token on Polygon produced 13,127,053 paths occupying 8.8 GB. Almost none of them could ever be profitable, and I had no way of knowing which. I had built a machine that was very good at producing candidates and had no idea how to rank them.

I deployed contracts on all three chains in July 2022: Ethereum on the 17th, BNB Chain on the 22nd and Polygon on the 24th. Ethereum used a dYdX flash loan. dYdX only exists on Ethereum, so the other two chains used DODO flash loans. Both versions swapped through the exchanges' routers.

The BNB Chain contract was deployed at 01:22 UTC on July 22nd (deploy) and made its first trade that evening, a loss of about $1.18. The Polygon contract was deployed at 03:01 UTC on the 24th (deploy). At 04:15 it made the first profitable trade of the project: 0x6f9c65c5…817dc2. It borrowed 0.176017 WPOL from DODO, sold it on QuickSwap for 5.2 million units of a token called IFA888, sold those on ApeSwap for 0.191624 WPOL, repaid the loan and kept 0.015607. After gas the net was 0.00695 POL, about half a cent.

DODO poolflash loanOur contractarbitrage, DODOQuickSwapWPOL / IFA888ApeSwapIFA888 / WPOLOur walletprofit

Gross profit, 0.015607 WPOL

0.015607

Paid in gas

−0.008656

Net, in POL

0.00695

Figure 2. The first profitable trade of the project, decoded from its logs: 0x6f9c65c5…817dc2. Both swaps are on Uniswap V2-style pools, one on QuickSwap and one on ApeSwap. Click a step to jump to it.

That win was one of nine successful calls on Polygon. The other eight lost small amounts, and a second round in late October brought five reverted transactions. During this iteration of the project, every transaction went through a node into the public mempool, and a transaction that reverts still pays for the gas it burned (we found a solution for this later in the project).

The reasons I stopped adding to this version were practical. The economics were poor, with tiny spreads against gas and no filtering. The design also did not scale: it was smothered by the sheer number of paths, which grows exponentially. By the end of November the repositories carried a note that they were not production ready and were kept for reference.

That Polygon transaction made me half a cent, and I felt ON TOP OF THE WORLD. It was my proof of concept, and it gave me more fire to continue. I could point at a transaction and prove that software I wrote had found money and taken it. Crazy.

3. A collaborator and the cloud (November 2022)

As I was thinking about next steps, my programmer friend Max joined with his first commit on November 15th, 2022. The project moved to a GitHub organization and onto Google Cloud, and the work split along the lines of what each of us was strong in. We revamped everything and built the platform from a clean slate: shared npm libraries, a local Kubernetes cluster, the data warehouse that staged the trade paths, continuous integration, Cloud Run services, a Dataflow deployment framework, a local Ethereum node, and the arbitrage service itself.

The libraries are small and boring on purpose. They exist so that every later service gets secrets, BigQuery and the node the same way.

4. Iteration 2: the offline pipeline (November 2022 to May 2023)

The central idea of the second iteration was that the bot should never search at runtime. Everything expensive would happen in a daily batch, and the live system would be a lookup.

The batch has five stages:

  • Ingest. Read pool-creation events from about 25 exchanges and write them to BigQuery, with a per-exchange cursor so every run is incremental.
  • Enrich. Add token metadata: name, symbol and decimals.
  • Filter. Remove pools with empty reserves and pools that cannot be part of any cycle, repeating until nothing changes. A rule-based Apache Beam pipeline cuts 214,061 pools to 2,749.
  • Enumerate. Run a depth-first search from 16 start tokens, up to four hops, to produce 224,124 cycles.
  • Index. Store the cycles as prefix trees keyed by the pool whose swap would trigger them, so a swap in pool P needs only a single key lookup to find P's tree.

The start tokens began as three: WETH, USDC and DAI, the tokens a dYdX flash loan could lend. Once our design no longer required a flash loan (used a flash swap instead), the list grew to 16. The depth limit of four was a judgement call. We expected most extractable value to sit within four hops and did not want the path count to grow without bound.

A Cloud Workflow ran the whole thing daily, fanning out per exchange, polling the Dataflow jobs, and swapping the new trees in behind a blue-green rename of the MongoDB collection so that the bot never saw a half-built index.

  1. Cloud Scheduler runs crypto-workflows daily: fan out per exchange, start each Dataflow job, poll it, swap the trees in.

  2. 1 · Ingest 25 exchanges

    bigquery-pool-ingestor

    ↓ pools

     

    V2 200,903 · V3 13,158

  3. 2 · Enrich batched eth_call

    bigquery-token-enricher

    ↓ tokens

     

    name, symbol, decimals

  4. 3 · Filter rules to a fixed point

    pool-refiner

    ↓ refined_pools

     

    from 10,730 active

  5. 4 · Enumerate DFS, depth 4

    path-resolver

    ↓ paths

     

    from 16 start tokens

  6. 5 · Index prefix trees

    tree-builder

    ↓ MongoDB trees

     

    swapped in blue-green

Figure 3. The daily batch. Each stage reads the table before it and writes the next; arrows down are writes, arrows up are reads. Only pools that swapped in the last 90 days (10,730) reach the filter, and its rules cut them to 2,749, about 1.3% of everything ingested. The cycles then outnumber the pools they come from. Each stage links to its repository.

The trees went through three homes in one month. They started in PostgreSQL as a table of path arrays on February 16th, moved to a MongoDB instance on our own hardware on February 28th at 01:43 UTC, and moved to MongoDB Atlas about ten hours later at 12:04. Running a database on the same machine as the node put load on the thing the bot depended on most, and a document store fits trees naturally. The earlier loader from Cloud Storage (zzz-mongo-loader) was dropped when the tree builder started writing directly.

5. Running our own node (November 2022 to June 2023)

The runtime needed a node it could trust to be fast and current, so we ran our own on a repurposed gaming tower: Ubuntu, microk8s, an execution client and a consensus client. We started with Besu and Teku and moved to Geth in the winter of 2023, between January 22nd and February 21st. Besu had runtime problems: it was not especially fast and it sometimes crashed or fell into an odd state. Geth performed much better.

Reaching the node from the cloud went through three designs. The first was a TLS reverse proxy that turned a custom header into a bearer token. The second was a small token service that checked an API key and signed a one-hour RS256 token, with client libraries that fetched, cached and attached it. After the move to Geth, the node's HTTP endpoint stopped enforcing the token. That was a deliberate trade-off: we opened that one endpoint to keep moving, exposed only standard JSON-RPC calls, advertised it nowhere, and kept the tower itself gated.

The token layer did not die with Besu. A freshness guard on Cloud Run used the same tokens to call a REST gateway to the tower's Kubernetes API and scale the listener to zero whenever the node was behind a commercial provider. A stale block height only ever produces stale trades.

A black gaming tower with a glass side panel, lit blue and green, showing an AMD processor cooler, a Gigabyte graphics card and a heat-sinked memory kit.
The tower the node ran on, a repurposed gaming PC.

6. The runtime (February to June 2023)

With the pipeline keeping the data fresh and a node of our own to ask, the last piece was a way to notice a swap the moment it happens. A swap moves a pool's price, and that can open a trade through any of the paths that pass through that pool. So the runtime listens for swaps, finds the paths a swap affects, and has the bot evaluate and execute them.

Two decisions keep the load down. The listener subscribes on the node only to swaps in the pools that survived the pipeline's filter, so the node does the filtering and nothing irrelevant ever reaches our code. And each event carries a direction flag, deliberately inverted, because after a swap one way the profitable paths run through that pool the other way.

How an event reaches the bot

listener → RabbitMQ → event filter → Pub/Sub → bot

  • geth-ws-listener watches our node for those swaps and puts {pool, direction, timestamp} on a RabbitMQ queue. It re-opens its subscription every 59 minutes. It replaced eth-log-listener, Max's first draft.
  • RabbitMQ runs on the home server next to the listener and the filter, so volume on this hop costs nothing.
  • event-filtering-service is a small Spring Boot service. It remembers when each pool last got through and drops any message for a pool that passed in the last 5 seconds, so the bot never repeats the same maths. It sends {pool, direction} for the rest.
  • Google Pub/Sub pushes each surviving message as an HTTP request to the bot on Cloud Run. It is the one paid hop, so it carries only events that survived de-duplication.
  • jam-bot reads the pool and direction, loads that pool's tree from MongoDB Atlas itself, and starts evaluating paths.

The rule behind the design is to de-duplicate before the paid hop, because Pub/Sub charges per message.

7. The math

On every swap event, the bot takes the pool's tree, evaluates every path in it, and decides how much to trade on the best ones. For a cycle of two pools, the optimal size has a closed form: it is the amount at which the two pools' prices are equal after the trade. A longer path can be collapsed, hop by hop, into one virtual pool, which brings us back to two. The idea comes from a paper on cyclic arbitrage, and we worked it through by hand on paper before writing any code.

Uniswap V3 was the hard part. Its liquidity is concentrated in price ranges, so inside a range a pool behaves like a V2 pool with larger virtual reserves, and at a range boundary the behavior changes. The optimizer swaps up to the next initialized tick, updates the pool's state, solves again, and repeats. The off-chain math has to agree with the contract down to the last unit, so we ported the V3 swap math to integer arithmetic in Java: about 1,100 lines in five commits, between May 17th and June 3rd, 2023.

  1. 1Is it worth trying?

    L120

    The paper

    Profit exists only if the product of the exchange rates around the cycle, net of fees, is greater than 1.

    The Java

    leftOperandNumerator = leftOperandNumerator.multiply(reserve1);
leftOperandDenominator = leftOperandDenominator.multiply(reserve0);
multipliedInputFees = multipliedInputFees.multiply(feeToPercent(trade.getPool().getFee()));
multipliedOutputFees = multipliedOutputFees.multiply(R2);
// …
return leftOperand.compareTo(rightOperand) > 0;

  2. 2Start with the first pool

    L171–173

    The paper

    a′1,n = a1,2,   a′n,1 = a2,1

    The Java

    orderedReserves = trades.get(0).getOrderedReserves();
BigDecimal inputReserve = new BigDecimal(orderedReserves.x());
BigDecimal outputReserve = new BigDecimal(orderedReserves.y());

    inputReserve is a′₁,ₙ and outputReserve is a′ₙ,₁.

  3. 3Fold in each next pool

    L175–188

    The paper

    a′1,n ← a′1,n·ai,i+1 / (ai,i+1 + r1r2·a′n,1)

    a′n,1 ← r1r2·a′n,1·ai+1,i / (ai,i+1 + r1r2·a′n,1)

    The Java

    for (int i = 1; i < trades.size(); i++) {
    BigDecimal R1 = feeToPercent(trades.get(i).getPool().getFee());
    // …
    BigDecimal iRNumerator = inputReserve.multiply(ithInputReserve);
    BigDecimal oRNumerator = R1.multiply(R2).multiply(outputReserve).multiply(ithOutputReserve);
    BigDecimal rDenominator = ithInputReserve.add(R1.multiply(R2).multiply(outputReserve));

    inputReserve = iRNumerator.divide(rDenominator, MathContext.DECIMAL128);
    outputReserve = oRNumerator.divide(rDenominator, MathContext.DECIMAL128);
}

  4. 4Merge the closing pool

    L175–188

    The paper

    The same formula, applied to the pool that closes the cycle, leaves one pool with reserves a and a′.

    The Java

    // No separate step: the loop above runs
// through the last pool as well.

  5. 5Solve for the best input

    L189–190

    The paper

    δop = (√(r1r2·a·a′) − a) / r1

    The Java

    BigDecimal numerator = feeToPercent(fee).multiply(R2).multiply(outputReserve).multiply(inputReserve).sqrt(MathContext.DECIMAL128).subtract(inputReserve);
return numerator.divide(feeToPercent(fee), MathContext.DECIMAL128);

    Here a is inputReserve and a′ is outputReserve.

  6. 6Not in the paper: V3 ticks

    L197–248

    The paper

    The paper's pools are constant-product, with no price ranges.

    The Java

    while (true) {
    BigInteger inputAmountToEarliestTick = null;
    populateInputAndOutputTradeAmountsNoTicks(optimalInput);
    // … find the earliest tick any trade in the path reaches
    if (inputAmountToEarliestTick == null) {
        optimalTradingVolume = optimalTradingVolume.add(optimalInput);
        return;
    }
    optimalTradingVolume = optimalTradingVolume.add(inputAmountToEarliestTick);
    swap(inputAmountToEarliestTick);
    if (!isValueExtractable()) return;
    optimalInput = getOptimalAmountFn.get().toBigInteger();
    if (optimalInput.equals(ZERO)) return;
}

    Swap up to the nearest tick, update every pool, check the path is still worth it, solve again, and repeat until no tick is in the way.

Figure 4. From the cyclic arbitrage paper to the bot. Each step of the paper's algorithm, next to the lines of Path.java that implement it, with some lines skipped. Here r1 is a pool's fee factor and r2 is the constant R2.

The papers we studied, the derivation and the V3 extension are on the math page.

8. The contracts

There were five generations of executor. Each was cheaper to run than the one before, because we kept moving work off chain.

GenDeployedDesignResult
0nonedYdX flash loan, 1inch and 0x, logic in the JavaScript botprototype only
1Jul 2022Ethereum: dYdX loan. BNB Chain and Polygon: DODO loan. Two routers eachone Polygon win, one BNB Chain loss
2Mar 21st, 2023dYdX loan. The bot chose the first amount and the contract worked out every later hop on chain from live reserves, swapping directly between poolsdeployed, never called
3Mar 28th, 2023JamBot V2: a thin executor that replays a list of calls built off chain. No loan: flash swaps12 trades, one profit (+$2.13)
4Jun 3rd, 2023JamBot V3: the same executor plus Uniswap V3 callbacks and about 20 V2-fork callbacks4 trades

The final contract is 194 lines. The bot builds a list of targets and payloads off chain. The contract replays them inside the pools' callbacks and checks one thing at the end: the balance of the starting token did not fall. One contract works with any exchange, and because the first pool's flash swap supplies the capital, there is no lender and no loan fee. Before anything is sent, the bot simulates the transaction against the real contract with eth_estimateGas. It only succeeds if that balance check passes, so a revert means we drop the candidate. The bot then subtracts the simulated gas cost from its own expected profit and sends only if something is left.

It makes one deliberate trade-off. The callbacks accept any caller while the lock is open, rather than checking that the caller is the pool we called, because the check costs gas on every hop. The contract holds no borrowed capital, and the final check requires the starting-token balance to rise, so the worst a malicious token could do is take one trade's profit.

Every deployed contract, with its address, deploy transaction and source, is on the contracts page.

9. Getting a transaction to land

Until March 2023, every transaction went to our own node, which broadcast it to the public mempool. From March we sent them to block builders instead, the design behind proposer-builder separation and the one Flashbots made popular. A builder assembles whole blocks and takes bundles of transactions directly from searchers like us, so a bundle never has to pass through the public mempool.

Our first plan for this was bread-catcher, a small separate service that Max wrote between March 21st and 28th. The bot would publish a finished trade to Pub/Sub, and bread-catcher would pick it up, sign it with the deployer wallet and send it privately to one relay, Flashbots, for the next block. The idea was to keep finding trades and sending them as two separate jobs. It was never deployed. On March 30th we moved submission into the bot itself, which signs the transaction and sends it to a dozen builders at once, targeting the next two blocks.

This had three practical consequences:

  • Failed attempts are free. A bundle that fails is simply not included, so nothing is paid and nothing shows up on chain.
  • Bundles are atomic and ordered. Everything in a bundle lands together, in the order we chose, or none of it lands.
  • No sandwiching. A private transaction cannot be front-run and back-run by someone watching the public mempool.

The chain data fits. Six of our 2022 trades failed, and none of the sixteen trades in 2023 did. Those sixteen landed in blocks with eleven different fee recipients, which is what sending to many builders looks like. Other searchers were finding the same opportunities, so we bid 99% of the profit per unit of gas, on purpose.

Public mempool · until March 2023

  1. 1Anyone watching the mempool sees the trade before it lands.
  2. 2They can copy it, bid higher, or sandwich it.
  3. 3A failed transaction is still included, and still pays for gas.

Block builders · from March 2023

  1. 1The bundle goes straight to builders, never through the public mempool.
  2. 2It is included whole, in the order we chose, or not at all.
  3. 3A failed bundle is dropped and costs nothing.
Figure 5. The same trade sent two ways. Through the public mempool it can be copied, outbid or sandwiched, and a failure still costs gas. As a bundle sent to builders it is included whole or not at all. From March 2023 every trade went the second way.

10. What the chain says

Every figure in this section comes from the chain, and each transaction can be checked. Every trade is listed on the ledger.

ChainContractTrade callsProfitableBest
Ethereum 2023JamBot V2 and V3161+$2.13 (gross $19.68, gas $17.55)
Ethereum 2022dYdX Arbitrage (manual calls)30no net movement
Polygon 2022DODO Arbitrage14 (9 succeeded, 5 reverted)1+0.00695 POL
BNB Chain 2022DODO Arbitrage2 (1 succeeded, 1 reverted)0−0.00127 WBNB
Ethereum 2023 totals over the 16 trades: gross $233.94, gas $338.42, net −$104.49. Most of the losing rows were deliberate tuning and debug runs, including three trades on June 6th with the profit gate switched off to test the path end to end.

The bot also logged its own candidates. On June 5th and 6th, 2023 it recorded 145 opportunities. The median gross was 0.0042 ETH, against a typical gas cost of about 0.0048 ETH. The system found real mispricings regularly, and the typical one was worth a little less than it cost to take.

On June 3rd, 2023 we cashed out the second contract in six transactions, then deployed the third 42 minutes later. The withdrawals sent 2,492,962 SHIB to each of the two owners (transaction) and 0.0769 WETH to the deployer wallet (transaction).

11. Operations under load

Running the bot on Cloud Run was largely a black box. Memory problems appeared and we tried different settings: the JVM heap was raised on April 6th, and on June 4th and 5th we moved to 8 GB and 6 CPUs, tuned garbage collection, cut concurrency, and stopped sharing pool state across requests. We never settled whether the cause was a leak, too little memory, the design, or all three. Tuning a system like this is hard, because every change has to be deployed and watched under live traffic before you learn whether it helped. We ultimately decided to stop the project before taking the Cloud Run tuning any further.

12. Results, and stopping

We stopped in June 2023, and the last commits are from June 5th to 7th. The reason was what the opportunities looked like. Either other searchers were competing for them, and the bidding pushed nearly all the profit to the block builder, or the gap was too small to cover gas in the first place. That told us the places we were searching were already well optimized by the market, and finding profit would take much more effort than we had been putting in. The effort had outgrown the upside.

What we would do differently today:

  • Curate the data more aggressively. That is what the unfinished token screener was for.
  • Widen the search. Try other chains and other trade shapes, not only cycles from a few start tokens, and support more protocols, in particular Balancer and Curve.
  • Listen to the mempool. We never watched pending transactions, on our own node or a provider's. We only reacted to swaps after a block had landed. Seeing a swap before it is included would let us search before the block instead of after it.
  • Add centralized exchanges. Compare pool prices with the prices on exchanges like Binance. That would open up a much larger search space, because an opportunity could be the gap between a pool and an order book, not only between pools. The exchange leg cannot settle atomically with the on-chain leg, so the risk would be different.
  • Arbitrage all the pools of a pair at once. Instead of looking for a cycle, buy where a token is priced low and sell where it is priced high, across any number of pools, and solve for the price at which they all balance. That is a convex optimization problem, and it was going to be our next big iteration.
  • Write the hot path in Rust, for lower latency and no garbage-collection pauses.
  • Study the block builders and host our bot geographically close to one.

Overall, this was one of the most intellectually challenging and rewarding projects I have worked on. My goal was to make a profit that could jump-start a future business. I didn't get that, but I did build something that found and captured money for me.

I picked up a wide range of new skills and sharpened the ones I already had. I took on tough data structures, implemented complex trading math, worked at the bit level in Java and wrote thread-local state. I put algorithms from college to real use, like depth-first search, and I learned how decentralized systems work. The list goes on.

It also taught me that sometimes all you need to know is the next right step, not the whole picture. I had no idea when I started how much I would be exposed to, and that naivety is what kept me going and kept me from losing hope.

I stopped after taking a healthy look at the time and effort it required. A full-time job and a family were among my other responsibilities, and something had to go. I still look back on it as some of my proudest work.

Most of all, I am grateful that I got to build this with a good friend. Partnering with Max towards a shared goal was a lot of fun, and it made the hard parts worth it.

Repository map

Every public repository behind the project. All but one are in the crypto-jambot organization. The first group is the final system.

These are the earlier versions, drafts and side tools that led up to it.

Glossary

  • AMM (automated market maker). A smart contract that holds two tokens and trades them at a price set by a formula instead of an order book. In a Uniswap V2 pool the two reserves multiplied together stay constant, so every trade moves the price.
  • Flash loan. A loan that is borrowed and repaid inside a single transaction. If it is not repaid, the whole transaction fails as if it never happened.
  • Flash swap. Uniswap's version of the same idea. The pool sends you the tokens first and checks at the end that it has been paid. We used it in place of a lender.
  • Mempool. The public waiting room for transactions that have been sent but not yet included in a block. Anyone can read it.
  • Block builder. A service that assembles the contents of a block and sells the space, including to searchers like us.
  • Bundle. A group of transactions sent privately to a block builder, to be included together and in order, or not at all.
  • Tick. In Uniswap V3, the price axis is cut into steps called ticks, and liquidity sits in ranges between them. A large trade can cross a tick, and the liquidity changes when it does.

References

  • Cyclic Arbitrage in Decentralized Exchanges

    Y. Wang, Y. Chen, H. Wu et al. · 2021 (arXiv 2105.02784, v3 Jan 2022)

    Studied closely. It is the source of the idea that a chain of pools can be collapsed into one virtual pool with a closed-form optimal trade. The bot's `Path` class cites it.

  • Uniswap v3 Core

    H. Adams, N. Zinsmeister, M. Salem, R. Keefer, D. Robinson · 2021

    With the v3-core contracts, the source for the off-chain V3 swap reimplementation.

  • Liquidity Math in Uniswap V3

    A. Elsts · 2021

    Companion reference for V3 liquidity and tick math.

  • An Analysis of Uniswap Markets

    G. Angeris, H.-T. Kao, R. Chiang, C. Noyes, T. Chitra · 2019 (arXiv 1911.03380)

    Background on constant-product market behavior.