top of page
Screenshot 2022-11-24 at 12.55.19 am.png

Unbiased DeFi Smart Order Router & Search Engine for DeFi 

Eta X is an open-source initiative to build an agnostic price discovery engine, smart order router (SOR) framework for decentralized exchanges (DEXs), and the growing list of liquidity sources in decentralized finance (DeFi). 

Contributors

1_ybm25-H2qzNEow-mUhthAQ-removebg-preview.png
Deeplink_Logo_white-removebg-preview.png
7b17b3_0e3ffd94789140c0867f5f71ab636312_mv2.webp
7b17b3_77eec7a5c77e456abc9c222b9b01eb7d_mv2.webp
7b17b3_f12499b84ddf40c4ace93ab721901fb7_mv2.webp
About

 

Flight aggregators like Expedia and Google Flights made it easier for us to find the best flight rate and fastest route instead of having to search each airline website one by one. Eta X acts like a search engine for DEX and DeFi, providing a single access point (Eta X APIs), allowing anyone to query the most efficient trade routes, discover better prices, match trade pairs and guesstimate slippage and price impact without attaching to a token or liquid pool. 

The Immediate Goals

  • Introduce efficient execution capabilities inspired by traditional finance order-matching systems.

  • Modern data aggregation architecture and ETL processes (L3 Atom & SuplerCluster integration).

  • Maintain an off-chain machine learning environment to detect frontrunning and failed transactions and simulate real-time price discovery, volatility, liquidity and price impact to improve the Eta X core functions.

What is Expected to be Developed?

  • Free API gateway to access Eta X core functions and query data—best prices, efficient trade routes and pairs, volatility and liquidity-related data, market activities, order matching, and real-time slippage estimates. The metadata and algos can be connected to CEXs, DEXs and smart contracts for execution. Over time, the off-chain machine learning environment will monitor, test, train and improve Eta X core SOR and price aggregation capabilities.

The Vision

DeFi space is moving fast. As the number of trading venues and LP pools grows, we will likely see the continued development of aggregators and smart order routers as the technology behind them becomes outdated. Instead of making another aggregator, we aim to introduce standardized frameworks like FIX, unbiased superior price discovery and route optimization engines for DEXs and the DeFi ecosystem without attaching its core functions to a token or an LP mechanism. 

We’ll likely see smart aggregators like Eta X be integrated into dApps and protocols as the APIs for DEX aggregators become the common engine amongst traders, exchanges, protocols and LPs to leverage its data and capital efficiency capabilities. 

Our vision is to accelerate DeFi adoption. All Eta X core functions and APIs will be free and open for everyone. As an open initiative, Eta X will be maintained by a decentralised community of supporters and users. In the coming weeks, collectively, we will publish all the methodologies, data, algorithms and back office operations of Eta X with the developer community who can use it to build smart DeFi applications. 


We believe DEXs and DeFi will play a significant role in crypto and Web 3.
 

Contributors & their Roles

L3Atom
  • Introduce modern data architecture and main data supply to Eta X

  • A well-thought-out scalable universal data collector framework (standardised code base, properties, schema) for CEXs, DEXs, blockchains and LP pools

  • Provide wider data coverage and access to historical and real-time market data.

  • Development of Supercluster data supply, data custody and ingestion services

  • Data ingestion accuracy, validity and reliability framework

  • Universal data collector framework (standardised code base)

  • Data coverage

  1. Orderbook, volume, exchange deposits & withdrawals, net flows, open interest, fees, liquidations & exchange administrative data

  2. On-chain - Level 1 - TVL (Total Value Locked, yield, liquidity providers volume & participants, loans outstanding, interest rates, deposits, and withdrawals, ROL (Return on liquidity), net flows, fees & protocol metadata

  3. On-chain - Level 2 - Supply changes, keys and ownership, UTXO data, gas fees, The Spent Output Profit Ratio (SOPR), Stock-to-Flow Ratio

  4. On-chain - Level 3 - Wallets, addresses data, blocks data, mining activities, mempool, network transactions

Deeplink
  • Eta X Project initiation, planning, roadmap design, stakeholders engagement and management, product design and development

  • Introducing data representation model for Eta X core, the off-chain machine learning environment.

  • Involved in the design & implementation of off-chain  RL/ML models

  • Data selection and classification

  • Machine learning models & schemes evaluation & selection

  • Data cleansing and data validation, data feature engineering and implementation

GDA Fund
  • Introducing well-established SOR technologies, order matching expertise, best execution practise, FIX frameworks from traditional finance and deep crypto and traditional finance industry 

  • Universal data collector frameworks: standardised data feeds for use in SOR models

  • DevOps, MLOps & computation setups

MIT
  • MIT will be formally assisting with researching towards large-scale graph processing frameworks and traversal algorithms, in particular, the application of ant colony optimization (ACO) algorithm based routing techniques. MIT will also be providing several research collaborators.

SingularityNet
  • SingularityNet will provide crucial machine learning expertise.

  • Data selection and classification

  • Machine learning models & schemes evaluation & selection

  • Involved in the design & implementation off-chain  RL/ML models.

Chainlink
  • Chainlink will provide their oracle and keeper services to connect any off-chain computations and data into Ethereum smart contract environments. This will likely play a crucial role in phases 3-4 of the project when the system begins transitioning into on-chain execution.

Sigmaprime
  • Sigma prime will conduct relevant research towards L2 solutions and sidechain integration to bring the system further on-chain.

  • This research into rollups, sidechains, and validator systems will assist in facilitating a Deeplink-native ecosystem for on-chain machine learning.

The Architecture

0.1.015.jpeg

Basics

 

What does Eta (η) mean?

 

η - Lowercase eta, the seventh letter of the modern Greek alphabet.


Eta η is used to represent many things across many fields, but a recurring theme in these representations is efficiency. 

  • Economics

    • η is elasticity, a measure of one economic variable’s change in response to another.

  • Mathematics, 

    • η-conversion, an extensionality concept in which two functions are equal if and only if they give the same results for all arguments.

  • Aeronautics

    • η propulsive efficiency, the percentage of chemical energy converted to kinetic energy.

  • Deep learning

    • η refers to the learning rate, the size of steps in the minimizing functions of backpropagation and stochastic gradient descent.

  • Electronics

    • η can represent the ideality factor of an electrical component, or how close it is to a mathematically perfect example of this component.

    • In power electronics, η represents the efficiency of a power supply (input/output).

  • Statistics

    • η in applied statistics refers to the partial regression coefficient, a variable used in partial regression, a technique used to show the impact of adding variables to models which already have one or more independent variables

DEX & Aggregators

Full documentation and references

 

What is a DEX?

A decentralized exchange (DEX) is a cryptocurrency exchange that operates without the involvement of a central authority or third party.

DEX Aggregators and the DeFi Boom

 

DEX aggregators are a somewhat newer concept that arose from the rise in DEXs. They have become more important to users in the recent DeFi boom as more and more users prioritize better prices on their trades. 

 

Evolution of DEX Aggregators

  • The first DEX aggregator was created as part of a hackathon competition in 2019. At the time, most participants felt that aggregated information from various decentralized exchanges was necessary. Sergej Kunz and Anton Bukov were the creators of the first DEX aggregator in the span of 18 hours during a hackathon which later served as the foundation for today’s — 1inch. Even though the first version was not completely functional then — it set an example of why DEX aggregators were necessary as a blockchain-based service.
     

  • Since then, the era of DEX aggregators began and has since continued to rise. With the crypto and blockchain revolution in personal and enterprise finance, DEX aggregators hold great value allowing investors — to make more informed decisions on their trades and swaps than ever before. Currently, these aggregators are most popular among the high-volume traders, while the retailers continue to manage things with DEXs directly.

 

DEX vs DEX Aggregator

  • Decentralized exchanges (DEXs) have seen a significant adoption since their inception. Even though DEXs and DEX aggregators may appear similar, they are quite different from each other. Even though both are built on blockchain technology — their functions are distinct, facilitating trading and investing in their own ways.
     

  • The difference between DEX aggregators and DEXs is that — DEXs do not provide users with the option to liquidate. DEX aggregators were created to solve this very issue. These liquidity aggregators provide its traders or investors with the opportunity to source liquidity from across various DEXs. On the other hand, they also offer the best rates for token swaps, which DEXs cannot provide. However, DEX aggregator integrations are often profitable to DEXs as they make way for additional users and volume.

 

Types of DEX Aggregators

There are generally two types of DEX aggregation — trading aggregation and information aggregation.

1. Trading Aggregation

 

The main job of these aggregators is to aggregate trades. These provide users with various options, so that they can choose the best route to make the most profitable trade.

 

2.  Information Aggregation

When trading, one of your most crucial resources is access to better information to make better decisions. Most traders rely on real-time data analysis to anticipate market movements, hence information aggregators are vital.

What are the Advantages of DEX Aggregators?

DEX aggregators are the latest type of blockchain-based services to rapidly make their place in the DeFi ecosystem. With DeFi growing in popularity, traders and investors have started adopting aggregators due to their advantageous ability to aid in informed decision making regarding trades and coin swaps.

Some of the prominent benefits of these aggregators attracting people

  • Provides Liquidity: A DEX aggregator offers a large pool of liquidity to traders wanting to trade vast amounts of digital tokens. Aggregators function as a collective of exchanges to do business which has to do with the transfer of one crypto to another.
    For example, say you want to convert your ETH tokens into USDT on a specific DEX, but you cannot do it due to a lack of liquidity on the platform. Using a DEX aggregator you can liquidate and exit your position without incurring excessive slippage.

  • Saves Time and Effort: One of the potential challenges most crypto investors or traders face is the upsurge of various tokens linked to diverse platforms. There are thousands of tokens in the market, and market participants must work hard to evaluate and design a sophisticated crypto strategy to get the best out of them.

  • DEX aggregators make this task easier by consolidating the research that a trader has to do in either real-time or otherwise. This saves a lot of time and effort for the user.

  • Better Pricing: When compared to one individual DEX, a DEX aggregator can provide better execution price as they are designed to help traders to liquidate at the highest rates.

  • Anonymity: Since DEX aggregators are an innovation added to DEXs, users just need to provide minimal information about themselves, in contrast to centralized exchanges which require you to complete the full KYC documentation process before signing in. For example, unlike traditional exchanges, KyberSwap does not hold any funds or collect any sensitive information of its users. Instead, it allows them to trade assets from within their own wallets.

  • Decentralization: As DEXs, DEX aggregators can also give you custodial control over your funds, unlike centralized exchanges. They allow you to trade from and to your crypto wallet directly.

  • Robust Infrastructure: DEX aggregators are like the ‘open-source’ platforms to make things much more transparent. They do not have a single source system but a community-based system that improves cybersecurity by being built on a blockchain.
     

Why are DEX Aggregators In Demand?

  • The most common issue people face while using the Ethereum network is — the high gas fees, which have negatively impacted DeFi protocols, pushing users to pay a high amount to trade or swap tokens. In such a case, DEX aggregators can help in providing the best swap or trade rates which can reduce gas costs to a great extent. This is the foremost reason for which these aggregators are gaining wide acceptance.
     

  • The liquidity aggregators can also react faster to the regulations by removing DEXs that are not compliant while introducing innovative financial services from the compliant ones. They can ensure greater flexibility where users can leverage the best out of the market.
     

  • Most DEX aggregators are the major beneficiaries of DeFi innovations. They can integrate with any decentralized exchanges keeping them up with DeFi’s limitless potential with low fees. Also, providing the user an all-in-one experience seamlessly.

 

How Do DEX Aggregators Work? 

  • DEX aggregators source liquidity from different DEXs and thus offer users better token swap rates than they could get on any single DEX. DEX aggregators have the ability to optimize slippage, swap fees and token prices which, when done right, offer a better rate for users.
     

  • For instance, a swap deal split between several DEXs can get a user an overall better price than a swap on any single exchange.
     

  • A DEX aggregator’s main task is to offer a user better swap rates than any specific DEX can offer and to do that in the shortest possible time. Other major tasks are protecting users from price impact and reducing the probability of failed transactions.
     

  • DEXs are generally interested in DEX aggregator integrations, as they can attract more users and volume. Recent data shows that high notional traders increasingly use DEX aggregators, while retail users still choose to access DEXs directly.

Smart Order Router (SOR)

Full documentation and references
 

What is a Smart Order Router?

A smart order router (SOR) is an automated process used in online trading, which follows a set of rules that look for and assess trading liquidity. The goal of an SOR is to find the best way of executing a trade, taking advantage of opportunities across a range of trading venues through advanced algorithms.

 

The Goal of Smart Order Routers

  • The goal of SOR is to find the best way of executing a trade. The concept itself is relatively simple, ‘smart’ refers to the intelligence involved in rapid and optimal decision making, and an ‘order’ is some type of instruction given with the expectation that that action will be taken.
     

  • ‘Routing’ refers to setting a course aimed at reaching a final destination. SOR is an automated process of handling orders, aimed at taking the best available price throughout a range of different trading venues.

 

The  ABC’s of Order Routing

  • Access to real-time market data from each venue or a consolidated stream from a commercial provider.
     

  • Basic software to identify and direct messages to appropriate destinations.
     

  • Connectivity to the selected execution venues. This process can be as simple as linking to one of the many Virtual Private Networks available within the financial services space or as challenging as maintaining dedicated lines to each of the desired venues.

 

Types of Smart Order Routes

  • Smart order routes are programmatic routes that come preconfigured but can also be modified to for certain preferences. The goal is to find the most efficient route for your order. This means an order can be broken up into multiple pieces as the algorithm selects and executes on the most efficient routes.
     

  • Smart order routes are usually configured to meet a certain objective. Some may favor fill speed rate over the fill price and vice versa. Keep in mind that access to various order routes is contingent on the access provided by your broker. Brokers offer differing sources to order routes and dark pools. 

 

Benefits of Smart Routes

  • Smart routes are simple to use, fast, convenient, and automated. They take the leg work out of order routing. The benefits of using smart routes boils down to three advantages over manually routing orders.
     

  • They can provide excellent executions since the algorithms are very fast and simultaneously scan the routes to meet your execution preferences faster than any human can.
     

  • In doing so, they can also provide improved liquidity as these algorithms can route orders to multiple destinations to fill your requested share size amounts. You may want several thousand shares of XYZ at a certain limit price. The smart order route may fill your order partially through three different ECNs and a dark pool within seconds. This results in improved fill prices as the smart route searches for the best fill prices simultaneously. 

Automated Market Makers (AMM)

Full documentation and references
 

What are Automated Market Makers?

 

Instead of relying on traditional buyers and sellers in a financial market, AMMs keep the DeFi ecosystem liquid 24/7 via liquidity pools.

 

Liquidity Pools and Liquidity Providers

  • Liquidity refers to how easily one asset can be converted into another asset, often a fiat currency, without affecting its market price. Before AMMs came into play, liquidity was a challenge for decentralized exchanges (DEXs) on Ethereum. As a new technology with a complicated interface, the number of buyers and sellers was small, which meant it was difficult to find enough people willing to trade on a regular basis. AMMs fix this problem of limited liquidity by creating liquidity pools and offering liquidity providers the incentive to supply these pools with assets. The more assets in a pool and the more liquidity the pool has, the easier trading becomes on decentralized exchanges.
     

  • On AMM platforms, instead of trading between buyers and sellers, users trade against a pool of tokens — a liquidity pool. At its core, a liquidity pool is a shared pot of tokens. Users supply liquidity pools with tokens and the price of the tokens in the pool is determined by a mathematical formula. By tweaking the formula, liquidity pools can be optimized for different purposes.
     

  • Anyone with an internet connection and in possession of any type of ERC-20 tokens can become a liquidity provider by supplying tokens to an AMM’s liquidity pool. Liquidity providers normally earn a fee for providing tokens to the pool. This fee is paid by traders who interact with the liquidity pool. Recently, liquidity providers have also been able to earn yield in the form of project tokens through what is known as “yield farming.”

 

Constant Product Formula

AMMs have become a primary way to trade assets in the DeFi ecosystem, and it all began with a blog post about “on-chain market makers” by Ethereum founder Vitalik Buterin. The secret ingredient of AMMs is a simple mathematical formula that can take many forms. 

 

The most common one was proposed by Vitalik as:

 

tokenA_balance(p) * tokenB_balance(p) = k

 

and popularized by Uniswap as:

 

x * y = k

 

The constant, represented by “k” means there is a constant balance of assets that determines the price of tokens in a liquidity pool. For example, if an AMM has ether (ETH) and bitcoin (BTC), two volatile assets, every time ETH is bought, the price of ETH goes up as there is less ETH in the pool than before the purchase. Conversely, the price of BTC goes down as there is more BTC in the pool. The pool stays in constant balance, where the total value of ETH in the pool will always equal the total value of BTC in the pool. Only when new liquidity providers join in will the pool expand in size. Visually, the prices of tokens in an AMM pool follow a curve determined by the formula.

In this constant state of balance, buying one ETH brings the price of ETH up slightly along the curve, and selling one ETH brings the price of ETH down slightly along the curve. The opposite happens to the price of BTC in an ETH-BTC pool. It doesn’t matter how volatile the price gets, there will eventually be a return to a state of balance that reflects a relatively accurate market price. If the AMM price ventures too far from market prices on other exchanges, the model incentivizes traders to take advantage of the price differences between the AMM and outside crypto exchanges until it is balanced once again.

The constant formula is a unique component of AMMs — it determines how the different AMMs function.

 

Automated Market Maker Variations

The DeFi ecosystem evolves quickly, but three dominant AMM models have emerged: Uniswap, Curve, and Balancer.

 

  • Uniswap’s pioneering technology allows users to create a liquidity pool with any pair of ERC-20 tokens with a 50/50 ratio, and has become the most enduring AMM model on Ethereum. 

  • Curve specializes in creating liquidity pools of similar assets such as stablecoins, and as a result, offers some of the lowest rates and most efficient trades in the industry while solving the problem of limited liquidity. 

  • Balancer stretches the limits of Uniswap by allowing users to create dynamic liquidity pools of up to eight different assets in any ratio, thus expanding AMMs’ flexibility.

Although Automated Market Makers harness new technology, iterations of it have already proven an essential financial instrument in the fast-evolving DeFi ecosystem and a sign of a maturing industry.

Best Execution


 

What is Best Execution?

Best Execution is the ability to buy or sell the desired amount of something at the lowest or highest possible net price.

Due to the many variables and difficulty in measurement, most people define it in terms of the trading process utilized. Those variables include:

  • Bid/ Offer spread or the difference between the bid and offer.

  • Fees paid (to exchanges brokers etc.)

  • Market impact cost of the movement from beginning to end of a trade.

  • Opportunity cost ( costs for waiting to trade, particularly if once trading intentions  have been leaked to the market.

 

What is Meant by Trading Process?

Trading can be as simple as

  • Calling one dealer and agreeing to buy or sell (or the digital equivalent of trading or on a screen).

  • Giving an order to an agent, and letting them decide, where and how to trade.

Trading can be as complicated as

  • Trading on several exchanges simultaneously while comparing quotes with those provided to you by dealers.

 

Examples of POOR Trading Process (for a large order)

Calling multiple dealers to request a quote at the same time for the same order or placing the entire order onto an exchange where the public can see it.

➔     Both create a market impact before trading even starts, resulting in very expensive trades.

 

Trading manually on individual exchanges, breaking the order up by hand.

➔     Exchange prices diverge and converge rapidly, making it impossible for manual trading to be optimal.

 

Importance of Fees Relative to Spreads

In equities, the fees charged by exchanges to remove liquidity are less than 30% of the tick size, but in crypto, the fees can exceed 100 times the tick size. The following table puts this in context by showing the average spread posted on several top exchanges alongside their fees: (Spreads calculated by sampled data from CoinRoutes).

Capture.JPG
Notes on Spread Data

  • Fees are a very significant driver of trading costs, being larger than bid-offer spreads most of the time.

  • While they vary, bid-offer spreads on exchanges are between 1 and 5 basis points considering their top of the book.​ This table doesn’t show this, but this is roughly 1 Bitcoin on average.

  • The fee differential between maker and taker is an important reason why CoinRoutes uses mostly passive orders. There are other reasons which will be explained later.

 

Notes on Book Depth Data

  • Exchange liquidity is quite robust & potentially cost-effective.

    • It is cheaper to trade 1000 bitcoin using smart routing than retail tends to pay.

  • To SEE exchange liquidity requires software to evaluate THOUSANDS of individual price levels.

    • Most systems, not built for crypto, are limited in the size of the book they can process.

  • Beyond 2000 Bitcoin, the liquidity does start to thin out.

    • This puts a premium on patience (minutes/hours) for working large blocks; It turns out that algorithmic trading outperforms smart routing overall as well, but this explains why market makers charge so much for very large trades.

High-Level Roadmap

Project Briefs 

  • Briefs summarize the scope, goals, and direction of the project - ✓ Complete.    

  • Add context for all departments involved in product development - ✓ Complete  

  • Offer clarity by compiling the most crucial information into one simple-to-understand document  - ✓ Complete 

  • Introduce stakeholders and their involvement

 

R & D and Innovation Process Framework (IPF)

  • R&D planning framework - ✓ Complete 

  • Solution Development (SD) Framework - ✓ Complete 

  • Implementing R&D strategy and assigning dedicated stakeholders for R&D to manage the strategy

  • Defining Innovation cycles - ✓ Complete 

  • Time-constraint lean double diamond framework

  • Development of foresight framework

  • Competition (Internal and external competitiveness and productivity)

  • Technological forces (Emerging trends, influences)

  • Wild Cards (Radical impacts with catastrophic consequences)

  • Technological shifts - ✓ Complete 

  • Forward linear associations

  • Measurable thresholds

  • Relationship between events and their impact

  • Finalizing deep insights & knowledge base - ✓ Complete 

 

Stakeholder Engagement

  • Map stakeholders - ✓ Complete 

  • Finalizing communication plan

  • Initial stakeholder engagement, project briefs to both external and internal, industry peer reviewers for each R&D segment

  • Scenario planning

 

Tech Feasibility Analysis 

  • Evaluate solution strategy based on feasibility.

  • Examine the operational requirements and conduct a preliminary production feasibility assessment.

  • Defining the product development stages - ✓ Complete 

  • Outline the design of the system requirement.

 

Service Design Process

A human-centered design approach that places equal value on the user experience and the R&D process.

  • Scenario, channels, services, touchpoints, and interactions mapping.

  • Development of service blueprint- visualizes the relationships between different service components, users, and product usability that are directly tied to touchpoints in a specific customer journey.

  • Defining Minimal Viable Ecosystem (MVE) - ✓ Complete 

 

Initial Product Design

  • Initial product development process.

  • Feature prioritization— ranking and organizing Eta X features based on Innovation Process Framework, project goals, and technical viability.

  • Convert Minimal Viable Ecosystem (MVE) insights into defining product lifecycle and stages 

  • Product development stage prioritization - ✓ Complete 

  • Design minimal specifications to test, measure and debug.

  • Deployment of continuous integration, delivery, and deployment.

  • Deployment of sandbox environment.

Eta X Development

V1.0

Using a simple web interface, the users are able to pick an asset pair to trade or swap. Then the user can query the best price, exchange rates, fees, best route, slippage and price impact. The UI will display all relevant information with reasonable latency.

Core Services
  • ​Price discovery

  • SOR. No execution initially

  • Reduced slippage

  • Lower gas fees

  • Pair matching

  • Traders

 

Back Office Operations-as-a-Service
  • Optimization Data & Meta-Models

  • Data Aggregation

  • PoS Security

  • Risk Assessment

  • Order Sizing

  • Credit Scoring

  • Devs

  • Protocols

Scope and Deliverables
  • Collect data for SOR from L3 Atom and SuperCluster.

  • Feed into a simple UI - similar to 1inch.

    • Price

    • SOR breakdown and logs

    • Transaction cost

    • Venue logs

    • Fees breakdown

1OHLkc_rvrY93V4sEhY1Icmp80WUwIgwSVhhlsA4SRPilm6hv-FlUCARL1jS-d4lIDmFG_sTHk3XfUbNL0k7x7_8CU

Initial Paper Release​

 

V2.0


Phase 2 introduces the real-time display of aggregators and a side-by-side comparison of their performance against Eta’s. This will be useful both for the user and the stakeholders as an evaluative tool of Eta’s performance in contrast to existing solutions.


The off-chain machine learning environment will monitor, compare and adjust its pathfinding and smart order routing choices to archive better capital efficacy (best price match, effective routes, lower slippages, and save on gas fees).

coNVruRvuUv2SyWw7GKRBw2YDHgi2zB1ypTXc3NTk6sQXfoEq41B8ZsBJAGjJJYJ2skZyRhIbdj96vQBV4Tves3P7D
Sample UI
evFITKi0D-5N6Pt4ppssm4dAGdYOk92hojsV7damB1BbEIHdzqGJquYYtW-971iSn5eEOF9JQF8OYnonj6pKQxLNJV
V3.0 
The users can query the following data and metadata via unified APIs.
  • Details about asset pairs, prices & pools

  • Liquidity concentration

  • Order sizing

  • SuperCluster

    • Normalized price discovery & MDF

    • Volatility

    • Market activities, trades data

  • LP pools metadata

  • Price discovery

  • Price aggregation

  • Liquidity traps & SFP detection

  • Liquidity pools correlations

  • Liquidity distribution

  • Liquidity fluctuations & volatility

  • Deeper liquidity cost optimization

  • Fee optimization

  • SOR

  • Slippage

  • Gas fees

  • Pair matching

  • Price impact

 

Off-Chain ML Training Model(s) Data & its Metadata
  • Training data set

  • Cross-validation data set

  • Test data set

Scope, Goals & Short-Term Deliverables

 

 

Machine Learning Scope

1. Problem formulation & problem framing  - Deeplink

  • Framing the problem in ML terms.​

    • Better understand both the problem and the data 

  • Non-ML solution or heuristic 

    • To benchmark and measure an ML solution against and Non-ML

 

2. Define Data and gather the data set  - Deeplink & SingularityNet  - ✓ Complete 

 

3. Data representation methods - Deeplink & SingularityNet

 

4. Data exploration - Deeplink & SingularityNet

  • Data type and distribution of data contained within each variable, the relationships between variables and how they vary relative to the outcome we are predicting or interested in achieving.

5. Data cleansing and validation - Deeplink & SingularityNet

  • Data cleansing and validation to identify and rectify inconsistencies, outliers, anomalies, missing data and other issues.

6. Data structuring - Deeplink & SingularityNet

  • Once data Deeplink and SingularityNet teams are satisfied with their data structure in a way to recognize and use

    • Training data set

    • Cross-validation data set

    • Test data set
       

7. Feature engineering and selection - Deeplink & SingularityNet

  • Data preparation before developing a machine learning model is feature engineering and feature selection.

8. Choose the most appropriate algorithm. - Deeplink & SingularityNet

  • SingularityNet, MIT & Deeplink team guidelines based on which one could select a particular machine learning algorithm based on the problem at hand. For example, if this is about creating a predictive model for estimating numbers such as price etc., one can choose one of the regression algorithms. If this is about classifying the input to one label, it must be a classification algorithm.
     

  • Start with a very simplistic model with minimal and most prominent features. This would help one to get started very quickly without spending time exploring the correct and most appropriate feature set. Many times, a lot of time is spent on identifying the most appropriate features.

    • Understand old algorithms like Djikstra and Bellman-Ford and modify these algorithms to fetch the initial best route using the platform data (on-chain & off-chain data). In order to do this, we will first need to develop a method for converting DEX pools into weighted graphs. It is possible that the method of calculation of the weights (viability of swaps) between nodes (DEX pools) will make up a large portion of the project.

    • Develop LSTM models with reinforcement learning to predict the best trading routes, and we could benchmark each model under different environments (low, high demand).

    • Since hyperparameters regulate the learning process in an RNN, they are highly sensitive. Determining the optimal hyperparameters is critical. Trading route Optimization using Evolutionary algorithms - Differential Evolution, Genetic Algorithms to optimize the best trading routes for the best price with low slippage.

    • Further automate this process of finding optimal hyperparameters using self-adaptive approaches used in our paper - jSO, MPEDE, and L-SHADE among others and benchmark them - using MSE, MAE, MAPE and also analyse computational run-time and requirements.
       

  • Plot learning curves to measure how error (prediction vs observed) varies with respect to some of the following:

    • Adding more training examples. In simple words, collect more data sets.

    • Adding more features

    • Metadata parameters (liquidity clusters, liquidity pools correlations, liquidity distribution and liquidity fluctuations)
       

9. Test the model against previously unseen data - Deeplink & SingularityNet

  • Somewhat representative of model performance in the real world, but still helps tune the model (as opposed to testing data, which does not)

    • Good train/eval split? 80/20, 70/30, or similar?

 

10. Parameter Tuning - Deeplink & SingularityNet

  • Hyperparameter tuning

  • Simple model hyperparameters may include a number of training steps, learning rate, initialization values and distribution, etc.

Data and Metadata  - Deeplink

Define data, data representation & data exploration - ✓ Complete 

 

1. Identifying the pools, DEXs and aggregators.

 

Pools

 

DEXs/Aggregators
  • Uniswap

  • Sushiswap

  • DexGuru

  • dYdX

  • 1inch.exchange

  • AirSwap

  • Bancor

  • CowSwap

  • Curve

  • Dodo

  • KyberSwap

  • Matcha

  • Mesa

  • Multichain

  • Oasis

  • Paraswap

  • RhinoFi

 

2. Employ our framework to systematically compare the top AMM protocols’ mechanics, illustrating their conservation functions, as well as slippage and divergence loss functions. 

 

Participants
  • Liquidity providers

  • Net Liquidity providers

  • Traders

  • MM

  • Protocols

Functions
  • Swap

  • Limit fill

  • SOR

  • Swap lot amount

  • Pairs

  • Net addresses

  • Total addresses

  • Total LPs

  • Net LPs

Economics
  • Liquidity reward

  • Staking reward

  • Fees fund

  • Liquidity withdrawal penalty

  • Swap fee - LPs fee for supplying exchange pairs

  • Gas fee

 

Core
  • Conservation function per pool

  • Pool mechanisms

    • Preset hyperparameters

    • State variables

    • Process variables

    • Functions

  • Pool structure

    • Asset-pair

    • Multi-asset

 

  • AMM mechanisms for liquidity providers

  • AMM mechanisms for traders.

Risk - LPs
  • Divergence loss

  • Volatility risk

  • Impermanent loss

  • Time lock - The loss of time value of locked funds

 

Risk - Trader
  • Slippage

  • Impermanent loss

3. Data selection and classification - ✓ Complete 

 

Raw
  • Pairs

  • Total Pairs

  • Liquidity concentration per pair (total QTY)

  • Liquidity concentration per pool (total assets) 

  • Total LPs

  • Unique LPs

  • Transactions - (15s)

  • Total address

  • Active addresses

  • Net addresses

  • Inflow assets (mempool and confirmed)

  • Total Inflow assets

  • Net Inflow assets

  • Outflow assets

  • Total outflow assets

  • Net outflow assets

  • Cumulative Vol

  • Total trading addresses

  • Unique trading addresses

  • Fees

  • Volume & Fees

  • ETH Gas fee

 

Enhanced
  • Liquidity Variation

  • Net Liquidity Variation

  • Market Depth

  • Inflow Volume

  • Total trading volume 1D

  • Total trading volume 7D

  • Total trading volume 30D

  • Total fees 1D

  • Total fees 7D

  • Total fees 30D

  • Average Transaction Size 1D

  • Average Transaction Size 7D

  • Average Transaction Size 30D

 

Indicators 
  • Trades Vol MA7

  • Trades Vol MA30

  • Volatility

Untitled spreadsheet - Google Sheets.jpg
DevOps Setups  - Deeplink - ✓ Complete 

  • Data ingestion

  • Data distribution

  • ML models running environment setups for v1 | Dev & Production

  • DKeeper computation framework (distributed nodes & computation

Data Collection 

 

In terms of infrastructure, we are imagining a simple setup where these collectors would be running on EC2 instances, dockerized and deployed with EKS. It could be a better idea to break up the workload into different containers as we require different types of data (and for replication purposes). These would connect to an Infura node, sending requests and processing responses. They would then send this data over Kafka/Redis to our WebSocket server where live events could be subscribed to, as well as archiving data in S3 (and potentially stored in a database setup where it could be queried, maybe with our GraphQL server).

VnuaezxVr3aY8DVYtggi1Oz0MXwD99rtuA17sr5YBCsueVAfqKVSlOJVqTcyUwCI6K-Fm4XUgBq4sglZEz4z9xpPNE

L3Atom

  • Dedicated AWS data streaming and data query service

  • Well-defined data schema  - ✓ Complete 

  • L3Atom documentation tailored to this use case

  • APIs - ✓ Complete 


 

SuperCluster - Deeplink

  • Custom indicators to develop supercluster feed

  • Overall crypto market liquidity. ​Last 24 total liquidity / 365 days

  • Overall crypto market volatility

  • Document

  • Historical data storage and access / REST and WebSocket

  • APIs

bottom of page