KYVE Launches Data Pipeline In Beta To Test Readiness For Public Use
Data Pipeline, a no code solution built by KYVE, which enables anyone to import data from the firm’s data lake into their preferred data backend including Snowflake, BigQuery and MongoDB, has been officially released in public beta, KYVE said on Monday, a launch intended to test the product’s readiness for use.
A product which can be used by organisations and private individuals in Web2 and Web3, Data Pipeline was first announced last November during Arweave conference in Lisbon by Co-founder Fabian Riewe who said that the firm’s goal was to make it as easy as possible for developers to integrate with KYVE. Data Pipeline can also be used by other professionals including analysts, data engineers and researchers.
Our goal was to make it as easy as possible for developers to integrate with KYVE. By creating this no-code solution, it makes it extremely straightforward for developers to onboard KYVE and integrates it into their local tech stack, Riewe said.
Data Pipeline is built on Airbyte and operates using the ELT (extract, load and transform) process. Its key features includes no-code solutions, decentralised and validated data and customisability.
1/ 🎊 KYVE's Data Pipeline is officially open for testing!
Now in public Beta, anyone can test importing KYVE testnet data into top data backends such as @SnowflakeDB, #BigQuery, #S3, @MongoDB, & more. An exciting milestone for Web2 & #Web3 data access.https://t.co/93cDMF4oUM
— KYVE is hiring 💫 (@KYVENetwork) December 5, 2022
KYVE acknowledges the challenge associated with using the ELT process where users have to begin the entire process of transforming data every time they want to use it but noted a solution that removes the limitation.
Since KYVE’s data lake stores and validates all types of raw data that can be used in many different ways, it just makes sense to go for an ELT (extract, load, transform) approach. This allows users to keep the original data in their database and transform it in however many ways they need, making it very flexible to their needs, it said.
On why it chose Airbyte as the foundation for Data Pipeline, KYVE said its goal was to provide decentralised product which it did by choosing a tech stack that is fully trustless and encourages developers using their local machines to have full control over how they use data.
Airbyte, being an open-source data-integration platform where you can easily implement a path for directing data from one place to another… , KYVE said.
“We decided not to build our own pipeline because we wanted to leverage the benefits that Airbyte already provides as well as tap into their already vast ecosystem.”
This solution opens new possibilities and removes obstacles in bringing Web2 data on-chain, sourcing Web3 data for Web2 and Web3 projects accessing reliable data. For KYVE, Data Pipeline solves roadblocks in the usage of data, including data manipulation and hacking risks within the blockchain industry which affects the stability of infrastructure.
Having just one source to access an entire bank of decentralised, valid data that can be imported into any data warehouse or backend would make working with data easier and more secure for all. This is where KYVE and its new Data Pipeline come in, KYVE also said.
The partnership and collaboration would be active when KYVE goes mainnet, the firm said. The mainnet launch is scheduled for 2022 according to a roadmap.
Data Pipeline’s code can be downloaded from KYVE’s Github and started on a local machine by following this quick guide on Github. Once set up, users can select a custom source, pull from one of KYVE’s data lake pools, and customize the sync settings to best fit needs. Data Pipeline completes the task.
Join our
