Democratizing Web3 Data Access:
In the Web3 landscape, extracting specific data from diverse blockchains can be a complex and cumbersome process, typically involving multiple remote procedure call (RPC) providers and APIs. Goldsky simplifies this challenge by directly ingesting data from blockchains, empowering its users to create real-time data pipelines that drive their applications. This approach eliminates the need for intermediaries, granting users full control over their data and customization options for their unique requirements.
Real-Life Use Case: Betting Website Transparency:
One notable example of Goldsky’s impact is its partnership with a popular betting website. This website tracks real-time betting trends, and to ensure transparency and fairness, it monitors bets and auctions on the blockchain layer. Goldsky plays a pivotal role in this process by ingesting data from the blockchain, transforming it in real time, and seamlessly syncing it to the website’s database. This data serves as the foundation for the website’s API layer, facilitating functions like betting order books and status updates.
The Evolution of Data Pipelines:
Recent industry shifts highlight the transition from using data platform technology solely for internal analytics to powering customer-facing features. However, many application developers lack deep familiarity with the intricate technology stacks underpinning data pipelines, such as Apache Kafka and Apache Flink. Goldsky steps in to democratize access to streaming data, offering straightforward API-driven tools that bring real-time data into applications, bridging the knowledge gap for developers.
Streamlining Data with Redpanda:
Goldsky’s data infrastructure relies heavily on a “streaming first” architecture powered by Redpanda, a Kafka-compatible platform known for its simplicity, speed, and durability. Redpanda serves as the primary database and data lake, housing vast amounts of data with minimal overhead. The tiered storage provided by Redpanda ensures near-infinite data retention without substantial cloud infrastructure costs, streamlining data management and simplifying archival processes.
Key Components: Sources, Data Pipeline, and Sinks:
Sources: Goldsky’s data sources are primarily blockchains, from which data is ingested in various ways, including direct indexing and the utilization of open source technology like Subgraphs. The platform enhances the usability of raw blockchain data by processing it into formats that cater directly to customer needs, presenting all data sets internally as topics backed by Avro schemas.
Data Pipeline: Apache Flink SQL forms the backbone of Goldsky’s transformation layer, enabling customers to perform simple SQL transformations, filtering, projections, and complex operations without the need for in-depth technical knowledge. Goldsky’s user-friendly APIs and GUI empower customers to harness the power of data pipelines effortlessly.
Sinks: Goldsky offers built-in integrations to support diverse databases and data stores, enabling customers to select their preferred sinks, such as PostgreSQL, S3, or Elasticsearch. PostgreSQL, frequently chosen by customers, excels in both transactional and analytical use cases and integrates seamlessly with existing applications.
Why Redpanda?:
Goldsky’s decision to adopt Redpanda as a foundational component of its data infrastructure stems from the platform’s efficiency, simplicity, and cost-effectiveness. Redpanda delivers high throughput with minimal hardware requirements, offers a streamlined architecture, and incorporates essential features like schema registry and proxy. Its durability, Jepsen-tested and Raft native architecture, and cost-efficient tiered storage make it an ideal choice for Goldsky’s data platform.
Goldsky’s Future in Web3:
Goldsky’s journey began with the belief that data streaming and stream processing could serve as essential building blocks. Over time, the platform has come to rely heavily on these elements, particularly for blockchain data. Goldsky continues to explore advanced use cases, including blockchain reorganizations, data enrichment, Top-N aggregations, and cross-blockchain data integration. As the Web3 ecosystem continues to evolve, Goldsky’s role as an enabler of innovation remains central, demonstrating the significance of accessible and efficient data platforms in the digital age.
Conclusion:
Goldsky’s Web3 data platform represents a significant advancement in streamlining data access, processing, and utilization within the blockchain and Web3 ecosystem. By democratizing developer access to streaming data and simplifying complex data pipelines, Goldsky empowers both experienced and novice developers to harness the potential of real-time data in their applications. With a robust architecture built around Redpanda and a commitment to enhancing Web3 technology, Goldsky is poised to play a pivotal role in shaping the future of data-driven innovation in the Web3 space.