Connect with us

Politics

Building a (Big) Data Pipeline the Right Way – ReadWrite

Published

on

Building a (Big) Data Pipeline


Gathering and analyzing data has been the craze of business for quite some time now. Yet, too often, the former takes hold of companies at such strength that no care is given to the thought of utilizing data. There’s a reason we had to invent a name for this phenomenon – “dark data.”

Unfortunately, data is often gathered without a good reason. It’s understandable – a lot of internal data is collected by default. The current business climate necessitates using many tools (e.g., CRMs, accounting logs, billing) that automatically create reports and store data.

The collection process is even more expansive for digital businesses and often includes server logs, consumer behavior, and other tangential information.

Building a (Big) Data Pipeline the Right Way

Unless you’re in the data-as-a-service (DaaS) business, simply collecting data doesn’t bring any benefit. With all the hype surrounding data-driven decision-making, I believe many people have lost sight of the forest for the trees. Collecting all forms of data becomes an end in itself.

In fact, such an approach is costing the business money. There’s no free lunch – someone has to set up the collection method, manage the process, and keep tabs on the results. That’s resources and finances wasted. Instead of striving for the quantity of data, we should be looking for ways to lean out the collection process.

Humble Beginnings

Pretty much every business begins its data acquisition journey by collecting marketing, sales, and account data. Certain practices such as Pay-Per-Click (PPC) have proven themselves to be incredibly easy to measure and analyze through the lens of statistics, making data collection a necessity. On the other hand, relevant data is often produced as a byproduct of regular day-to-day activities in sales and account management.

Businesses have already caught on that sharing data between marketing, sales, and account management departments may lead to great things. However, the data pipeline is often clogged, and the relevant information is only accessed abstractly.

Often, the way departments share information lacks immediacy. There is no direct access to data; instead, it’s being shared through in-person meetings or discussions. That’s just not the best way to do it. On the other hand, having consistent access to new data may provide departments with important insights.

Interdepartmental Data

Rather unsurprisingly, interdepartmental data can improve efficiency in numerous ways. For example, data on the Ideal Customer Profile (ICP) leads between departments will steer to better sales and marketing practices (e.g., a more defined content strategy).

Here’s the burning issue for every business that collects a large amount of data: it’s scattered. Potentially useful information is left all over spreadsheets, CRMs, and other management systems. Therefore, the first step should be not to get more data but to optimize the current processes and prepare them for use.

Combining Data Sources

Luckily, with the advent of Big Data, businesses have been thinking through information management processes in great detail. As a result, data management practices have made great strides in the last few years, making optimization processes a lot simpler.

Data Warehouses

A commonly used principle of data management is building a warehouse for data gathered from numerous sources. But, of course, the process isn’t as simple as integrating a few different databases. Unfortunately, data is often stored in incompatible formats, making standardization necessary.

Usually, data integration into a warehouse follows a 3-step process – extraction, transformation, load (ETL). There are different approaches; however, ETL is most likely the most popular option. Extraction, in this case, means taking the data that has already been acquired from either internal or external collection processes.

Data transformation is the most complex process of the three. It involves aggregating data from various formats into a common one, identifying missing or repeating fields. In most businesses, doing all of this manually is out of the question; therefore, traditional programming methods (e.g., SQL) are used.

Loading — Moving to the Warehouse

Loading is basically just moving the prepared data to the warehouse in question. While it’s a basic process of moving data from one source to another, it’s important to note that warehouses do not store real-time information. Therefore, separating operational databases from warehouses allows the former to separate as a backup and avoid unnecessary corruption.

Data warehouses usually have a few critical features:

  • Integrated. Data warehouses are an accumulation of information from heterogeneous sources into one place.
  • Time variant. Data is historical and identified as from within a particular time period.
  • Non-volatile. Previous data is not removed when newer information is added.
  • Subject oriented. Data is a collection of information based on subjects (personnel, support, sales, revenue, etc.) instead of being directly related to ongoing operations.

External Data to Maximize Potential

Building a data warehouse is not the only way of getting more from the same amount of information. They help with interdepartmental efficiency. Data enrichment processes might help with intradepartmental efficiency.

Data enrichment from external sources

Data enrichment is the process of combining information from external sources with internal ones. Sometimes, enterprise-level businesses might be able to enrich data from purely internal sources if they have enough different departments.

While warehouses will work nearly identical for almost any business that deals with large volumes of data, each enrichment process will be different. This is because enrichment processes are directly dependent on business goals. Otherwise, we would go back to square one, where data is being collected without a proper end-goal.

Inbound lead enrichment

A simple approach that might be beneficial to many businesses would be inbound lead enrichment. Regardless of the industry, responding quickly to requests for more information has increased the efficiency of sales. Enriching leads with professional data (e.g., public company information) would provide an opportunity to automatically categorize leads and respond to those closer to the Ideal Customer Profile (ICP) faster.

Of course, data enrichment need not be limited to sales departments. All kinds of processes can be empowered by external data – from marketing campaigns to legal compliance. However, as always, specifics have to be kept in mind. All data should serve a business purpose.

Conclusion

Before treading into complex data sources, cleaning up internal processes will bring greater results. With dark data comprising over 90% of all data collected by businesses, it’s better at first to look inwards and optimize the current processes. Including more sources will exile some potentially useful information due to inefficient data management practices.

After creating robust systems for data management, we can move on to gathering complex data. We can then be sure we won’t miss anything important and be able to match more data points for valuable insights.

Image Credit: rfstudio; pexels; thank you!

Julius Cerniauskas

CEO at Oxylabs

Julius Cerniauskas is Lithuania’s technology industry leader & the CEO of Oxylabs, covering topics on web scraping, big data, machine learning & tech trends.

Politics

Fintech Kennek raises $12.5M seed round to digitize lending

Published

on

Google eyed for $2 billion Anthropic deal after major Amazon play


London-based fintech startup Kennek has raised $12.5 million in seed funding to expand its lending operating system.

According to an Oct. 10 tech.eu report, the round was led by HV Capital and included participation from Dutch Founders Fund, AlbionVC, FFVC, Plug & Play Ventures, and Syndicate One. Kennek offers software-as-a-service tools to help non-bank lenders streamline their operations using open banking, open finance, and payments.

The platform aims to automate time-consuming manual tasks and consolidate fragmented data to simplify lending. Xavier De Pauw, founder of Kennek said:

“Until kennek, lenders had to devote countless hours to menial operational tasks and deal with jumbled and hard-coded data – which makes every other part of lending a headache. As former lenders ourselves, we lived and breathed these frustrations, and built kennek to make them a thing of the past.”

The company said the latest funding round was oversubscribed and closed quickly despite the challenging fundraising environment. The new capital will be used to expand Kennek’s engineering team and strengthen its market position in the UK while exploring expansion into other European markets. Barbod Namini, Partner at lead investor HV Capital, commented on the investment:

“Kennek has developed an ambitious and genuinely unique proposition which we think can be the foundation of the entire alternative lending space. […] It is a complicated market and a solution that brings together all information and stakeholders onto a single platform is highly compelling for both lenders & the ecosystem as a whole.”

The fintech lending space has grown rapidly in recent years, but many lenders still rely on legacy systems and manual processes that limit efficiency and scalability. Kennek aims to leverage open banking and data integration to provide lenders with a more streamlined, automated lending experience.

The seed funding will allow the London-based startup to continue developing its platform and expanding its team to meet demand from non-bank lenders looking to digitize operations. Kennek’s focus on the UK and Europe also comes amid rising adoption of open banking and open finance in the regions.

Featured Image Credit: Photo from Kennek.io; Thank you!

Radek Zielinski

Radek Zielinski is an experienced technology and financial journalist with a passion for cybersecurity and futurology.

Continue Reading

Politics

Fortune 500’s race for generative AI breakthroughs

Published

on

Deanna Ritchie


As excitement around generative AI grows, Fortune 500 companies, including Goldman Sachs, are carefully examining the possible applications of this technology. A recent survey of U.S. executives indicated that 60% believe generative AI will substantially impact their businesses in the long term. However, they anticipate a one to two-year timeframe before implementing their initial solutions. This optimism stems from the potential of generative AI to revolutionize various aspects of businesses, from enhancing customer experiences to optimizing internal processes. In the short term, companies will likely focus on pilot projects and experimentation, gradually integrating generative AI into their operations as they witness its positive influence on efficiency and profitability.

Goldman Sachs’ Cautious Approach to Implementing Generative AI

In a recent interview, Goldman Sachs CIO Marco Argenti revealed that the firm has not yet implemented any generative AI use cases. Instead, the company focuses on experimentation and setting high standards before adopting the technology. Argenti recognized the desire for outcomes in areas like developer and operational efficiency but emphasized ensuring precision before putting experimental AI use cases into production.

According to Argenti, striking the right balance between driving innovation and maintaining accuracy is crucial for successfully integrating generative AI within the firm. Goldman Sachs intends to continue exploring this emerging technology’s potential benefits and applications while diligently assessing risks to ensure it meets the company’s stringent quality standards.

One possible application for Goldman Sachs is in software development, where the company has observed a 20-40% productivity increase during its trials. The goal is for 1,000 developers to utilize generative AI tools by year’s end. However, Argenti emphasized that a well-defined expectation of return on investment is necessary before fully integrating generative AI into production.

To achieve this, the company plans to implement a systematic and strategic approach to adopting generative AI, ensuring that it complements and enhances the skills of its developers. Additionally, Goldman Sachs intends to evaluate the long-term impact of generative AI on their software development processes and the overall quality of the applications being developed.

Goldman Sachs’ approach to AI implementation goes beyond merely executing models. The firm has created a platform encompassing technical, legal, and compliance assessments to filter out improper content and keep track of all interactions. This comprehensive system ensures seamless integration of artificial intelligence in operations while adhering to regulatory standards and maintaining client confidentiality. Moreover, the platform continuously improves and adapts its algorithms, allowing Goldman Sachs to stay at the forefront of technology and offer its clients the most efficient and secure services.

Featured Image Credit: Photo by Google DeepMind; Pexels; Thank you!

Deanna Ritchie

Managing Editor at ReadWrite

Deanna is the Managing Editor at ReadWrite. Previously she worked as the Editor in Chief for Startup Grind and has over 20+ years of experience in content management and content development.

Continue Reading

Politics

UK seizes web3 opportunity simplifying crypto regulations

Published

on

Deanna Ritchie


As Web3 companies increasingly consider leaving the United States due to regulatory ambiguity, the United Kingdom must simplify its cryptocurrency regulations to attract these businesses. The conservative think tank Policy Exchange recently released a report detailing ten suggestions for improving Web3 regulation in the country. Among the recommendations are reducing liability for token holders in decentralized autonomous organizations (DAOs) and encouraging the Financial Conduct Authority (FCA) to adopt alternative Know Your Customer (KYC) methodologies, such as digital identities and blockchain analytics tools. These suggestions aim to position the UK as a hub for Web3 innovation and attract blockchain-based businesses looking for a more conducive regulatory environment.

Streamlining Cryptocurrency Regulations for Innovation

To make it easier for emerging Web3 companies to navigate existing legal frameworks and contribute to the UK’s digital economy growth, the government must streamline cryptocurrency regulations and adopt forward-looking approaches. By making the regulatory landscape clear and straightforward, the UK can create an environment that fosters innovation, growth, and competitiveness in the global fintech industry.

The Policy Exchange report also recommends not weakening self-hosted wallets or treating proof-of-stake (PoS) services as financial services. This approach aims to protect the fundamental principles of decentralization and user autonomy while strongly emphasizing security and regulatory compliance. By doing so, the UK can nurture an environment that encourages innovation and the continued growth of blockchain technology.

Despite recent strict measures by UK authorities, such as His Majesty’s Treasury and the FCA, toward the digital assets sector, the proposed changes in the Policy Exchange report strive to make the UK a more attractive location for Web3 enterprises. By adopting these suggestions, the UK can demonstrate its commitment to fostering innovation in the rapidly evolving blockchain and cryptocurrency industries while ensuring a robust and transparent regulatory environment.

The ongoing uncertainty surrounding cryptocurrency regulations in various countries has prompted Web3 companies to explore alternative jurisdictions with more precise legal frameworks. As the United States grapples with regulatory ambiguity, the United Kingdom can position itself as a hub for Web3 innovation by simplifying and streamlining its cryptocurrency regulations.

Featured Image Credit: Photo by Jonathan Borba; Pexels; Thank you!

Deanna Ritchie

Managing Editor at ReadWrite

Deanna is the Managing Editor at ReadWrite. Previously she worked as the Editor in Chief for Startup Grind and has over 20+ years of experience in content management and content development.

Continue Reading

Copyright © 2021 Seminole Press.