Snowflake is investing heavily in improving the experience of building streaming pipelines. Let’s see how you can work with streaming data today on snowflake and what are some of the upcoming new features that are either rumored or announced already.
Observability is something that is used by engineers in the context of building reliable systems. Observability is not a new term, you can find a great article about observability on wikipedia. What is important to keep is the following definition: *Observablity is a measure of how well internal states of a system an be inferred from knowledge of its external outputs.*
Benn argues in his post about the importance of standardization as a too to organize the messiness of the data ecosystem. One of the areas that standards have became important in many different, and not very obvious ways, is data ingestion. By standardization in data ingestion, we mainly refer to a number of frameworks that help us build connectors that are either extracting or loading data from and to different systems. Benn mentions as the main reason for seeking standardization the messiness of the data ecosystem, but I’ll argue that there are a couple of different reasons that these data...