Press "Enter" to skip to content

Category: Architecture

Scaling Out vs Scaling Up

Jordan Braiuka compares two models for scaling:

We often get questions from customers about the best way to add capacity to their cluster. Is it better to add nodes, or simply to increase the capacity in their nodes? Unfortunately, the truth is there is no best way—like all complex issues in distributed systems, there are benefits and drawbacks to each scaling approach. 

While each of our highly distributed systems (Apache CassandraApache Kafka, etc.) have slightly different implementations of scaling, the concepts remain consistent across most distributed systems. 

Click through for a comparison between the two approaches. As the article indicates, both are meaningful strategies and your choice might come down to a combination of the technology stack and the problem at hand.

Leave a Comment

Trying a Read-Only API

Mark Litwintschik reviews ROAPI:

ROAPI is an API Server that exposes CSV, JSON and Parquet files without the need to write any code. The project was started by Qingping Hou around this time last year. Qingping had spent the better part of four years working at LinkedIn prior to joining Scribd as a Senior Engineer. He is also a committer to both the Apache Airflow and Arrow projects.

ROAPI is made up of 4K lines of Rust. This line count is low due to the intense use of 3rd party libraries. These include Apache Arrow for, among other things, Parquet support, Arrow’s DataFusion Project, which provides SQL and query execution support, Actix, which provides the HTTP interface and Rusoto, the AWS SDK for Rust.

Click through to see how to set it up and how to use it.

Leave a Comment

Distribution Techniques in Azure Synapse Analytics

Gauri Mahajan takes us through three distribution techniques when working with Azure Synapse Analytics dedicated SQL pool tables:

Data warehouses host much larger volumes of data compared to transactional databases, the volume of reads is much more compared to writes and queries tend to result in much larger result sets compared to queries that retrieve scalar values or paginated record sets from transactional databases. Due to this nature of data warehouses, there is a higher impetus on the server to perform faster. Modern data warehouses like AWS Redshift, Azure Synapse, Snowflake and others employ approaches like data sharding where data is distributed horizontally on multiple nodes which process data in parallel. This approach is highly scalable as nodes can be easily added to a data cluster as the storage and performance need increases. One key aspect that is different for tables hosted on such data warehouses is that tables are distributed horizontally using different distribution algorithms, so that all the nodes in an Azure Synapse cluster have an equal share of responsibility for hosting, processing, and delivering data for any given query to maximize performance.

In this article, we will learn about the table distribution styles supported in an Azure Synapse and how to use them for creating distributed tables.

Read on to learn more. This is an example of something we don’t think about on the SQL Server side, so when moving to Azure Synapse Analytics dedicated SQL pools, it can be easy to get this wrong and end up with sub-optimal performance.

Comments closed

Software Development Estimation is Hard

Andy Leonard makes a point:

I recently blogged about punching developers in the brain when a software deliverable deadline is slipping. The title of the post is The Question Unasked. In this post, I would like to address software development estimation.

In the past, I have stated “Either all software developers are pathological liars or software development is inherently inestimable.”

Read on for Andy’s take. For mine, the overly short version is that incentives are set up for software estimation to fail due to perverse incentives and a lack of requirements. Ask the question, “How long will it take to build a skyscraper?” and you might get some answer, but it certainly won’t be the answer to the pertinent question, “How long will it take to build this skyscraper?” That’s because we haven’t talked at all about location, building plans, blueprints, regulatory requirements, financing, or any of the hundreds of other things which organizations talk about and do before they start slapping mortar on bricks. With software development, we’re expected to wing it and solve all of those sorts of problems along the way. Making it even better, more often than not, the skyscraper “pivots” to being a suspension bridge.

Comments closed

An Overview of Bicep

Steve Jones pumps and he pumps:

Bicep is a transpiler, meaning it takes one language and translates it into another. In this case, the Bicep language will move code into the ARM JSON templates. JSON is really for machines, not humans, so the idea is to give sysadmins and developers an easy way to describe resources they need to deploy into Azure.

The language is new, and it’s on Github. This is a DSL (domain specific language), which means it was designed for a specific purpose. With the 0.3 release, the language is built into the Azure CLI and Azure PoSh utilities, so this will do the transpilation for you. There’s also a decompiler to go from an ARM template back to Bicep. It’s also supported by Microsoft, which is always a plus if you need to call for some issue.

Click through for more information.

Comments closed

An Overview of Function-as-a-Service

Grace Ol’Halloran lays out the basics of serverless computing in cloud platforms:

The term serverless computing can be misleading; how can you compute things without a server? Well, the answer is that you don’t. The term “serverless” comes from the idea that the server is abstracted from the developer, and is totally maintained by the cloud provider. In other words, the developer doesn’t really care what environment their code is run in; they just need it hosted somewhere where it can be executed. This removes the responsibility of infrastructure configuration and maintenance from the developer, but naturally gives them less flexibility and control over the environment.

It took me watching several presentations before I really understood the value behind serverless compute.

Comments closed

Where Databases Fit in the Always-Valid Domain Model

Vladimir Khorikov asks an important question:

Today, we’ll talk about an important question: how does the application database fit into the concept of Always-Valid Domain Model?

In other words, is the database part of the always-valid boundary or should you consider it an external system and validate all data coming from it?

Pre-read, my answer was no, databases are part of the external world and your domain model needs to validate every time because who knows what weirdo did something to your data while it slept.

Post-read, well, you’ll have to read to find out.

Comments closed

A Primer on Apache Cassandra Reads and Writes

Utkarsh Upadhyay explains some of the internals of reading and writing with Apache Cassandra:

Apache Cassandra is a type of No-SQL database. It handles large amounts of data across many commodity servers. Being a highly scalable and high-performance distributed database, it provides high availability with no single point of failure. Here in this blog, mainly I focused on Reads and writes in Cassandra. And For Cassandra architecture, you can refer to this blog Apache Casandra: Back to Basics. So let’s get started with this blog on Apache Cassandra: Reads and Writes.

Click through for a comparison between Cassandra and MySQL, followed by a high-level architectural explanation of read and write operations in Cassandra. Though one thing which raises my eyebrow is the statement that reads in Cassandra are O(1). I don’t know that not to be the case, but I’m inclined to say it doesn’t sound right.

1 Comment

Multi-Cloud Pros and Cons

James Serra lays out some of the benefits and drawbacks of using multiple cloud providers:

A discussion I have seen many companies have is if they should be single-cloud (using only one cloud company) or multi-cloud (using more than one cloud company). The three major Cloud Service Providers (CSPs) that companies use for nearly all use cases are Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP).

Without spoiling it too much, James is not really sold on the idea.

Comments closed

The Basics of Event-Driven Architecture

The Aiven team has a nice primer on event-driven architecture:

What happens when one link in the chain goes down? Requests that are waiting for a response don’t receive one at all. They continue to wait, or they time out. The entire application is blocked. What’s more, as the number of services increases, the number of synchronous interactions between them increases as well. In such a situation, a single system’s downtime affects the availability of other systems as well.

An alternative approach is building a microservices application on an event-driven architecture (EDA). Event-driven architecture is made up of decoupled components — producers and consumers — which process events asynchronously, often working through an intermediary, called a broker. That might feel like a mouthful. Don’t worry — we’re going to walk through these concepts one step at a time. In this article, we’re going to look at the components that make up event-driven architecture, why you would use this paradigm, and how to implement it.

Read on to see what makes it so interesting.

Comments closed