Moving from Magento to Microservices-Based Commerce

Back in 2014, I was part of a team that inherited a large custom Magento e-commerce application. I had worked with PHP before (the language Magento is written in), but this was an entirely different beast. On top of Magento being new, the previous development team had not adhered to best practices, injecting business logic and custom queries directly in view files. The whole application was a mess and a security risk.

Soon after I joined, we started to make a plan for migrating the platform to microservices. Besides being difficult to maintain, making product updates in Magento was slow, and the user interface was cumbersome. None of our engineers liked working in it, and while the admin tools allowed our team to modify products, Magento didn’t support all the attributes and product types we needed at the time.

It took about a year to complete the migration, but we gradually reduced and eliminated our Magento e-commerce store. Along the way, we learned a lot about how this migration works and made some mistakes that I’ll try to steer you away from in this guide.

Admittedly, Magento has changed a lot in the past seven years, but its monolithic architecture hasn’t. In this guide, I’ll outline some of the reasons your team might consider the move from Magento to e-commerce microservices and the steps you can take to start this migration. Along the way, you’ll see which elements of Magento you might replace first and some options in case you don’t want to build all of your microservices from scratch.

Why Microservices?

The most obvious question when considering a major move like this is, “why?” Moving away from a monolithic commerce platform is no small matter, but there are a few key reasons you might consider moving from Magento to microservices:

• Security: Magento—especially the self-managed, open source version—has always been a security nightmare. Sixty-two percent of Magento stores operating today have an unaddressed security vulnerability which might leave customer or company data vulnerable to hackers.
• Technical Complexity: For many e-commerce stores, Magento adds a lot of overhead that probably isn’t necessary. For example, instead of adding attributes directly to products, developers have to hook into its EAV model to add product features. This slows down development and means deep, Magento-specific knowledge is required.
• Flexibility: As with all monolithic platforms, it can be hard to granularly scale Magento as your e-commerce application grows. Installations are typically made on a single server, and Magento doesn’t give you much room to swap out databases or add caching layers based on your needs.

Moving from Magento to Microservices

While the exact process you use will vary depending on whether you’re using the open source or hosted version of Magento and how many extensions you’re using, the following is a general look at the steps you’ll need to take during your migration.

As with any significant architecture change, I don’t recommend a grand rewrite. Instead, figure out how you’ll untangle the monolithic mess and slowly, piece by piece, move it over to microservices.

The following steps are roughly what my team did back when we made this migration a few years ago, so hopefully, this gives you a starting point for your migration today.

1. Decide where to start the migration

One of the hardest parts about unraveling a monolith is deciding where to start. There are two approaches that teams typically take:

Option 1: Start with the area that’s easiest to separate and migrate

For example, if you have product reviews on your Magento site and they’re decoupled from the core product data, you might want to start here. Migrating product reviews to a new microservice will help you get the process started, but it probably doesn’t provide a ton of value to your business (unless product reviews happened to be limiting growth).

Option 2: Start with the area that provides the most business value

This was the approach my team took. We had several pieces of functionality that Magento made exceedingly hard to implement, but one big one was our in-app e-reader. We needed to allow customers to sign up, purchase a book, and then read it on our platform, but Magento wasn’t built with this use case in mind.

Our engineering team started by dismantling the e-reader we hacked into Magento and moving it into a new single-page application backed by a reader microservice. At first, that service called Magento’s API to ensure the user was authenticated and had access to the book, but eventually, we migrated those services too.

Your team’s challenges will be unique. For example, if you’re trying to improve your store’s conversion rate, you might want to start with the checkout. If you’re trying to improve signup and authentication, you might start with an authentication microservice. By commencing your microservice migration with the investment that brings the most business value, you’ll build trust and give leadership a reason to continue investing in the process.

At this stage, you should also make the build vs. buy decision. One of the advantages of microservices is that you can mix services from internal teams with those from third-party providers. This allows you to pull in a PIM (product information manager) from a company like fabric while building your own in-house order management system if you prefer (although fabric offers an OMS too).

2. Build a middle layer

You’ll likely want to make requests to Magento from your new microservices or new user interface during your migration. In this case, an API abstraction layer is likely a good investment.

This piece—sometimes called an API gateway—acts as a router to pass all requests to either the monolith or microservices and allows you to run both in tandem while you migrate.

We did not start off using an API gateway. Instead, we made each service communicate directly with our frontend, making changes harder to manage as we moved to our microservices.

Another element you’ll need to build as you migrate more of your functionality to microservices is a service mesh. Your service mesh handles some of the shared operational functions typically absent from microservices (e.g., logging, load balancing, routing) and helps minimize the need to duplicate functions like authentication or authorization between services.

Whether you build your own or use fabric for your e-commerce microservices, you need a solution that includes scalable networking, logging, and security models. This will ensure that latency and poor visibility don’t stop your progress towards microservices.

3. Migrate the data model

Depending on which piece of your Magento application you decide to migrate first, getting your data out of Magento and into your new microservices can be quite a challenge.

During our migration, getting data from some of the supporting pieces of Magento wasn’t too hard, but migrating the core product catalog proved to be a big job. We wanted to move from Magento’s fragmented EAV model to a flatter, fixed model specific to our domain.

If your product catalog is simple, you might be able to use the admin-facing Magento export feature, but this option falls short if you have custom product types (like downloadable or bundled products). It also doesn’t include quantity data, orders, or carts, so you’ll have to export those separately.

You’ll likely have to write some custom PHP or SQL code to export much of your data from Magento. Because of Magento’s complex EAV table system, it’s much easier to export data if you first switch to their flat catalog. After that, your product catalog will be cached into a single flat table which is easier to export and query against.

Next, you have to export data for orders, customers, carts, etc. Doing so will be a tedious process as each requires its own SQL query. For example, this query will get your orders with their payment and shipment IDs:

SELECT sales_order.entity_id AS "Order ID",
    sales_order_payment.entity_id AS "Payment ID",
    sales_shipment.entity_id AS "Shipment ID"
FROM sales_order
     LEFT JOIN sales_order_payment ON (sales_order.entity_id = sales_order_payment.parent_id)
     LEFT JOIN sales_shipment ON (sales_order.entity_id = sales_shipment.entity_id)
     # Note: you can add any filters as WHERE queries here
ORDER BY sales_order.created_at

Now you can export each of these query results to a .csv file and import them into your new microservice. This import process will vary depending on the language of your new service and shape of your new database, but it could be a simple Node script.

This example uses the popular csv-parse package:

const db = require('./database');
const fs = require('fs');
const parse = require('csv-parse');
 
const parser = parse({columns: true}, function (err, orders) {
    orders.forEach(function (order) {
        // You can transform your data or save it here
        db.save(order);
    });
});
 
fs.createReadStream(__dirname+'/orders.csv').pipe(parser);

If you want to run both Magento and your new service in tandem, you’ll need to set up a data pipeline so that records are piped to your new microservice whenever they are added. MySQL doesn’t support triggers on views, so you’ll have to set up a service to check for updates and pipe them over to your new database.

4. Test, replace, repeat

As we migrated pieces of our infrastructure from Magento to microservices, we tested the changes and kept a close eye on the logs. If we saw anything that looked out of place, we’d roll back or quickly push a fix to the broken service. We repeated this process for each major piece of our e-commerce store until all the microservices were rolled out.

Our application was relatively small, but it still took our team almost a year to complete the migration. During the process, we sped up and simplified our application significantly, and ultimately helped the business learn more and iterate on our offerings faster.

Conclusion

As you’ve seen, migrating to microservices from Magento is no small task. It could take months or years, but if you can start small with the pieces that provide the most business value, it’s a worthwhile investment.

On the other hand, if you’d prefer not to make this migration on your own, let our team know. fabric offers a scalable microservice-based solution for e-commerce stores that need to scale up. We can help you carry out a migration from your existing Magento store and ensure that your business continues to grow along the way.

Karl Hughes

Tech advocate and writer @ fabric. Previously CTO @ Graide Network.