Magento 2.4.0 reliability and Composer musings
Our first episode, when we talk about the non-marketing details of Magento 2.4, some speed improvements, and Composer.
Show notes
(full transcript below show notes)
Yireo - The unstable release of Magento 2.4.0
Magento Semantic Version Checker
(full transcript below show notes)
Yireo - The unstable release of Magento 2.4.0
Magento Semantic Version Checker
hirak/prestissimo
Composer 2
narrowspark/automatic-composer-prefetcher
Magento 2.3.5 + Content Security Policy (CSP): A Fool's Errand
Private Packagist
Satis
Magento 2 Marketplace Mirror by Fooman
Ivan Chepurnyi on Twitter
Stephan Hochdörfer on Twitter
Full transcript
PJ: All right, everybody. You're welcome at the first Registry podcast. The Registry podcast is done by me, Peter Jaap Blaakmeer
Jisse: And me, Jisse Reitsma
PJ: We're two Dutch Magento developers. I run an agency called Elgentos, and Jisse what do you do?
Jisse: I run an agency called Yireo and what I do is actually I'm not a real agency, but more like a trainer of developers.
PJ: Yeah. I've actually received training from you.
Jisse: Yeah, you did, right? So, sometime back on react. So yeah, I'm kind of busy with a lot of front end stuff, but I think the initial thoughts for this podcast, the Registry is to talk about back end development instead. But yeah, it's your ID so why is it called the Registry in the first place?
PJ: I needed a name that struck a chord with Magento developers and since the Registry is deprecated in Magento too I thought, “Let's just re-live the name if we use it.”
Jisse: Yeah. So the Registry really feels like old. To me, there's also this reminder towards the old Windows age. I'm not sure if Windows is still using a registry. I don’t know anymore.
PJ: Isn't it register, like register.exe?
PJ: Well, it's register.xe or it's referred to as the registry, I thought. But anyway, there's a podcast now called The Registry.
PJ: Yep. And we're going to talk about, well, you said mainly back end development and I guess that's debatable. I'm a back end developer first and foremost, you are originally a back end developer, but you've gravitated towards the front-end a bit more lately.
Jisse: Yeah. Well, and that's maybe also because there's kind of a need within the front end developers to discuss, basically the alternatives that you have for Magento 2 front-ending because there's a lot of things to tell about the Magento 2 frontends. I simply just dive into it a little bit more. But yeah, maybe that's the whole idea about this Registry as well, to talk about interesting stuff to either back end developers or front end developers.
PJ: Yeah or front end stuff for back end developers, which is hard for us.
Jisse: Yeah or back end stuff for front-enders.
PJ: I always have the feeling that's easier than the other way around, but that might be my bias being a back end developer.
Jisse: Yeah. Yeah. But yeah, let's keep the current episode one a little bit focused on back end development stats, but also maybe it may be some general news. So I think when discussing what kind of topics to pick up on I think it was kind of obvious to tell at least a little bit about Magento 2.4, which was just released one week, two weeks ago. I don't know anymore, but recently, a little bit longer, right?
PJ: Yeah.
Jisse: Time flies. But anyway, instead of just doing all of the marketing fluff and saying like, “Oh, Magento 2.4.0 is wonderful. It's great. You should upgrade. I think in this podcast, we want to be a little bit more developer minded or honest or
PJ: Critical?
Jisse: I think so, yeah. So there are multiple features like the MSI performance that went up by refactoring a lot of things. There was this storyline on Twitter telling basically that the dozens of queries were all reduced back into just a couple of queries needed on the front end. Well, that's of course, good news, but it also makes you wonder whether the MSI package was before 2.4.0, then not just really a performance hog.
PJ: It was.
Jisse: Yeah, apparently. So, it's good that it is fixed. Well--
PJ: Have you worked with MSI a lot?
Jisse: No, actually just a little bit, just trying to integrate things with it. But, yeah, I have to be honest, I'm a trainer, so I'm not really diving into projects myself that much, and if I do, that's always a specific area and unfortunately, MSI is still on my to-do list because I like the codebase. I like the way that things are set up in a clean way, that a domain-driven development is implemented and then IDs, like CQRS come into being, but I have to admit that once I see all of this functionality in the codes, I wonder also how difficult it is to configure everything in a real-life project.
PJ: Yeah. Well, we've played around with it. We did one implementation and it kind of backfired at that point because we had some synchronization issues with an ERP coming in and then you have to disable the stock reservations and they need a separate extension for that. So it got muddy at one point and we decided to take a different approach. So in the end we don't use MSI at this point in time, so I can't really say anything about the performance in 2.4.
Jisse: No, but you’re saying currently then you're using the old catalog inventory?
PJ: So basically, yeah. We pull in necessary information from an ERP when we need it and where we need it, but it's not integrated in MSI.
Jisse: So, what I wonder about all of the new features that come into Magento more and more and more is whether all of those agencies or whether all of those developers and merchants out there actually need all of those features at all. Some people do, and of course, it's a wonderful thing that MSI was developed and that there's so much opportunity suddenly with all of those modules. But there's still, I think also a lot of merchants out there that only need a simple inventory management, so basically the old catalog inventory. And now that's slowly being deprecated, which makes me wonder, like, okay, but is the community then maybe able to pick up on maintaining such a simple inventory for a longer time? I think it should be doable, but there still needs to be somebody to pick up on it.
PJ: Yeah. I'm curious to see when MSI will completely replace the old catalog inventory module. Not sure it will eventually happen at all.
Jisse: No. Well, a similar story that is there, of course, is that MySQL, as a search engine has been dropped in favor of Elasticsearch. And again, I think this was clearly announced by Magento as well in all of the release notes about 2.4. So I'm not blaming them for anything, but it's more that I wonder how many merchants are actually needing this and how many merchants are just frustrated about this because they still want to use the old MySQL search.
PJ: Yeah well, I thought the other way. I thought, “Everybody's already using Elasticsearch, let's just kill off MySQL full-text search anyway.
Jisse: Yeah. So, on the technical side, I think it's a good ID, but I simply don't know how many merchants would still be using the old search.
PJ: I think it's okay because Elasticsearch is very easy to configure. All the big hosting companies have Elasticsearch even in their cheapest packages, so I don't think that's a really big deal. MSI is a bit more complex to implement. So that might be a bigger barrier when you compare it with MySQL full-text search versus Elasticsearch.
Jisse: Yeah, exactly. Well, and other than that, 2.4 also ships with silly things like two-factor authentication, and I've seen already three modules two days disable that again.
PJ: Yeah, well, there's nothing wrong with two-factor authentication per se. The problem is that they kind of hardcoded it in the admin and kind of force it upon you. So a lot of developers who upgraded to 2.4 were unpleasantly surprised by this sudden hard requirement to do a, to have a 2FA.
Jisse: It’s pretty cool that there are modules out there already to help you disable the stuff again as well.
PJ: Yeah, I figured it's great that Magento bundles 2FA with Magento, but it's not great that they force it upon you. And I get the reasoning behind it, but it's such a big struggle for developers to now get it started even on a clean install, that it might've been wiser to make it an option to enable it.
Jisse: Yeah. And then maybe another thing that is noteworthy is that there's a security text module for, I guess those people that are unable to just create a security built text file public folder. But it's actually leading me into something that was already there in 2.3. 4 or 5 CSP which stands for Content Security Policy, which is also a cool feature if you think about it, but it's just causing so many difference or warnings--
PJ: Horribly implemented.
Jisse: Yeah, it’s horribly implemented as well.
PJ: Yeah. Have you read Max Chadwick's post on CSP in Magento?
Jisse: I did, but I forgot--
PJ: He burned it alive. Yes. It's at maxchadwick.xyz I think it is? Yeah. He has a blog post on CSP and why it is a terrible idea, or not a terrible idea, but why it's a terrible implementation on Magento and how to disable it.
Jisse: So maybe it's a good idea to include a couple of those links that we talk about, and share them with the audience later.
PJ: Yeah.
Jisse: Yeah, so, further than that, I think there are a couple of things that might be related to performance when it comes to 2.4. So for instance, PHP 7.4 came out, or what supported now, MySQL 8 is supported like we mentioned--
PJ: We're running Magento 2.3.5 on 7.4 as well. So I'm not sure if it officially supports it now, but it worked anyway, and in the back end where something weird was happening, but we haven't run into any issues.
Jisse: But with my SQL 8? Are you also able to run that?
PJ: Nope, haven't tried yet.
Jisse: No, I believe that's actually something related to the customer authentication mechanism or password verification mechanism, or I should say password encryption mechanism underneath in MySQL. I thought that had to do something with that.
PJ: It's going to be interesting to see how fast MySQL 8 adoption will be because one of the big things in MySQL 8 is for a pure functional point of view, is the ability to store JSONs in MySQL. I’m curious to see how fast extension providers will make use of that feature in MySQL 8 and force people to upgrade.
Jisse: Yeah. Well, other than that, like the performance of MySQL 8 is supposed to be a little bit better regarding read statements, so simply just to select queries. However, I saw benchmarks a while back, I think it was at the first release of MySQL 8, that actually proved that MySQL 5.7 was more performant than MySQL 8. So I'm just wondering whether the community indeed is going to adopt this as soon as possible. But yeah, so I'm personally maintaining a couple of modules myself and I think you guys are as well with the elgentos. And from my side, most of the work that I needed to do to make all of my modules compatible with 2.4, was just to upgrade all of the PHP unit tests which was kind of a clumsy thing because the rest of the code was working, but I still needed a couple of hours just to make sure that was compatible. I'm not sure if you experienced anything like any work needed for your own modules?
PJ: I'm going to let you in on a little secret here. We haven't upgraded to 2.4.0 for any of our clients. We tried to upgrade a few, and ran into some issues that everybody ran into. We looked at the growing list of bugs on GitHub, and we decided, “No, we're not going to do a 0.0 release or a 0.0.0 release in this case.” We're going to wait it out, weather the storm, and we're going to upgrade to 2.35 P2. And then wait until 2.4.1 is released and then upgrade to that one.
Jisse: Yeah. So I'm seeing a lot of different features that are actually postponed to 2.4.1 but you're also saying that there are quite a lot of bugs, at least by monitoring the issue list on GitHub.
PJ: Yeah. And I've heard from Ivan Chepurnyi, that some performance issues, which, you know, everything is slow according to Ivan.
Jisse: He's always finding performance issues. That’s his secret power.
PJ: He had some stuff which made me rethink our initial goal to upgrade everything to 2.4.0. And, let's just say that Magento's history with 0.0, releases isn't that stellar.
Jisse: No, I hear this more with agencies that over time they basically learned from it. Well, I was there as well with 2.0, 2.1, 2.2, and I thought, actually with 2.3, that the whole thing improved pretty much. So I actually upgraded to 2.3.0. as soon as it came out and it was a little bit troublesome, but not a huge amount of bugs. And I honestly don't know about the States with 2.4.0, whether it's a lot worse or better or--
PJ: You know what I think the problem is with the 0.0 releases, every once in a while, Magento of course, needs to bump from 2.3 to 2.4 or whatever to do a double release. And they have to stuff it with new functionality, otherwise it wouldn't warrant a minor upgrade. So they focus on new features and I have the feeling that bug fixes are put on a sidetrack in that process. And then they pick up all the old bug fixes for 2.0 for the top of the one release again. That's been my feeling with 2.2.0 and 2.3.0. And, yeah, of course, the platform itself has become a lot more mature and stable since the 2.2 or even the 2.1 or 2.0 era, so there's definitely a big improvement in terms of stability. But stability is always relative, and if we can prevent some of these issues in the double release by just waiting it out for one version for one quarter, then we're okay with that.
Jisse: Yeah, so personally, I encountered issues with bundled extensions, but that again was mostly on the testing side with integration testing and unit testing. I became pretty angry about that and I won't name the vendor anymore, but, to put adults in the specific place, I was a pretty mailer about it. That's a reference to who it was, but anyway I was pretty upset because as a third party extension provider, I'm trying to do all of that work to make sure that the extension is fitting in nicely with 2.4. And actually, because I'm not an official partner, I didn't see any kind of pre-release either. So actually on the day when 2.4 came out, I worked my ass off just to make sure that everything was patched properly. And I prepared things, but I couldn't release my new modules because I didn't know the exact semantic versioning of all of the new stuff. And to me, it was kind of disappointing to hear that basically, some bundled extensions gave issues. So you also encountered issues there?
PJ: I think the bundles extension are the main source of problems. I totally disagree with what Adobe or Magento is doing there bundling third-party extensions in there in a core of Magento. I can see it from a commercial standpoint and I hope they earn a lot of big bucks for it…
Jisse: They do.
PJ: …Because otherwise, it wouldn't be worth the trouble that those extensions are introducing in codebase.
Jisse: Yeah. So I'm personally running a repository with replace packages for some packages from the core, MSI and etc., but also this replace package for bundled extensions. I think that's actually one of my most popular packages out there. And it's not even me because I believe that actually integer.net came up with that original suggestion.
PJ: Yeah, you stole it.
Jisse: Yeah, I simply stole it, but I package it in such a way that it was more useful. But it's kind of interesting to see that such a thing is needed. To me, it’s always funny that Magento as a commercial entity goes that far into bumbling basically all of those bundled extensions in Magento as a core, and then actually all of us community members try to get rid of them again. They make you feel like, "Hey, but is there maybe a different approach?
PJ: Less code is less complexity.
Jisse: Yeah, I think so. Well, the other news was that Magento also released their own semantic versioning checker just a week ago or so. So basically they used this tool internally to prepare the Magento 2.4 release just to determine what kind of Git commits were made on the repositories and how it would translate into new semantic versions for all of the different packages. And I think it's worth just to see all of those tools become open source, and hopefully, actually, that all of the third-party extension developers also stick with those practices. Similarly, like the PHP coding standards, or unit testing, integration testing. Still, it's basically all of those bugs in, theoretically, the bundled extension, but it might also be other extensions, all of those bugs that prevent people from upgrading basically shows also that there's still a strong need for better extension quality, I guess.
PJ: Yep. Okay.
Jisse: Yeah. So, what else is up?
PJ: Composer 2.
Jisse: Yeah, so I've played around with Composer 2 myself already a little bit, but not with Magento because I tried to do this, I don't know, like months ago already. Stephan Hochdörfer, he also communicated with the composer team on how to upgrade to Composer 2 and then make it work with Magento 2. Well, I tried to follow along, but it didn't work out for me and then I just let go for that moment. I haven't tried it since, but, yeah, it's said to be improving the download times tremendously. So that would be awesome.
PJ: Yeah. I haven't tested it with Magento either, but we have tested it on Laravel projects and yeah, the faster download times is of course number one, but package resolving is also a lot faster, the SAT resolving. Plus you can speed up your resolving by setting up - let's see how they call it. I can't remember what they call it, but you can set up like filtered repositories, where you can say, “I don't want this package from this repository, but I want it from another one” or you can use only or accept, so it doesn't search through the entire repository for the package you need. You can aim it towards a certain package within the repository.
Jisse: It sounds a little bit like the replace trick that we just mentioned, but now actually more on steroids. And I heard something before that the replace trick would have been removed also in Composer 2, but I didn't check up on that anymore.
PJ: Right, well, Composer 2 has repository priorities. So, if replace doesn't work anymore in Composer 2, you might be able to create your own repository with the exact name of those packages, and then place that repository as your highest priority, and then it will fetch and like an empty file. Let's say, if you replace the MSI module then in the exact pathway, would otherwise be written a file that says, "This package has been removed by the composer repository that's been added for composer notification or something. That's repository priorities. That's pretty useful because we have a third party extension we use from a vendor, we have their composer repository in our Private Packagist, and then we need our own fork of it. So now we have to rename the package and use the new package name, but then we can just assign a priority to our own repository packages-- It's pretty cool.
Jisse: Yeah, that's pretty cool. What does it feels like with Composer 2? they simply just listened to all of the hacks and workarounds that people try to make under Composer 1? So for instance, a pretty common tool nowadays with Composer 1 is prestissimo. But, yeah, that's becoming obsolete as soon as you are under Composer 2, but that's only for the download part, to parallel, the downloads. Yeah, and I read something about this repository priorities and I remember that I thought like, "Oh, this is creating cool opportunities", but I haven't played around with it.
PJ: A prefetcher as well. I didn't know about that one.
Jisse: No, it's actually not. It's not a Composer 2 related, but it's a Composer 1 plugin just like prestissimo, and it's basically trying to do a real prefetch of all of these different packages. So that's kind of like, the http2 caching mechanism is able to prefetch certain resources in faster ways, supposedly. But in Symfony project, I actually have used this and I experienced an improvement in Magento 2. I've already had a couple of times, that the Magento 2 installation, which actually stuck because of some kind of internal thingy that they're trying to do. So, yeah, there are third-party tools basically within composer trying to improve performance. But I think it's all done by just a couple of individual developers maybe, and trying to come up with smart things. But I'd rather just wait for also Composer 2 to come out with.
PJ: Yeah. So I'm looking at the read me and what the prefetcher does is you can tell composer to only look for packages, for instance of packages in a certain version release line. So say 2.X and it won’t look in 1.X. This feels like something that could be added into Private Packagist as well. Do you use Private Packagist or Satis or--?
Jisse: Well, I'm using my own Satis instance and I'm just using Private Packagist because of some companies like Elgentos.
PJ: We're a big fan. We'll Satis does the exact same thing basically, although and Nils and Jordi (from Packagist) are adding some pretty cool features on top of price packages, like security audits. So it will give a security warning if a package has a known vulnerability in a certain version, but prefetching... So this prefetching feels like a perfect addition to Private Packagist where you could say, “Okay, I want to have this package inside my Private Packagist, but I don't care about all the old versions. I just need free 3.X and upwards or some things to really speed up the SAT dependency lookup.”
Jisse: Yeah, exactly or something similar. So I played around with it. I tried to configure it a little bit. So the example in the read me actually goes with the symfony symfony and another package. So I tried this for Magento packages. But yeah, so in my case, it kind of blew up. It seems to resolve things in a loop or something. But yeah, and talking about Private Packagist, I played around with also the feature of mirroring the Magento marketplace, improving download speeds. Now, I thought it would be really like a click click system where you would say like, “Oh, but this specific package, or actually all of the packages on the marketplace, let's copy them, download them and use them within Private Packagist instead.” I was a little bit disappointed that it didn’t work exactly that way, but yeah.
PJ: I think you can because it will add the package to your Private Packagist subrepository as soon as you require it. I think you can also manually add it by putting in a composer.json or a composer.lock.
Jisse: Yeah. But then still my impression was actually that it would only just do--
PJ: It will do a full copy.
Jisse: Yeah, of a single package, but not all of the dependencies with a package either. So actually--
PJ: The problem arises when you have a username and password combination for a marketplace mirror. We have a bunch of clients with each have their own Magento marketplace username password combination because they purchase extensions in the marketplace sometimes even though we tell them not to, and then we need to get them from composer. So every client needs to have its own unique marketplace mirror within Private Packagist. So if it were the case that it would completely mirror it, then we'd have like thousands of packages. Yeah.
Jisse: Yeah. So how do you solve then this issue of a marketplace performance? You're not or--?
PJ: We try not to use the marketplace for extensions.
Jisse: Okay, but still for the core you're using it anyway.
PJ: Yeah. It does mirror it and catch it on Private Packagist.
Jisse: Of course, yeah. That's true. Yeah.
PJ: Once you've required it, it will create a mirror for the package you have required, just not for the entire market place.
Jisse: No. And it's actually pretty cool that it's kind of like catching on demand without you doing anything for it. So, in the past, Alan Storm wrote a long, long time ago, I believe it was really like five years ago or so, a tutorial on how to mirror the marketplace by using Satis. Then I played around with it myself and I wrote a followup blog on literally what to do to make this happen because Alan Storm is never just saying like, “Well, this is what you should do.” He's always writing a tutorial in-depth and just going as deep as possible. But yeah, the bottom line was that I had my own private mirror of the marketplace and I was amazed with how good the performance was. So instead of actually just hosting a mirror, like with Private Packagist in Helsinki or Stuttgart, or I don't know where the packages the mirrors are, but then I chose basically my own private VPS instead then the download times were just reduced to almost nothing. So that was pretty cool.
PJ: Right. Yeah. So you do know that Kristoff from Fooman has a Magento marketplace mirror, right?
Jisse: Yeah. But, I don't know, again also where his mirror is actually located.
PJ: Good point.
Jisse: The cloud?
PJ: The URL ends on .nz so it might be the exact opposite of the world.
Jisse: New Zealand. Exactly, yeah. And I think the main point about playing around with Private Packagist or playing around with Satis yourself, is that you could theoretically just reduce the download times because the Magento marketplace itself is definitely located in the US, I believe.
PJ: It's here in Kristoff’s blog. The mirror is using AWS infrastructure, including CloudFront CDN.
Jisse: Okay. So that would be pretty good.
PJ: When I traceroute it, it ends up in AMS 54 CloudFront. So in our case, it's actually being served from Amsterdam.
Jisse: Exactly. Yeah. And on top of it what Kristoff solution, Fooman’s solution actually has to offer is of course also removing the authentication of all of the open source free packages. So why authenticate if all of those composer packages are freely available anyway?
PJ: He does warn not to use it in production though. I guess that’s more for security.
Jisse: Yeah, but that's just a disclaimer. Yeah, exactly. He doesn’t want to be responsible for it.
PJ: “When I get hacked and somebody injects their own code into my repository, I'm not responsible. “
Jisse: Yeah, exactly. It's just to prevent that the whole community has come to the New Zealand guy.
PJ: He can probably see who is doing it. In access logs or something.
Jisse: Yeah, cool. And also on top of it--
PJ: I think it’s time, Jisse.
Jisse: Oh yeah. So yeah, it's already half an hour, you mean?
PJ: Yeah.
Jisse: Cool.
PJ: We tried to do half an hour but we can easily fill up a full hour, but I think--
Jisse: Or a half-day or something.
PJ: I think we could go on for weeks on end.
Jisse: Yeah, exactly. There's so much in this Registry.
PJ: But we have a ton about Composer left on our cheat sheet, but I think we could just move that to the next episode.
Jisse: Yeah. And I think this is also the goal that we tried to reach, right? Simply to have a nice little conversation between two Dutch guys that among them with the two of them know quite a bit. And then if you combine that all together, then hopefully, it's a nice, interesting podcast that hopefully, some people find useful.
PJ: And if you do, and if you don’t, hit us up on Twitter @peterjaap and @yireo, or @jissereitsma.
Jisse: Yeah, but we'll post, I think, most of those details anyway, and let us know as well whether it's interesting, whether there should be an episode 2. It could also be that basically a lot of people are saying like, “Hey guys, just don't do this.”
PJ: Another podcast.
Jisse: Not another podcast.
PJ: I have podcast burn out from listening. So that's why I thought, “Let's just create one.”
Jisse: Yeah, exactly. And top of it, I think having a podcast without the bullshit would also be good. So hopefully--
PJ: By backenders is for backenders.
Jisse: Exactly. For nerds, by nerds or something. Yeah.
PJ: Cool. See you next time.
Jisse: Thanks for listening and catch up with you soon.
PJ: Bye. Bye.
Composer 2
narrowspark/automatic-composer-prefetcher
Magento 2.3.5 + Content Security Policy (CSP): A Fool's Errand
Private Packagist
Satis
Magento 2 Marketplace Mirror by Fooman
Ivan Chepurnyi on Twitter
Stephan Hochdörfer on Twitter
Full transcript
PJ: All right, everybody. You're welcome at the first Registry podcast. The Registry podcast is done by me, Peter Jaap Blaakmeer
Jisse: And me, Jisse Reitsma
PJ: We're two Dutch Magento developers. I run an agency called Elgentos, and Jisse what do you do?
Jisse: I run an agency called Yireo and what I do is actually I'm not a real agency, but more like a trainer of developers.
PJ: Yeah. I've actually received training from you.
Jisse: Yeah, you did, right? So, sometime back on react. So yeah, I'm kind of busy with a lot of front end stuff, but I think the initial thoughts for this podcast, the Registry is to talk about back end development instead. But yeah, it's your ID so why is it called the Registry in the first place?
PJ: I needed a name that struck a chord with Magento developers and since the Registry is deprecated in Magento too I thought, “Let's just re-live the name if we use it.”
Jisse: Yeah. So the Registry really feels like old. To me, there's also this reminder towards the old Windows age. I'm not sure if Windows is still using a registry. I don’t know anymore.
PJ: Isn't it register, like register.exe?
PJ: Well, it's register.xe or it's referred to as the registry, I thought. But anyway, there's a podcast now called The Registry.
PJ: Yep. And we're going to talk about, well, you said mainly back end development and I guess that's debatable. I'm a back end developer first and foremost, you are originally a back end developer, but you've gravitated towards the front-end a bit more lately.
Jisse: Yeah. Well, and that's maybe also because there's kind of a need within the front end developers to discuss, basically the alternatives that you have for Magento 2 front-ending because there's a lot of things to tell about the Magento 2 frontends. I simply just dive into it a little bit more. But yeah, maybe that's the whole idea about this Registry as well, to talk about interesting stuff to either back end developers or front end developers.
PJ: Yeah or front end stuff for back end developers, which is hard for us.
Jisse: Yeah or back end stuff for front-enders.
PJ: I always have the feeling that's easier than the other way around, but that might be my bias being a back end developer.
Jisse: Yeah. Yeah. But yeah, let's keep the current episode one a little bit focused on back end development stats, but also maybe it may be some general news. So I think when discussing what kind of topics to pick up on I think it was kind of obvious to tell at least a little bit about Magento 2.4, which was just released one week, two weeks ago. I don't know anymore, but recently, a little bit longer, right?
PJ: Yeah.
Jisse: Time flies. But anyway, instead of just doing all of the marketing fluff and saying like, “Oh, Magento 2.4.0 is wonderful. It's great. You should upgrade. I think in this podcast, we want to be a little bit more developer minded or honest or
PJ: Critical?
Jisse: I think so, yeah. So there are multiple features like the MSI performance that went up by refactoring a lot of things. There was this storyline on Twitter telling basically that the dozens of queries were all reduced back into just a couple of queries needed on the front end. Well, that's of course, good news, but it also makes you wonder whether the MSI package was before 2.4.0, then not just really a performance hog.
PJ: It was.
Jisse: Yeah, apparently. So, it's good that it is fixed. Well--
PJ: Have you worked with MSI a lot?
Jisse: No, actually just a little bit, just trying to integrate things with it. But, yeah, I have to be honest, I'm a trainer, so I'm not really diving into projects myself that much, and if I do, that's always a specific area and unfortunately, MSI is still on my to-do list because I like the codebase. I like the way that things are set up in a clean way, that a domain-driven development is implemented and then IDs, like CQRS come into being, but I have to admit that once I see all of this functionality in the codes, I wonder also how difficult it is to configure everything in a real-life project.
PJ: Yeah. Well, we've played around with it. We did one implementation and it kind of backfired at that point because we had some synchronization issues with an ERP coming in and then you have to disable the stock reservations and they need a separate extension for that. So it got muddy at one point and we decided to take a different approach. So in the end we don't use MSI at this point in time, so I can't really say anything about the performance in 2.4.
Jisse: No, but you’re saying currently then you're using the old catalog inventory?
PJ: So basically, yeah. We pull in necessary information from an ERP when we need it and where we need it, but it's not integrated in MSI.
Jisse: So, what I wonder about all of the new features that come into Magento more and more and more is whether all of those agencies or whether all of those developers and merchants out there actually need all of those features at all. Some people do, and of course, it's a wonderful thing that MSI was developed and that there's so much opportunity suddenly with all of those modules. But there's still, I think also a lot of merchants out there that only need a simple inventory management, so basically the old catalog inventory. And now that's slowly being deprecated, which makes me wonder, like, okay, but is the community then maybe able to pick up on maintaining such a simple inventory for a longer time? I think it should be doable, but there still needs to be somebody to pick up on it.
PJ: Yeah. I'm curious to see when MSI will completely replace the old catalog inventory module. Not sure it will eventually happen at all.
Jisse: No. Well, a similar story that is there, of course, is that MySQL, as a search engine has been dropped in favor of Elasticsearch. And again, I think this was clearly announced by Magento as well in all of the release notes about 2.4. So I'm not blaming them for anything, but it's more that I wonder how many merchants are actually needing this and how many merchants are just frustrated about this because they still want to use the old MySQL search.
PJ: Yeah well, I thought the other way. I thought, “Everybody's already using Elasticsearch, let's just kill off MySQL full-text search anyway.
Jisse: Yeah. So, on the technical side, I think it's a good ID, but I simply don't know how many merchants would still be using the old search.
PJ: I think it's okay because Elasticsearch is very easy to configure. All the big hosting companies have Elasticsearch even in their cheapest packages, so I don't think that's a really big deal. MSI is a bit more complex to implement. So that might be a bigger barrier when you compare it with MySQL full-text search versus Elasticsearch.
Jisse: Yeah, exactly. Well, and other than that, 2.4 also ships with silly things like two-factor authentication, and I've seen already three modules two days disable that again.
PJ: Yeah, well, there's nothing wrong with two-factor authentication per se. The problem is that they kind of hardcoded it in the admin and kind of force it upon you. So a lot of developers who upgraded to 2.4 were unpleasantly surprised by this sudden hard requirement to do a, to have a 2FA.
Jisse: It’s pretty cool that there are modules out there already to help you disable the stuff again as well.
PJ: Yeah, I figured it's great that Magento bundles 2FA with Magento, but it's not great that they force it upon you. And I get the reasoning behind it, but it's such a big struggle for developers to now get it started even on a clean install, that it might've been wiser to make it an option to enable it.
Jisse: Yeah. And then maybe another thing that is noteworthy is that there's a security text module for, I guess those people that are unable to just create a security built text file public folder. But it's actually leading me into something that was already there in 2.3. 4 or 5 CSP which stands for Content Security Policy, which is also a cool feature if you think about it, but it's just causing so many difference or warnings--
PJ: Horribly implemented.
Jisse: Yeah, it’s horribly implemented as well.
PJ: Yeah. Have you read Max Chadwick's post on CSP in Magento?
Jisse: I did, but I forgot--
PJ: He burned it alive. Yes. It's at maxchadwick.xyz I think it is? Yeah. He has a blog post on CSP and why it is a terrible idea, or not a terrible idea, but why it's a terrible implementation on Magento and how to disable it.
Jisse: So maybe it's a good idea to include a couple of those links that we talk about, and share them with the audience later.
PJ: Yeah.
Jisse: Yeah, so, further than that, I think there are a couple of things that might be related to performance when it comes to 2.4. So for instance, PHP 7.4 came out, or what supported now, MySQL 8 is supported like we mentioned--
PJ: We're running Magento 2.3.5 on 7.4 as well. So I'm not sure if it officially supports it now, but it worked anyway, and in the back end where something weird was happening, but we haven't run into any issues.
Jisse: But with my SQL 8? Are you also able to run that?
PJ: Nope, haven't tried yet.
Jisse: No, I believe that's actually something related to the customer authentication mechanism or password verification mechanism, or I should say password encryption mechanism underneath in MySQL. I thought that had to do something with that.
PJ: It's going to be interesting to see how fast MySQL 8 adoption will be because one of the big things in MySQL 8 is for a pure functional point of view, is the ability to store JSONs in MySQL. I’m curious to see how fast extension providers will make use of that feature in MySQL 8 and force people to upgrade.
Jisse: Yeah. Well, other than that, like the performance of MySQL 8 is supposed to be a little bit better regarding read statements, so simply just to select queries. However, I saw benchmarks a while back, I think it was at the first release of MySQL 8, that actually proved that MySQL 5.7 was more performant than MySQL 8. So I'm just wondering whether the community indeed is going to adopt this as soon as possible. But yeah, so I'm personally maintaining a couple of modules myself and I think you guys are as well with the elgentos. And from my side, most of the work that I needed to do to make all of my modules compatible with 2.4, was just to upgrade all of the PHP unit tests which was kind of a clumsy thing because the rest of the code was working, but I still needed a couple of hours just to make sure that was compatible. I'm not sure if you experienced anything like any work needed for your own modules?
PJ: I'm going to let you in on a little secret here. We haven't upgraded to 2.4.0 for any of our clients. We tried to upgrade a few, and ran into some issues that everybody ran into. We looked at the growing list of bugs on GitHub, and we decided, “No, we're not going to do a 0.0 release or a 0.0.0 release in this case.” We're going to wait it out, weather the storm, and we're going to upgrade to 2.35 P2. And then wait until 2.4.1 is released and then upgrade to that one.
Jisse: Yeah. So I'm seeing a lot of different features that are actually postponed to 2.4.1 but you're also saying that there are quite a lot of bugs, at least by monitoring the issue list on GitHub.
PJ: Yeah. And I've heard from Ivan Chepurnyi, that some performance issues, which, you know, everything is slow according to Ivan.
Jisse: He's always finding performance issues. That’s his secret power.
PJ: He had some stuff which made me rethink our initial goal to upgrade everything to 2.4.0. And, let's just say that Magento's history with 0.0, releases isn't that stellar.
Jisse: No, I hear this more with agencies that over time they basically learned from it. Well, I was there as well with 2.0, 2.1, 2.2, and I thought, actually with 2.3, that the whole thing improved pretty much. So I actually upgraded to 2.3.0. as soon as it came out and it was a little bit troublesome, but not a huge amount of bugs. And I honestly don't know about the States with 2.4.0, whether it's a lot worse or better or--
PJ: You know what I think the problem is with the 0.0 releases, every once in a while, Magento of course, needs to bump from 2.3 to 2.4 or whatever to do a double release. And they have to stuff it with new functionality, otherwise it wouldn't warrant a minor upgrade. So they focus on new features and I have the feeling that bug fixes are put on a sidetrack in that process. And then they pick up all the old bug fixes for 2.0 for the top of the one release again. That's been my feeling with 2.2.0 and 2.3.0. And, yeah, of course, the platform itself has become a lot more mature and stable since the 2.2 or even the 2.1 or 2.0 era, so there's definitely a big improvement in terms of stability. But stability is always relative, and if we can prevent some of these issues in the double release by just waiting it out for one version for one quarter, then we're okay with that.
Jisse: Yeah, so personally, I encountered issues with bundled extensions, but that again was mostly on the testing side with integration testing and unit testing. I became pretty angry about that and I won't name the vendor anymore, but, to put adults in the specific place, I was a pretty mailer about it. That's a reference to who it was, but anyway I was pretty upset because as a third party extension provider, I'm trying to do all of that work to make sure that the extension is fitting in nicely with 2.4. And actually, because I'm not an official partner, I didn't see any kind of pre-release either. So actually on the day when 2.4 came out, I worked my ass off just to make sure that everything was patched properly. And I prepared things, but I couldn't release my new modules because I didn't know the exact semantic versioning of all of the new stuff. And to me, it was kind of disappointing to hear that basically, some bundled extensions gave issues. So you also encountered issues there?
PJ: I think the bundles extension are the main source of problems. I totally disagree with what Adobe or Magento is doing there bundling third-party extensions in there in a core of Magento. I can see it from a commercial standpoint and I hope they earn a lot of big bucks for it…
Jisse: They do.
PJ: …Because otherwise, it wouldn't be worth the trouble that those extensions are introducing in codebase.
Jisse: Yeah. So I'm personally running a repository with replace packages for some packages from the core, MSI and etc., but also this replace package for bundled extensions. I think that's actually one of my most popular packages out there. And it's not even me because I believe that actually integer.net came up with that original suggestion.
PJ: Yeah, you stole it.
Jisse: Yeah, I simply stole it, but I package it in such a way that it was more useful. But it's kind of interesting to see that such a thing is needed. To me, it’s always funny that Magento as a commercial entity goes that far into bumbling basically all of those bundled extensions in Magento as a core, and then actually all of us community members try to get rid of them again. They make you feel like, "Hey, but is there maybe a different approach?
PJ: Less code is less complexity.
Jisse: Yeah, I think so. Well, the other news was that Magento also released their own semantic versioning checker just a week ago or so. So basically they used this tool internally to prepare the Magento 2.4 release just to determine what kind of Git commits were made on the repositories and how it would translate into new semantic versions for all of the different packages. And I think it's worth just to see all of those tools become open source, and hopefully, actually, that all of the third-party extension developers also stick with those practices. Similarly, like the PHP coding standards, or unit testing, integration testing. Still, it's basically all of those bugs in, theoretically, the bundled extension, but it might also be other extensions, all of those bugs that prevent people from upgrading basically shows also that there's still a strong need for better extension quality, I guess.
PJ: Yep. Okay.
Jisse: Yeah. So, what else is up?
PJ: Composer 2.
Jisse: Yeah, so I've played around with Composer 2 myself already a little bit, but not with Magento because I tried to do this, I don't know, like months ago already. Stephan Hochdörfer, he also communicated with the composer team on how to upgrade to Composer 2 and then make it work with Magento 2. Well, I tried to follow along, but it didn't work out for me and then I just let go for that moment. I haven't tried it since, but, yeah, it's said to be improving the download times tremendously. So that would be awesome.
PJ: Yeah. I haven't tested it with Magento either, but we have tested it on Laravel projects and yeah, the faster download times is of course number one, but package resolving is also a lot faster, the SAT resolving. Plus you can speed up your resolving by setting up - let's see how they call it. I can't remember what they call it, but you can set up like filtered repositories, where you can say, “I don't want this package from this repository, but I want it from another one” or you can use only or accept, so it doesn't search through the entire repository for the package you need. You can aim it towards a certain package within the repository.
Jisse: It sounds a little bit like the replace trick that we just mentioned, but now actually more on steroids. And I heard something before that the replace trick would have been removed also in Composer 2, but I didn't check up on that anymore.
PJ: Right, well, Composer 2 has repository priorities. So, if replace doesn't work anymore in Composer 2, you might be able to create your own repository with the exact name of those packages, and then place that repository as your highest priority, and then it will fetch and like an empty file. Let's say, if you replace the MSI module then in the exact pathway, would otherwise be written a file that says, "This package has been removed by the composer repository that's been added for composer notification or something. That's repository priorities. That's pretty useful because we have a third party extension we use from a vendor, we have their composer repository in our Private Packagist, and then we need our own fork of it. So now we have to rename the package and use the new package name, but then we can just assign a priority to our own repository packages-- It's pretty cool.
Jisse: Yeah, that's pretty cool. What does it feels like with Composer 2? they simply just listened to all of the hacks and workarounds that people try to make under Composer 1? So for instance, a pretty common tool nowadays with Composer 1 is prestissimo. But, yeah, that's becoming obsolete as soon as you are under Composer 2, but that's only for the download part, to parallel, the downloads. Yeah, and I read something about this repository priorities and I remember that I thought like, "Oh, this is creating cool opportunities", but I haven't played around with it.
PJ: A prefetcher as well. I didn't know about that one.
Jisse: No, it's actually not. It's not a Composer 2 related, but it's a Composer 1 plugin just like prestissimo, and it's basically trying to do a real prefetch of all of these different packages. So that's kind of like, the http2 caching mechanism is able to prefetch certain resources in faster ways, supposedly. But in Symfony project, I actually have used this and I experienced an improvement in Magento 2. I've already had a couple of times, that the Magento 2 installation, which actually stuck because of some kind of internal thingy that they're trying to do. So, yeah, there are third-party tools basically within composer trying to improve performance. But I think it's all done by just a couple of individual developers maybe, and trying to come up with smart things. But I'd rather just wait for also Composer 2 to come out with.
PJ: Yeah. So I'm looking at the read me and what the prefetcher does is you can tell composer to only look for packages, for instance of packages in a certain version release line. So say 2.X and it won’t look in 1.X. This feels like something that could be added into Private Packagist as well. Do you use Private Packagist or Satis or--?
Jisse: Well, I'm using my own Satis instance and I'm just using Private Packagist because of some companies like Elgentos.
PJ: We're a big fan. We'll Satis does the exact same thing basically, although and Nils and Jordi (from Packagist) are adding some pretty cool features on top of price packages, like security audits. So it will give a security warning if a package has a known vulnerability in a certain version, but prefetching... So this prefetching feels like a perfect addition to Private Packagist where you could say, “Okay, I want to have this package inside my Private Packagist, but I don't care about all the old versions. I just need free 3.X and upwards or some things to really speed up the SAT dependency lookup.”
Jisse: Yeah, exactly or something similar. So I played around with it. I tried to configure it a little bit. So the example in the read me actually goes with the symfony symfony and another package. So I tried this for Magento packages. But yeah, so in my case, it kind of blew up. It seems to resolve things in a loop or something. But yeah, and talking about Private Packagist, I played around with also the feature of mirroring the Magento marketplace, improving download speeds. Now, I thought it would be really like a click click system where you would say like, “Oh, but this specific package, or actually all of the packages on the marketplace, let's copy them, download them and use them within Private Packagist instead.” I was a little bit disappointed that it didn’t work exactly that way, but yeah.
PJ: I think you can because it will add the package to your Private Packagist subrepository as soon as you require it. I think you can also manually add it by putting in a composer.json or a composer.lock.
Jisse: Yeah. But then still my impression was actually that it would only just do--
PJ: It will do a full copy.
Jisse: Yeah, of a single package, but not all of the dependencies with a package either. So actually--
PJ: The problem arises when you have a username and password combination for a marketplace mirror. We have a bunch of clients with each have their own Magento marketplace username password combination because they purchase extensions in the marketplace sometimes even though we tell them not to, and then we need to get them from composer. So every client needs to have its own unique marketplace mirror within Private Packagist. So if it were the case that it would completely mirror it, then we'd have like thousands of packages. Yeah.
Jisse: Yeah. So how do you solve then this issue of a marketplace performance? You're not or--?
PJ: We try not to use the marketplace for extensions.
Jisse: Okay, but still for the core you're using it anyway.
PJ: Yeah. It does mirror it and catch it on Private Packagist.
Jisse: Of course, yeah. That's true. Yeah.
PJ: Once you've required it, it will create a mirror for the package you have required, just not for the entire market place.
Jisse: No. And it's actually pretty cool that it's kind of like catching on demand without you doing anything for it. So, in the past, Alan Storm wrote a long, long time ago, I believe it was really like five years ago or so, a tutorial on how to mirror the marketplace by using Satis. Then I played around with it myself and I wrote a followup blog on literally what to do to make this happen because Alan Storm is never just saying like, “Well, this is what you should do.” He's always writing a tutorial in-depth and just going as deep as possible. But yeah, the bottom line was that I had my own private mirror of the marketplace and I was amazed with how good the performance was. So instead of actually just hosting a mirror, like with Private Packagist in Helsinki or Stuttgart, or I don't know where the packages the mirrors are, but then I chose basically my own private VPS instead then the download times were just reduced to almost nothing. So that was pretty cool.
PJ: Right. Yeah. So you do know that Kristoff from Fooman has a Magento marketplace mirror, right?
Jisse: Yeah. But, I don't know, again also where his mirror is actually located.
PJ: Good point.
Jisse: The cloud?
PJ: The URL ends on .nz so it might be the exact opposite of the world.
Jisse: New Zealand. Exactly, yeah. And I think the main point about playing around with Private Packagist or playing around with Satis yourself, is that you could theoretically just reduce the download times because the Magento marketplace itself is definitely located in the US, I believe.
PJ: It's here in Kristoff’s blog. The mirror is using AWS infrastructure, including CloudFront CDN.
Jisse: Okay. So that would be pretty good.
PJ: When I traceroute it, it ends up in AMS 54 CloudFront. So in our case, it's actually being served from Amsterdam.
Jisse: Exactly. Yeah. And on top of it what Kristoff solution, Fooman’s solution actually has to offer is of course also removing the authentication of all of the open source free packages. So why authenticate if all of those composer packages are freely available anyway?
PJ: He does warn not to use it in production though. I guess that’s more for security.
Jisse: Yeah, but that's just a disclaimer. Yeah, exactly. He doesn’t want to be responsible for it.
PJ: “When I get hacked and somebody injects their own code into my repository, I'm not responsible. “
Jisse: Yeah, exactly. It's just to prevent that the whole community has come to the New Zealand guy.
PJ: He can probably see who is doing it. In access logs or something.
Jisse: Yeah, cool. And also on top of it--
PJ: I think it’s time, Jisse.
Jisse: Oh yeah. So yeah, it's already half an hour, you mean?
PJ: Yeah.
Jisse: Cool.
PJ: We tried to do half an hour but we can easily fill up a full hour, but I think--
Jisse: Or a half-day or something.
PJ: I think we could go on for weeks on end.
Jisse: Yeah, exactly. There's so much in this Registry.
PJ: But we have a ton about Composer left on our cheat sheet, but I think we could just move that to the next episode.
Jisse: Yeah. And I think this is also the goal that we tried to reach, right? Simply to have a nice little conversation between two Dutch guys that among them with the two of them know quite a bit. And then if you combine that all together, then hopefully, it's a nice, interesting podcast that hopefully, some people find useful.
PJ: And if you do, and if you don’t, hit us up on Twitter @peterjaap and @yireo, or @jissereitsma.
Jisse: Yeah, but we'll post, I think, most of those details anyway, and let us know as well whether it's interesting, whether there should be an episode 2. It could also be that basically a lot of people are saying like, “Hey guys, just don't do this.”
PJ: Another podcast.
Jisse: Not another podcast.
PJ: I have podcast burn out from listening. So that's why I thought, “Let's just create one.”
Jisse: Yeah, exactly. And top of it, I think having a podcast without the bullshit would also be good. So hopefully--
PJ: By backenders is for backenders.
Jisse: Exactly. For nerds, by nerds or something. Yeah.
PJ: Cool. See you next time.
Jisse: Thanks for listening and catch up with you soon.
PJ: Bye. Bye.