Full transcription below show notes

Show notes
Extensions mentioned
https://github.com/davidrjonas/composer-lock-diff
https://github.com/magento/quality-patches/blob/master/patches.json
https://github.com/mage2tv/magento-cache-clean
https://github.com/kiwicommerce
https://github.com/olivertar/m2_api_product_images
https://github.com/experius/Magento-2-Module-Experius-MissingTranslations
https://github.com/netz98/headless-guillotine
https://github.com/markshust/magento2-module-disabletwofactorauth
https://github.com/magento/security-package

Meld diff tool - https://meldmerge.org/

Architecture repo - https://github.com/magento/architecture
Database MediaStorage in M2 - https://github.com/magento/architecture/issues/111
File system framework in m2 - https://github.com/magento/architecture/issues/219

Upcoming episodes:
#4 Willem Wigman & Hyva
#5 CI/CD

Full transcription

Jisse: Hi, everybody. Welcome to episode number three of The Registry with me, Jisse Reitsma and-- 

Peter Jaap: Hello, Peter Jaap here.

Jisse: Cool. So there we are. Yeah, episode three. We started off with episodes one and two, kind of like, a little bit awkward, a little bit eerie like, “Was this going to work? Did we only make it to produce a single episode? Or would this be a series?” But slowly, I think it’s becoming a series, right? So that's a cool thing.

Peter Jaap: I think it's officially a series where you have three.

Jisse: Okay. Not a miniseries, but maybe it's becoming also like one of those long-going series as well. Yeah.

Peter Jaap: Yeah. 

Jisse: So the concept is simple. It's two Dutch guys talking about random things related to Magento 2, related to developer stuff, things that we find interesting. Yeah, and maybe the kickstart of everything is we're recording this actually, while Magento 2.4.1 has been out already for, I don't know, like one week, two weeks? Two weeks ago already. So Peter Jaap, did you upgrade already?

Peter Jaap: Well place your bets. What do you think?

Jisse: I think you didn't.

Peter Jaap: No, we didn't. We're helping everybody on to 2.3.6 at this point, which is a really easy upgrade from 2.3.5-p2 where most of them are coming from. I just did one today from 2.3.4, which also was a pretty small step. Wasn't that big, that much of a problem so I like. 2.3.6 was an easy upgrade so I don't know, should I go to 2.4.1? What are your thoughts? 

Jisse: Well, if I have to admit, so I'm running a few production shops, and they're actually just doing fine. I see also, agencies that I'm helping out struggling with the same kind of issues that you have. So the real question is like, do you really need it or not? Well, you don't need it to get rid of MySQL search because well, you can do that already with the older version. So yeah, there's not that many improvements that you might be eager to adopt unless you're running a Hettle shop already, something based upon PWA Studio or-- And personally, what I find-- Yeah, go ahead. 

Peter Jaap: Or I did upgrade one shop to 2.4.1 because we're starting Hyva implementation on it and that requires 2.4. So I did do one upgrade and it was a fairly simple shop and that went pretty easy. But yeah, that's because of Hyva, but we'll get into that next week. But yes, I lied. I did do one.

Jisse: Hoofa, hoof, hoof, hiff, hife, hifa…

Peter Jaap: Hoova, hoova, hivva, hyva

Jisse: I don’t know what it’s being pronounced as but yeah so actually, this episode-- 

Peter Jaap: Willem pronounces it as hoova. 

Jisse: Exactly. Yeah. And this is episode three.

Peter Jaap: It’s Finnish or something. 

Jisse: Yeah. Well, definitely next episode, he's joining us to explain everything, including the right pronunciation and including the real meaning.

Peter Jaap: Oh, good because I don't understand anything.

Jisse: Exactly.

Peter Jaap: I think that mainly has to do with it being front end stuff and I'm not very good on that.

Jisse: Now, for me for the upgrade to Magento 2.4. So I've been doing that upgrade for a couple of simple shops and there, I didn't really find any issues. One of the things that I'm kind of eager to play around with is the controller decomposition. We mentioned this in the previous episodes, and I can see really the differences in loading times that a controller based upon the old extending upon the abstract action or the admin actions that are there. And then suddenly replacing it with one of these, or just an implementation of an interface, a clean constructor that you basically have for your own dependencies, it's just reducing per controller loads about 50 to 60 milliseconds per Ajax goal or per load. That's pretty awesome. But yeah, it's not included in the core. So none of the core modules have been adopting to that new strategy yet because of, well, I don't know, backwards compatibility, and so on. But yeah, so let's not talk about the upgrade any further.

Peter Jaap: So just ripping out all the dependency injection in these controllers is giving you this speed increase.

Jisse: Yep, which makes you wonder, of course, like, “Hey, what on earth is all of that dependency stuff then doing?” Well, it's dealing with events, it's dealing with, maybe also really cool stuff, plugins that are doing all kinds of tricks in there. But if you want to reduce it down to your own controller deciding upon what needs to be done in a very clean and direct way, it's kind of cool.

Peter Jaap: So if you use the decomposited, decom, what’s the word?

Jisse: Decomposition

Peter Jaap: Decompositioned controller, you don't have any events triggering, or is that handled in a different way?

Jisse: So as far as I know, the events are simply skipped. Yeah.

Peter Jaap: Okay. Interesting. So that's a thing to watch out for.

Jisse: Yeah. But maybe that's also the strategy that I think should become more popular - less is more. So instead of just adding more functionality to Magento 2, sometimes it's actually good to reduce functionality, remove things that you have out of the box, just for the sake of performance.

Peter Jaap: So configuration over convention. Less magic, more explicitly, yeah.

Jisse: Yeah.

Peter Jaap: Okay

Jisse: Yeah, now let's skip the rest of the talk about the upgrades. I'm maybe going to ask you next time again, as well.

Peter Jaap: This will be a recurring item in the podcast, “Did PJ upgrade to 2.4?”

Jisse: Yeah, so episode 26, “PJ, have you upgraded already to 2.4.1.?”

Peter Jaap: Who knows? Maybe I'll do one every week and then in like 20 episodes from now, all our clients are on 2.4.

Jisse: Exactly. Yeah, and other news that is out there is Composer 2. So I've been playing around with Composer 2 myself quite a bit already, even trying to implement it in a couple of my own continuous integration steps on my own personal GitLab instance. And if you're using Composer 1 together with prestissimo, then actually the speed is already pretty cool. But now out of the box, it's also there with Composer 2. And you're only going to bump into actually two areas of issues. One of them is something I'm still not able to explain in a maybe correct way, but there's a feature in Composer 2 called canonical repositories. And basically, canonical means that as soon as Composer is going to search for a specific package, it's going to find that package maybe in a single or the first repository that it encounters. But then theoretically, if that same repository is also registered under the same name in other repositories as well - so for instance, both on packages in the Magento Marketplace - then Composer 2 might be bumping into issues. And the solution is quite simple. Just simply add a “canonical is false” flag to the Magento Marketplace, and then you're good to go. 

The second one is actually more problematic, as in that if you're using Composer 1 plugins, then those Composer plugins are no longer compatible with the new Composer 2 API. So that makes you wonder like, “What kind of composer plugins do I have?” Well, prestissimo is one of them. So you can't use prestissimo but well, that's kind of common sense, because the same functionality is in Composer 2 already, but there's a couple of others as well. Two of which come to my mind right now are, first of all, the Composer plugin that Magento is shipping. So I'm not sure what the official name of it is again, but it's managing your Composer dependencies

Peter Jaap: Magento Composer Root Plugin.

Jisse: Yeah, exactly, that one. And that's basically currently not compatible so that basically if you're using Composer 2, this Root Composer Plugin doesn't have the ability anymore to update your Composer 2 JSON file to the latest version if you're upgrading Magento - well, that's a pity. And a second one was more problematic, that Magento has been migrating their Zend stuff from Zend into Laminas.

Peter Jaap: Laminas

Jisse: And then there's this Laminas backwards compatibility layer of some kind. So, I didn't really dive deep into it, but I got an impression that actually this Laminas compatibility thing is allowing for backwards compatibility with Zend goals so that you can still fall back to Zend classes or Zend namespaces while actually it's being replaced with Laminas. Well, that again doesn't work, so theoretically that's going to break also systems if you're trying to use Composer 2 with the Magento 2 versions. But yeah, there's a simple flag ignore platform requirements, which allows you to still install Magento 2 under a PHP 5. I’m not sure if that's going to work but the save flag allows you also to skip those Composer plugins and ensure it works, it's speedy, it's nice. Yeah. Did you play around with Composer 2?

Peter Jaap: Yeah. [I’ve been 10:54] using it as much as possible. Although you just said prestissimo, that the speed difference between Composer 1 with prestissimo and Composer 2 isn't that big, I have a different experience. Because not only the downloading of the packages is faster in Composer 2 - that's what prestissimo did for Composer 1 - but also the SAT dependency tree resolving is much, much, much faster and smarter implementation than it was in [Composer 1 11:27]. So if you're going to do a huge Magento upgrade with tons of packages and dependencies, it will resolve up to 10 or 20 times [faster 11:40]. That's the main benefit for me using Composer 2. 

Jisse: Cool. 

Peter Jaap: So speaking of Composer, I ran into this also during one of my Magento upgrades. I needed a way to see which packages were upgraded during a Composer upgrade or a Composer require. So sometimes you are in a project and you run a Composer require whatever package and that will also upgrade the dependencies of that package, which you might already have in there. And it will show these upgrades and these installs in the output of the require command. But usually, I don't really look at it, or I don't save it anywhere. And then, later on, I think, “Oh, maybe one of those packages that updated might have a bug in it” if I run into something. So then I have to manually go through the Composer lock files to see which have changed. And that's a mess because it's a diff file and that's not pretty cool. 

So there's this package, it's called davidrjonas/composer-lock-diff, and it's basically a little diff tool, like a PHP binary, that diffs your two Composer lock files, and it will show you a table of the previous version it was in the old Composer lock file, the current version, and it will give you a link to the GitHub repository with already the version numbers in the “compare” functionality. So you can just copy-paste the link and open it and then you'll see exactly what code changed between these two versions of the package that's listed. And this is great stuff, so we've already put it into a protocol at our agency. Whenever you do an upgrade, and when you finish the upgrade, generate the lock diff file and post that into the issue as well so someone who runs into problems down the line can have a nice overview of what actually happened. You can actually pass Git commit references to the binary. So it will fetch it from Git and then compare it with the one in head or the one in a different branch or whatever. So great, great little tool.

Jisse: Yeah, you love your Composer stuff, right? 

Peter Jaap: I love Composer so much. And it's not just I love Composer, it's sometimes so hard to see with the diffs and the patches and there's so much going on at any time, so any tool is welcome. Oh, it just reminded me of one. I'm running Ubuntu Linux distribution, and I ran into this GUI tool that’s called Meld, M-E-L-D. You can just use apps to install it, and you can diff entire vendor or entire directories. So I now use it in addition to the upgrade patch helper I mentioned in the last podcast, to diff the old vendor there with a new vendor there and then you'll have a much better visual way to see the differences between the two files that have been upgraded instead of the patch file. The upgrade patch helper generates itself. So, extra tip.

Jisse: Yeah, and it's basically still useful if you want to upgrade sooner or later to this Magento new version of 2.4.1 but you're still a little bit wary what kind of changes are in there and if you agree with those changes, of course, yeah.

Peter Jaap: Yeah, exactly. I could do an upgrade to 2.4.1 in probably a few hours, instead of 10s of hours that it used to be without all these tools. And be certain that I didn't miss anything. Yeah.

Jisse: Yeah. And the challenge is still that maybe the Magento official version is just going up and introducing new, new features, but you don't want to have all of those features all of the time just to be in your shop. So you're bumped into this repository of Magento as well called the Quality Patches.

Peter Jaap: Yeah, so this blew my mind. I didn't know this existed. 

Jisse: Oh, me either.

Peter Jaap: And I asked my colleagues and some other-- So this is one of those hidden gems. So it's a Magento repository, it's Magento/quality-patches, quality patches. It's a tool to apply patches to different Magento versions. I haven't actually tried the tool itself but the interesting thing I've found was the patches of JSON file that's in that repository. This lists quality patches, like what the name says. So if somebody did a PR in the Magento repository, and it fixes a bug for certain Magento versions, this is a 700 line long file with, let's see, 67 quality patches for various issues in the Magento code base with a version constraint mentioned in the JSON file itself. So you can pretty much copy-paste this into your project and run a Composer install and let the [final 17:00] Composer patches helper do its thing. And then it will install quality patches for the mentioned bugs, which might have been backported to 2.2 or 2.3 and are now in the latest version. But if you're not running the latest version, this is the next best thing.

Jisse: Yeah. So the gem is still that it's all about the patches, of course, all of those patches are within that same repository as well. But specifically, the JSON file is just ready to go to import by using the [final 17:35] Composer patches tool, for instance, just to get going with it. Yeah, it's pretty cool. But yeah, it's kind of like something we discovered ourselves just now as well, while it has been going already for four months in a row. So it also makes me wonder how many people know about this indeed.

Peter Jaap: Exactly. The repository is four months old. The last release was 13 days ago. I never knew about this. I just happened to run into it, I don't even remember where. But I have a feeling nobody knows about this one.

Jisse: No, but maybe it falls together with another project that I literally have nothing to say about yet, which is the Magento SUT tool, Software Upgrade or System Upgrade Tool, SUT. 

Peter Jaap: Self Update Tool?

Jisse: Or self, I don't know, but it's-- 

Peter Jaap: I don't remember either.

Jisse: It's part of maybe a community engineering or I don't know, but Magento [is] basically trying to make the whole process of upgrading easier. And actually, every episode, we start with a question like, “Hey, PJ, did you upgrade to 2.4.1 yet?” And then he answers no, but for reasons, right? And then maybe this is just part of that same thing as well, that they’re trying to get the upgrade process more under control. But yeah, SUT, the Adobe/Magento SUT tool was announced at MageCONF, and unfortunately, I was unable to attend for that talk. So hopefully, the slides are coming online soon, or the video or the upgrade tool itself, because yeah, I’m just looking to know whether it’s--

Peter Jaap: This is just one of Magento’s secret projects.

Jisse: Excellent. Yeah.

Peter Jaap: I think it stands for the Secret Upgrade Tool.

Jisse: Yeah, that might be as well, right? Yeah, but so yeah, cool stuff. Well, there's also a couple of other things that I find are nice to know about also. We talked about Magento 2.4 quite some time ago, and then there are various security packages that came out with 2.4 but they've been also partially in 2.3 as well. So a whole bunch of reCAPTCHA modules with a whole bunch of two-factor authentication tricks. And I think we already discussed this [in] the previous episodes as well, so let's not go into the details of two-factor authentication or reCAPTCHA. Other than that, there's actually a lot of third party efforts in a community going on to disable two-factor authentication again or to disable all those reCAPTCHAs, which I find funny. But yes, so the story I wanted to discuss anyway is on the robots.txt file. 

So the robots.txt module that ships now with Magento, allows you to generate a robots.txt file on the fly. And that's kind of nice, as in, I've been explaining actually this mechanism during a training as well, just to show people that the module itself is hooking into the front controller’s dispatching mechanism. So it's adding a new router on top of the existing routers. And then if you know the front controller structure a little bit, part of because it's also a question on the certification program, but there's this loop going on between all of the routers, so one router is just mentioning like, “I don't want to pick up this request so let's continue to another router.” And the robots.txt is just adding another router on top of it. And then if the request is for robots.txt, the robots.txt router is picking up on it, delegating it into a controller, delegating it into the XML layout, so in the end, the robots.txt file is generated, actually, with a lot of PHP logic. 

And the funny thing is actually in there that in most of the sites that I encountered, that single goal for a robots.txt file using dynamic logic takes up about 400, 500, 600 milliseconds. So it’s actually just something that is so overly complicated. Well, I personally have a really golden solution for that. Just navigating to the pub folder, create a robots.txt file, and then actually, the speed of that file will be just awesome. So yeah, it's just a funny thing that Magento sometimes is just adding all of those dynamic modules, while theoretically, having just a simple mechanism of a simple file in place is good enough as well.

Peter Jaap: Yeah, we usually set up an NGINX rewrite rule for robots.txt and place them outside the pubroot

Jisse: Exactly. Well, the only thing that might be still a good thing about that module is, of course, the dynamic nature so that if you wanted to deny or include specific categories for specific store view, then still that PHP logic is okay. But again, it's more or less the same

Peter Jaap: I would have preferred an implementation where it would run a cron and then you could tell it where to save that robots.txt file to. 

Jisse: Yeah, yeah. Well, and because of the performance impact, my recommendation would also be just to disable that robots.txt module and just come up with a better, smarter alternative instead.

Peter Jaap: Yeah. All right. Going back to the two-factor authentication disable thing, when I did my 2.4 upgrade, I used the Magento 2 module disabled two factor of package from Mark Shust. That worked perfectly so shout out to Mark for that one.

Jisse: Yeah, and to mention, maybe, that there are more alternatives out there, but I think also that that Mark’s module was just the cleanest and simplest approach as well. But it's funny to see that it's kind of like a silly thing to install. It's a module to disable yet other functionality and other modules. But yeah, it's just a nice little thing to have. So yeah, shout out to Mark.

Peter Jaap: Yeah, also shout out to M.academy, Mark’s course on Magento development. We use it to onboard our new Magento hires, and it's great stuff. So check that out if you haven't. To make a segue on to a different Magento wizard doing courses, Mage2 TV, Vinai. Vinai has Magento cache clean tool, I’d call it. It's sponsored by Mage2.tv and the Mage-- I can't live without it anymore. It's a file watcher for your Magento. It's a JavaScript thingy. You just run it by running cache clean.js to access watch, and it will watch all your files that have anything to do with cache. So if you update a PHTML file, it will clear the block HTML and the full page cache. If you update and layout XML file, it will clear the layout cache for you. So, there are also shortcuts. If you press “A”, it will clean everything. So it’s awesome.

Jisse: And maybe even better yeah, it's not only about the cache, but it's also about the generated code. So for instance, if you're playing around with the console commands or with controllers, then it's also frequently needed to wipe out the generated classes. So the interceptor classes around the controller from the generated folder, and that's being picked up by the Mage2 TV cache cleaner as well. So yeah, awesome tool. 

Peter Jaap: It works fast, it works flawless. I'm not even thinking about cache anymore during development when I have this thing on, it's great.

Jisse: That's one of the main takeaways as well, right? So do not develop with all of the caching disabled, unless you're somebody who can live without the developer flow. But if you care about efficiency and productivity, make sure that all of the cachings are enabled, theoretically, the full-page cache is disabled. But then actually, you really need a smooth way to refresh all of the caching. So yeah, that's a good shout out to Vinai. Hi Vinai. Yeah, so other stuff. Maybe it's also nice to mention that if you're playing around with headless, so creating your own BWA, something, maybe, based upon that PWA Studio, or VueStorefront or Deity, or building it yourself, then you might find actually that it's still possible to navigate to the original front end of Magento. So let's call it the Luma front end. And that's, of course, a silly thing. And then maybe you can find a way of disabling that. Maybe you're crazy enough to also play around with the PWA Studio upwards tool. But yeah, there's another tool much better for it - the netz98 headless guillotine. So yeah, the name of the module is a little bit macabre. So it's reminding me of the time of Napoleon where all of the people got their heads chopped off. But the headless guillotine is actually just chopping off the head so that leaves you with just the head or the body, I don’t know. I’m a little bit fuzzy but it’s allowing you to automatically redirect any attempt to navigate through the front end, navigate elsewhere.

Peter Jaap: Actually, it doesn’t navigate you elsewhere, it throws an exception.

Jisse: Oh, oh, sorry. Yeah. So I've been there thinking about another module myself, but yeah, it's pretty useful anyway. And maybe to mention as well--

Peter Jaap: That we implemented this and then we got like 1000 Sentry notifications because some bots triggered an exception on it during the night. So I woke up and our whole Sentry quota was depleted because some bot found the back end Magento installation for a PWA shop.

Jisse: Yeah, so do make sure to use it with care. Yeah, and maybe to mention as well, netz98, the creator of this module is also, of course, the creator of Matron, and well, who doesn't know about Matron?

Peter Jaap: Yeah, shout out to Christian.

Jisse: Yeah, so what else is there? There's plenty of modules still to discuss, right? So we always love to keep going and going and going.

Peter Jaap: I have so many. 

Jisse: It's too many, maybe as well. So let's maybe just-- 

Peter Jaap: I do have one. 

Jisse: Okay, go ahead.

Peter Jaap: I do have one. It’s not an extension but actually multiple extensions, but it's one organization. They're called KiwiCommerce. 

Jisse: From New Zealand?

Peter Jaap: They're a relatively unknown UK player.

Jisse: Exactly, I don’t know.

Peter Jaap: They’re from the UK, not New Zealand, unlike what the name suggests. So they have some great extensions on GitHub. We're using pretty much all of them. They have a Chrome scheduler, they have a login and customer functionality, which I hear isn't necessary anymore in 2.4 because it's now a default option.

Jisse: They give you it by default, yeah. 

Peter Jaap: Yeah. Okay. So they have a customer password reset extension and admin activity logger. We use that one on pretty much all our stores. That's a great extension. Enhanced SMTP extension, we use that one a lot as well to hook up Office 365, or G Suite or whatever, SendGrid, you name it. And an inventory log extension, which, as it says, logs the inventory mutations. It's very useful if you have an external ERP system that changes stuff. And the great thing about this one is that it will also log stock changes that are done on the database level. So it has a trigger on the table, and that will listen for changes on the table. So if you have a four-party ERP system that hooks into the database directly, which, unfortunately, some ERP systems do, then this extension will also log those. So that's a great feature this one has, which I haven't seen in any other of the competitor extensions. 

Jisse: Cool.

Peter Jaap: So shout out to KiwiCommerce.

Jisse: Yeah, but there's only one downside. I noticed actually, that maybe pretty recently, all of the repositories are now archived by the owner. So that means basically that they're no longer with Magento. Or maybe they just change their approach of open source. But anyway, the open source repositories are there, right? So you can just fork it, you can use IDs. And that's maybe also the whole point of open source

Peter Jaap: Yeah, I think they recently archived those models. I don't know why, I guess it's the OSS burden. I don't blame them. So fork it, the initial code is pretty good. We've been forking them into our local repository as well. Maybe someone can stand up and become a maintainer on all of them. So yeah, still great work.

Jisse: Awesome. Awesome. Yeah. So we could keep on going, but maybe that's good for another time as well because we're doing a series, right, so who cares?

Peter Jaap: Yeah, yeah, no, we could go on for hours on end. We did the last few times. Let's do a short one this time. 

Jisse: Exactly. Yeah, maybe let's look into the future as well. So we started to mention Willem Wigman on his new project, Hyva, I don't know how that's pronounced

Peter Jaap: We'll add him next week how it’s pronounced.

Jisse: Exactly, but then maybe that's also nice to actually really talk about the future of front-ending in Magento. And not just talk about PWA, but actually also about his work and the opportunities that his front end is going to give us. 

Peter Jaap: Yep, excited. 

Jisse: Sounds like a good idea to dedicate that whole episode to Hyva, no to Hoova, that's the right pronunciation. Then maybe also the next episode after that, that would make it episode five, do something with CI/CD, testing stuff, gold quality checking, etc. I don't know, but I think we’ve got plenty of things to talk about.

Peter Jaap: Definitely. All right.

Jisse: Great. 

Peter Jaap: Thanks, Jisse. See you next week.

Jisse: Thank you Peter Jaap. See you next week indeed. And thanks for listening, and I hope you follow us back. Let us know also if you have other interesting topics that you would like us to talk about. We're happy to dive into other stuff as well. So yeah, hear from you soon. See you.

Peter Jaap: See you.