What We Learned at Mage Titans Manchester 2016

Last Saturday 12th November myself and eight of my teammates donned our iWeb hoodies, packed our rain macs and headed up the M6 / West Coast Main Line for the 3rd Mage Titans Manchester 2016 conference.

We had a good cross section of the team representing:

• Neil Boughton – Technical Director
• Rebecca Troth – Project Manager (& Guest Speaker!)
• Andrew Pemberton –  Magento Developer
• Andy Smart – Magento Developer
• Andrew McCombe – Magento Developer (that’s me)
• Tom Williams – Magento Developer
• Rich Jones – Front End Designer
• Gareth Joyce – Front End Developer
• Ricki Larkin – Magento Developer

By the end of the day I think it’s fair to say we all felt in a similar state of mind.

But come Monday morning and a quick glance at #MageTitansMCR it was great to see the after-glow of the conference and the huge amount of energy and ideas emanating from the community.

What did we take out of it? For a start, lots of notes! So, in the interest of community contribution and collaboration, here they are. It’s a mixed bag of perspectives with notes from a few different team members but hopefully they serve to provide some further food for thought. If you’ve got anything to add or would like to ask a question, feel free to drop us a line on Twitter (handles included below).

Ben Marks – Guest Host

Notes from Andrew Pemberton / @p3mbo & Andrew McCombe / @euperia

• Focusing on improving the M2 Developer Experience
• “Magento 2 still has many dark corners”
• Making the online training for Magento 2 free for the whole of December and January. Magento U will have video content.
• Official docker container coming soon for Magento 2. This will be for Windows and Mac users alike. Get going quickly and easily.
• Working on improving patch quality for Magento 1
• Have been working with SnowDog to improve the front end on Magento 2, SASS based
• Consistent feedback is that M2 is better than M1 once you’re over the initial learning curve
• Changing approach to patching. Running through partners first before mainstream releases

Watch the talk (prank)

Vinai Kopp – Stories From the Other Side

Notes from Andrew Pemberton / @p3mbo & Andrew McCombe / @euperia

• Exploring OOP and functional programming to become a better developer.
• Great use of the laser pen!
• Was wearing a Magento Hero t-shirt which is very fitting.
• Referred to book: The Little Schemer
• Touched on the history of Object Oriented Programming (OOP)
• Talk was about Pure Functions in PHP.
• Lessons: Learn differences between mutable and immutable objects
• Summary: Concepts from the functional programming world can be valuable in OOP world.

View Slides / Watch the talk

Fabian Schmengler – Porting A Complex Extension To Magento 2

Notes from Andrew Pemberton / @p3mbo

• Uses a PC – what a legend!
• His talk was on porting a large extension from M1 to M2. In this case it was an SOLR search extension, with features such as autosuggest fuzzy search.
• The main goal was re-use which meant extracting as much business logic from their M1 plugin to an independent library.

Fabian’s process was divided into a number of steps:

• Write tests – integration tests, at this point he didn’t write unit tests because you shouldn’t write unit tests existing code.
• Lovely use of a picture of a LEGO Magician.
• Split large classes up into smaller ones.
• Kept original methods but delegated calls to the new classes to make life easier.
• Avoid gold plating; e.g. – it doesn’t need to be perfect. When it’s ‘good enough’ – stop.

• Implement a bridge between his new library code and Magento 2
• Can use code conversion tools as a start.

• Integrate the library and M2 using M1 code as an example.
• Dropped support for earlier M2 versions and only supported the latest version.
• Don’t use the core code as best practice examples since there is still lots to be done in it.
• Look for the best practice tag on Stack Exchange instead.
• Don’t hesitate to ignore the best practices if needed!
• Don’t use the object manager – EVER!

View Slides / Watch the talk

Tony Brown – An Optimisation Story

Notes from Andrew McCombe / @euperia

• Keep Calm and Monitor All The Things!
• Tools used: New Relic, Blackfire.io
• Blackfire can hook into CI and start a profile and catch problems going into production.
• Datadog: Centralise logging.
• With Full Page Caching (FPC_ we should serve as much as possible from the cache, e.g Homepage. Obviously some things are personalised, e.g basket.
• Areas such as mini-baskets will need caching separately. This is referred to as hole-punching.

Example module 1: Opening times

The first example  was a block that displays the opening times on the footer of all pages.

• Custom dates etc – uses a block and it used on every page on the site.
• Ask yourself: How often should this block output change?

Example module 2: My Favorites

The second example a was a ‘My Favorites’ section that is personalised to each user.

• Shows heart icon whenever you have a product that is favourited.
• How does Magento handle cache validation of personalised data? Basket saves in a cached personalised container.
• Include favourites in javascript with minimal data (for example, product ID’s only.). Add heart icon via JS after page load. Serve cached full page and render changes in browser.
• One discovery was a crontab entry that cleared the cache every minute.  The reasons for this are unknown, but you should share reasons – everyone is learning.
• Magento 2 Full Page Caching with Varnish still has the same challenges as with Magento 1 – independently flushable content. Can use Varnish Edge Side Includes (ESI) to serve private content. Use AJAX and store in the browser’s localStorage.
• Gotcha: If you use cacheable=false in a block xml it will invalidate the caching for entire page, and not use FPC. Don’t use $_isScopePRivate – is obsolete/deprecated.
• A Magento 2 FPC Monitoring tool not available at the moment. Varnish is serving the full hits so we can’t use a Magento module for New Relic.

Final Thoughts:

Plan for FPC at the start of the project, monitor as much as possible, embrace the mistakes, and learn as much as possible.

View Slides / Watch the talk

Max Bucknell – Future of Magento 2 Front-end

Notes from Andrew McCombe / @euperia

Max discovered that not many people were looking into or blogging about Magento 2 front-end development.

You can do your own thing with Magento 2 front-end. If you can do that, why have a front-end framework at all? Magento loves the mom/pop shops. Some customers optimise for longevity rather than coolness.

SASS

• Devs love SASS
• Snow.dog is becoming a core initiative. SASS is likely to become core. Try using the Snow dog theme.

Fast Compilation

• Core: measured in minutes, not seconds.

ES6 support:

• ES6 is the next generation of the Javascript language, will become mainstream soon.
• Currently compiled via Babel. Can be used with Grunt/Gulp.
• Tradeoffs: build step is slower – but it is worth it.

Performance

• Best practice: use AJAX for customer data, varnish everything else.
• First rule of security: keep software updated.
• Content Security Policy (CSP) An HTTP header that informs a browser to trust specific domains on page.
• Goal: 100% CSP in Magento 2.
• To do this we need to remove inline scripting in Magento 2

View Slides / Watch the talk

James Cowie – Objectively thinking

Notes from Andrew McCombe / @euperia

• Background on OOP
• Insight into thinking
• Insight into workflow
• Core principle of OOP is messaging

SOLID principles

S: Single Responsibility – A class should only do one job
O: open/closed. – objects should be extensible, but closed for modification: use interfaces, abstraction etc.
L: Liskov substitution principle – “objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program.”
I : Interface Segregation A client should never be forced to implement.
D: Dependency Inversion Principle.

• Magento 2 is pretty good at this. Every time you see an interface there should be a concrete implementation.
• “A Framework is one of the tools that helps you develop better or faster” – Symfony documentation.
• Convenience v Maintainability: important to strike a balance.
• Selenium, Behat, mink – i.e Integration/Acceptance tests.
• Using Behat to generate acceptance xcriteria, written in Gherkin lang
• Example showed building a Google product feed, using Behat to make it testable. The code enforced the SOLID principles using an example of extending the module to output xml as well as json.

View Slides / Watch the talk

Vitaly Korutun – Building Complex Forms in Magento 2

Notes from Andrew McCombe / @euperia

Vitaly attended the previous day’s hackathon where he realised that a lot of the content in his planned talk was outdated or invalid. Therefore he modified his talk to:

“Building pretty much anything with Magento 2 Front-end components”

An in-depth breakdown of the reason for the topic change can be found on Andreas von Studnitz‘ post at https://www.integer-net.com/reality-adjustment-why-a-talk-about-ui-compontents-changed-scope-and-title/.

• Collaborative approach with M2
• M2 Files count: Javascript, PHP, and XML.
• Much, much more Javascript – 12k in Magento 2 v 286 files in Magento 1
• Less XML

Standardisation

• M2 follows modern standardisation practices. more js files == granular modularity and flexibility reusable extensible js.
• More than requirejs provides. inline components – no inline js DI – dependency injection cuts down on configuration.

Unification

• Lots of stuff could be done the same way on server and in browser.
• Supports sync and async rendering. Output on server for Varnish, output on browser.
• Makes Single Page Applications (SPA) easier to do in Magento 2.
• All info is available at devdocs.magento.com
• Don’t be afraid to ask on Github
• Feel free to update documentation if you feel you can improve it. It’s open source!
• 4400 words in US Constitution, 19,403 in Magento documentation
• Forget about jQuery style element and class selectors. It’s not how it works anymore.
• DOM is not immutable.

Isolation

• M2 admin
• Use xdebug/xhprof to learn what is happening with PHP. Use JS profiling tools to learn how the front-end works.
• Don’t use PHP to render complex forms. Use UI components to render the forms via AJAX calls to get the data required.
• Example is getting the grid in the product page in the admin. In Magento 1 it would submit all the fields from the product ‘toolbar’ form. In Magento 2 we only submit changed information.
• No standard <form>tag anymore.

Interaction

• Interaction in admin, i.e. select element option change can trigger an AJAX request to get new data – no page reloading.
• UI components it uses configurable settings to handle form interactions. These are described on the devdocs.
• There are improvements in the Javascript performance in M2, e.g iteration over list elements, pagination etc.
• Form improvements (from >2.1.3 onwards), and grids (>2.0.0)

Extensibility

• UI components are using a similar mindset to classic FE development
• UI components on the server are not really up to the same standard as front-end
• Use $.async to manipulate DOM within the asynchronous components.

View Slides / Watch the talk

Fabrizio Branca – Redundancy Rocks.

Notes from Andrew McCombe / @euperia

• Talk about reliability
• In the beginning – 1 server. LAMP. multi-sites. Make faster, add Redis, varnish etc
• Separate boxes for most parts. Now multiple points of failure. (SPOF, MPOF)
• Solution: Redundancy
• Moving from 1 instance to multiple brings more issues. Monitoring, deployment, sessions etc.
• “Master” approach – SPOF “Cloud” buzzword for this decade. Provision resources when needed.

Reference Architecture

• Multiple regions Magento Frontend: DB AWS
• Multi A-Z cache: Redis  / Elasticache
• Load balancing: AWS or Route53
• Varnish Cloudfront / CDN / WAF = use max-age ands-max-age headers.
• Also use S3 for static files
• Dedicated Mage backend servers for Magento workers – runs crons on separate instance

Building Blocks

• AWS is fault intolerant. EC2 – server instances – provisioning
• Use auto scaling groups for EC2. Not about auto-scaling traffic.
• Actually about resilient auto-failing. i,e, one EC2 instance fails, it auto launches another.
• Fully automated launch configuration
• If you want only 1 instance – still put into auto scaling group. No more 3am wake ups!

Launch Configuration

• Define the type:base AMI (region specific, OS) Provisioning script: ansible, puppet etc.
• Bake your own AMI’s. Create your own, provision, and then ‘save’ it for future use. AKABaking

Immutable Deployments

• Split the disposable parts from the non-disposable, i.e, web front ends are disposable, DB master isn’t.

Connecting the blocks

• Loose coupling by using service discovery. Route 53 DNS using cnames, use AWSAPI, use tags.

Monitoring

• Keep an eye on metrics.
• Cloudwatch (Amazon), Datadog, New Relic
• Be Prepared!! Try to anticipate any problem. If you can’t, work on recovering fast instead.

Stress limits

• Know your infrastructure’s limits.
• Learn and use Apache JMeter for testing.

Choose the right partners

• Amazon Digital Ocean Payment providers CDNs etc
• Verify that all the services and providers in your critical path
• KISS! Keep It Stupidly Simple.

NIHS (Not invented here syndrome)

• Don’t try to solve the problems yourself. Pay other services that can do the job for you, i.e New Relic etc.

Questions

How does Fabrizio handle switching to mysql master/slaves?

Use CNAME to switch to master/slave.

How do you rollback if something fails?

For critical CI deployments do not destroy previous build. Blue green deployment. Use N98 Magerun to deploy DB changes, via a Magento worker that processes updates/upgrades.

View Slides

Anna Volkl – Secure Input and Output Handling

Notes from Andrew McCombe / @euperia

• How not to suck at Data Input and Output
• XSS 65% of all websites globally suffer
• XSS it is #3 in OWASP top 10 list
• Every feature adds a risk, every input and output adds risk
• Input > Process > Output
• Security is usually bottom of the tech-progression scale. Hardware, Software, Security.
• Stop ‘last minute security’. Don’t deal with security as a last minute thing – always implement up front in code.

Front-end input validation

• User experience – show users errors early
• Magento: look at rules.js for js validations
• You can add your own validator, in front end add data-validation=”validation-rule” attribute.
• Never rely on javascript to validate code. Do it server side too.
• Same for web services, cookies etc.
• Magento uses Zend Framework Validators.

View Slides / Watch the talk

Max Pronko – Ups and Downs of real projects based on Magento 2

Notes from Ricki Larkin / @Triickii

About Max Pronko

Max Pronko used to work at Magento but now works internally at The Irish Store (https://www.theirishstore.com)

The Irish Store is a season business, doing 95% of their annual turnover around Christmas time. Their existing site was launched in 2013 and had 4,000 products (around 1,000 configurables) and over 500 categories.

About the talk

After 8 months of development, they have recently launched their new site on Magento 2. They wanted to upgrade to Magento 2 to improve all of the ‘touch points’ for their users (packaging, emails, website, tone of voice, brand).

The plan was to start the project in January 2016 and finish in August 2016. During the project the workload increased dramatically, the team also had to fix a lot of M2 bugs themselves. As the workload increased they used SCRUM and worked in 2 week sprints with daily stands up to get the basic functionality working with a plan to add features in later sprints. During the data import they found that they needed to exclude more columns than they needed to include so decided to do a manual data integration.

They installed 13 third party extensions but 1 only passed their internal QA (Realex Payments), the rest failed.

Their internal targets for the site were multi-availability, 99.9% uptime, 3 second page load time and able to handle < 300 orders per day.

They had to change their AWS (Amazon Web Service) hosting from 2 servers (1x site, 1x db) to 5 servers with load balancing. They also use tools like New Relic, Pingdom and Webpagetest to keep an eye on their Magento store’s performance.

They assume that Magento 2 releases will be every 3 months. Performance testing that they do is based around number of orders, as that is the main point of the site. They expect more features in Magento 2 and due to the documentation they expect the client/merchant to pick up and use these features. The infrastructure of a site should be thought about at the start of the development process, so you know what will need caching, etc.

View Slides / Watch the talk

Tadhg Bowe – Migrating Data from Magento 1 to Magento 2

Notes from Ricki Larkin / @Triickii

About Tadhg Bowe

Irish. Tadhg is a Magento Lead Developer at Screen Pages.

About the talk

Magento Enterprise client migrating to Magento 2. They have 21k+ customers, 5k+ products, 45+ orders, 100k+ quotes. The site had 15+ third party extensions and 40+ in-house extensions.

They used the built in Magento import tool to import the data across, he said the key was in the prep work, using third party extensions where possible and lots of dry runs.

After getting errors on the data import use the map.xml file to fix data.

View Slides / Watch the talk

Rebecca Troth- Magento is for Life, not just for Christmas

Notes from Ricki Larkin / @Triickii

About the talk

The talk was about the importance of keeping Magento up-to-date. After the initial “warm up” Rebecca gave us some stats on Magento sites and what version they are running, worryingly there are 11 sites out there still running Magento 1.2!

Some agencies treat Magento sites like a factory production line, running sites off and then never updating or evolving those sites.

A new Magento site is a lot like having a new puppy (Max): they require daily attention, daily updates to content, products, banners and CTAs help with their SEO performance and help with keeping customers engaged. It is also import to keep the hosting platform up-to-date to allow Magento to perform to it’s full ability.

Training merchants to use the platform is also crucial in getting the most out of Magento. When updates and patches are released, we should make sure that these are rolled out and not ignored. Max 2 promises to make this process less painful with the updates on the second to last slide.

Take away: give your Magento plenty of love, care, and attention to get the best out of it.

View Slides / Watch the talk

Nils Preuß – Evolution of the Core Import Interface

Notes from Ricki Larkin / @Triickii

About the talk

Magento 1 API import was really slow. Magmi import can throw up errors. The best way to import data was through the Import/Export feature in the admin, importing data via a CSV.

Magento 2 uses the same SOAP/REST API but it is slow, the Import/Export feature is still the better way to import but it’s not automatic. It does now support external images though. There are 81 fixed columns in the export, no matter what data the user decides to export. Multiple attributes are exported in a single column, delimited by a vertical bar with the values of that attribute delimited by a comma.

There are automated CSV import extensions out there that extended the M2 import/export feature. The one referenced was: https://firebearstudio.com/the-improved-import.html

Watch the talk

Jason Woosley – Community Engagement 2.0

Notes from Ricki Larkin / @Triickii

About Jason Woosley

Jason is Vice President of Technology at Magento. He is responsible for engineering, quality, operations, PMO, user experience, design and technical documentation. If you find a bug in Magento it’s his fault! He is “rubbish on Twitter”.

About the talk

Jason talked about the community engagement from Magento and the need for it to improve. He acknowledged the excellent community that Magento has, which is backed up by the 6,000 contributions to Magento 2 through github.

When they launched Magento 2 Merchant Beta should have been non-production even though it was put on some production environments, this lead to Magento 2.1 being an RC (Release Candidate) and was specifically for development purposes only, non-production! Magento have also got the Magento 2 B2B to the Alpha stage now, so they are looking to work closer with agencies and merchants. It is the first time Magento have released code before the MVP product. They are hoping to release code before all features are completed, this would be for non-production deploy so that extension developers can see the majority of the core updates and work with them, without waiting for the full official release, it gives them a headstart.

Jason said that Magento have learnt lessons from the SUPEE-8788 release and are looking to work closer with the community.

View Slides / Watch the talk

Footnotes

If you need a further fix of Mage Titans, check out their YouTube channel.

It was also great to hear that Mage Titans have raised £5,300 for worthy causes to date, with this year cause being the Royal Manchester Children’s Hospital.

Roll on 2017 when Titans continues the roadshow in Milan, USA, Mexico and Manchester once more.

A truly rewarding day for all involved!