Before you ask, yes, the image at the beginning of this post is a conductor, not a composer. The open source PHP package management Composer has a conductor as its official logo, and who am I to argue? An image of somebody scribbling away on sheet music may have been less compelling. A harmless sleight of hand.

As I work on more and more Drupal 8 sites, I am becoming convinced that given enough time, nearly all Drupal 8 sites will need to be managed with Composer. I think that's kind of a big deal.

Keepin' it on the download

Right now, you can still go to drupal.org, and you can download all the nice neat package of code to run a Drupal site that will work out of the box. Often this file package gets called a tarball. You can download the files either as a .zip or a tar.gz (hence the name tarball). Unpackage those files and away you go.

The same has been true for installing a module or contributed theme. Download a file package, put it in the right directory, then install the module or theme on your site.

You could speed this process up with the drush dl command, which takes cares of downloading those files, unpacking them and putting them in the right place. Want to get the latest version of Drupal, a module or a theme? drush up and you are all set.

Getting off the island

However, things swiftly go a bit differently with Drupal 8. With D8, we "got off the island" and started taking advantage of code in the wider Drupal community. This is cool! We no longer have to write the same bits of code that everybody else is writing.

How do we take advantage of this shared code? Package management tools take care of looking at a set of package dependencies, getting you the right versions, and can even get package dependencies for your dependencies. A good package manager sorts through the guidelines for what versions of a particular package are requested. Multiple packages may have the same dependency, and the package manager sorts through those requirements to come up with one set of files that fits all those requirements.

Composer is a widely used package management tool for the PHP community, and so that is what we are using in Drupal. While the download from drupal.org comes with all the dependent packages zipped up with the files for Drupal itself, as of Drupal 8.1, if you download the git repo for Drupal core, those dependencies are not in the repo. Instead, once you download the repo, you need to type composer install in the repo root, and then all the dependencies are downloaded into a vendors folder.

Not too much trouble so far, right? Well...

The autoloader

You may have heard that Drupal is more object oriented. I think I heard that mentioned once or twice. Rather than big giant files with lots and lots of functions, Drupal 8 has lots and lots of folders with lots and lots of files with smaller bits of code that can be used in a more flexible way. How do we take advantage of all that code?

Drupal 8 makes use of the PSR-4 standard for PHP class loading. That means if you put certain class files in certain folders, you can make use of those classes in other parts of your code base by referencing those classes using a namespace based on that folder structure.

What if you want to make use of that code we got off the island? It is sitting outside of where the rest of our Drupal files are defined, over in that vendors folder. Well the bit that takes care of pulling all of these various namespaces together is the autoloader. This code takes care of setting up the tools so that you can reference class files using a namespace whether they are defined within the Drupal namespace or in a namespace for a package from another vendor.

How does the autoloader know about all of these packages? Through Composer! The composer.json file that defines the versions necessary for each package also gets used to set up the autoloader file.

If you are just using Drupal core, this is no problem. Whether you are using the git repo or downloading all the files from drupal.org, the composer.json file and thus the autoloader will be set up in the same way.

So no problem, right? Well...

Most sites use more than core

If you have worked with a Drupal site, I imagine you may have used a contributed module once or twice. Yay, taking advantage of code written by others!

Wait, doesn't this sound a bit like getting off the island, sharing code and such?

Drupal knows how to look for modules in certain locations within its file structure. So if you download a module and put it in the right place, no problem, Drupal can use it. You don't have to do anything special to make that happen.

But what if you download a contributed module, and that module wants to get off the island too? If Drupal can make use of third-party PHP code, it is reasonable that contributed modules would want to do so.

How do they define those dependencies on third-party code? Through composer.json of course! But what if a contributed module depends on a slightly different version of a PHP package than what is used by Drupal course. Or if it depends on a package not used by Drupal at all? If each contributed module managed its dependencies on its own, it would likely run into conflicts with similar packages used by Drupal core.

So the way to fix this is to have a module register its dependencies through composer.json, then pull in that module not by simply downloading the code and putting it in the right folder, but instead registering that module with your project's composer.json file. Then, Composer will take care of downloading the module and its dependencies and integrating those dependencies into the overall project autoloader, so the right versions are downloaded, and we don't have any conflicts.

Okay now the problem is solved, right? Right? Well...

drupal-composer/drupal-project

If you register a contrib module with composer through something like composer require drupal\projectname, then that module will get placed in the... vendors folder. And then Drupal won't know where to find it, because it is looking in the modules directory.

So what you need is some special instructions in your composer.json file so that contrib modules get moved to the right location in the Drupal project structure. You can find just those sort of instructions in the github repo, drupal-composer/drupal-project. That provides an excellent place to start setting up your Drupal project using Composer. The composer.json file in that repo will take care of putting modules where they belong.

But wait, when I run composer install on this, all my Drupal files end up in a web folder. And my vendor files end up in a folder that sits alongside the web folder. So how do I make this work again? Do I upload this whole repo to my web host? Just the web folder? But then where do the vendor files go?

The Drupal Project model of setting up the web and vendor folders side by side is a good model. You want the web folder to be your webroot, and for all the other files to sit outside of your webroot. The challenge is that many inexpensive web hosting solutions might not support this. Can you in theory mess with the Composer settings so all the vendor files end up inside the web folder? Yes? Maybe?

My point is just that this gets complicated quickly. I use Pantheon for my web hosting, and they now officially support nested webroots. I actually stumbled on the documentation for this in a Github issue a while back, so I already am using this on my site. I recently moved my config out of the webroot, which is probably a good thing for security.

So we're all set now right? Well...

Scaffolding files and subtree splits

There are other complications, however. Most of Drupal's essential files are in its core folder. However a few files live outside of the core directory. These are referred to as scaffolding files, because they are mostly there to help you scaffold up your site and get it set up. By default, the Drupal Project will pull those scaffolding files off of what is called a subtree split of the main Drupal repo.

This normally works just fine. But when I recently tried to build a Drupal site from Composer for the 8.3.x branch, I found the subtree split files weren't available yet, because the branch was opened a few weeks ago. Resources are tight at the Drupal Association, and the process that creates the subtree split (which has to be done manually) had not been run yet. I was wondering if I was doing something wrong, but no a process just hadn't run. People are busy, that's understandable. Hopefully this process can get automated in the future to avoid this sort of problem.

I ran into this because I set up my local for core development using Composer. How I did that is another story: I ran into a few quirks.

Another reason you might end up caring about scaffolding files is if you use a webhost that has its own versions of the scaffolding files. I found that Pantheon had some special versions of some of the scaffolding files (primarily a Pantheon settings file.) So I had to add the following to the end of my composer.json file:

    "drupal-scaffold": {
      "source": "https://raw.githubusercontent.com/pantheon-systems/drops-8/{version}/{path}",
      "includes": [
        "sites/default/settings.pantheon.php"
      ]
    }

I may have modified a script file that ran the Scaffolding. It has been a while, so I am having trouble remembering. This took a while to figure out! Hopefully eventually it will be better documented. But the end result is that I can get the Scaffolding files from the drops-8 repo for Pantheon, rather than the drupal.org repo, which allows me to stay up to date with security updates for Drupal 8.

Can we go further down the rabbit hole? Yes!

Updating project dependencies

Right, so I mentioned early on in this post that you drush up was a handy command to update your modules or even Drupal itself to the latest version.

While Composer has a composer update command, you generally never want to use it on a project. That will iteratively go through all your dependencies and update them to the most recent version possible. However, Drupal core (or a contrib module) have not necessarily been tested with the latest versions of third-party PHP packages. In theory the version specification should be written in a way where updates shouldn't break anything, but it doesn't always work that way in practice.

Instead, you generally want to run composer update drupal/core to update Drupal itself, or for a contrib project composer update drupal/projectname. That will take care of updating just that item, without messing around too much with other dependencies.

However, as a I discovered with a recent security release, sometimes it is helpful to run composer update drupal/core --with-dependencies. There was security update for core that was supposed to update the third-party Guzzle library, but initially it only did so if you added --with-dependencies. After muddling around with figuring this out, a patch eventually got accepted to more tightly define the version requirements for Guzzle in Drupal's composer.json, so that --with-dependencies is not required. So you might want to use --with-dependencies going forward. I am still not entirely clear on that.

One thing you may find is that running composer update can take a while. 5-8 minutes is not uncommon. Having xdebug running on your local apparently slows things down. I have seen some things that can automatically disable xdebug when composer runs and re-enable it afterwards: I need to look into that more. Still, composer update, not fast.

Oh, and if you are working on a Composer-based project with other developers, you want to commit the composer.lock file to your repo. That file specifies the exact version of each package file that you need for the project. So if you make an update that changes the composer.lock versions, and then somebody else does a git pull on the repo, with that change, they run composer install to get the latest changes.

One of the reasons you probably do want your project composer.json file outside of your webroot is so that updates to Drupal core don't accidentally mess with any changes you made to composer.json. If you just use the composer.json file in the webroot that comes with the downloaded files from drupal.org or from downloading Drupal with git, then that composer.json file could get overwritten with updates. So if you have done a composer require to pull a contrib module or theme into your project, that could get wiped out with an update. You can also run into conflicts with updates to vendor files as well.

So keeping your composer.json file out of your webroot to manage your project is a safer way to avoid future conflicts with updates.

So I think you get the idea, which is that this is complicated.

Composer and site builders

You might think this is something not many people will run into as an issue. Maybe you're right! For now.

I found I needed to move to a Composer-based setup because I wanted to use the AMP module on my site. This is a module I worked on with my co-workers, and it helps to get your Drupal site outputting pages in an alternate AMP format. These AMP pages are super fast on mobile and can get shown on search engine result pages with special indicators letting people know they will load quickly.

Anyhow, the AMP module makes use of an external PHP library we developed. This PHP package can be used by other projects outside Drupal for processing HTML into AMP HTML, which is pretty nifty. The AMP module pulls in this library with composer.json, and some of AMP's dependencies conflicted with Drupal's, which is all taken care of with the autoloader... if the entire site is managed by Composer.

Another example: Drupal Console. This command line tool uses a lot of the same Symfony components that Drupal uses. Keeping those in sync, particularly with the dev versions of Drupal, can get pretty tricky. I found that if I wanted to use Drupal Console with my core development local, I needed to manage that local with Composer. Otherwise I would often run into version conflicts for certain dependencies, which resulted in inscrutable errors.

AMP is not the only module that sets its own PHP package dependencies with composer.json. As time goes on, I imagine more and more contrib modules will make use of Composer dependencies. As soon as your Drupal site needs just one contrib module with Composer dependencies, your entire site needs to be managed with Composer.

Let me repeat that: if your Drupal site uses only one contrib module with Composer dependencies, you need to move your entire site to a Composer-based install.

Maybe that's uncommon now, but I don't expect it to stay that way. Eventually, I think we'll see most Drupal 8 sites needing to be installed with Composer. That means Composer will be a tool not just for core developers, not just for contrib developers, but for site builders as well.

I am not opposed to us moving to Composer to manage Drupal sites. I think it has a lot of benefits and the approach makes sense.

I just know that I have been butting heads with Composer for months now. Maybe I am unique in having these issues, but I doubt it. I have to think others will run into these problems as well.

So while right now you can continue to download Drupal and contrib modules and themes from drupal.org, how long will that be true? Not because drupal.org won't allow those downloads, but because Composer will be required for most sites?

Hopefully with time we will figure out the issues people are having with Composer, document solutions, and things will get easier with time. I think it is just important to recognize that this transition is coming. This is a big change from Drupal 7 and previous versions of Drupal.

What effect will this have on how easy it is to use Drupal, particularly for site builders. I have grown more comfortable working on the command line, but for a long time, that was scary for me. Are we okay forcing nearly all site builders to interact with Drupal through the command line?

Or how about how composer-managed sites probably work best if the composer.json file sits outside the webroot, ideally with vendor and config files moved out of the webroot as well. Does that increase the hosting requirements for Drupal?

We're living in a Composer-based world now, so we're going to have to figure these things out. Hopefully we can find ways to mitigate these challenges, but that may not be easy. The way we work with Drupal 8 is just pretty different from Drupal 7, and we will need to adapt.

I started this by mentioning Composer has a conductor as its logo, just a minor change, just a sleight of hand. I think there is an analogy there, that Drupal 8 may look like previous versions of Drupal, but calling a conductor a composer does not necessarily make it so.