JS and the future of Drupal's front end: A chat with Socrates

Drew power walked down the gravel path, fists pumping back and forth. A cloud of birds burst from a nearby bush, scattering in all directions. Eyes widening, Drew skidded to a halt. Stiff arms pressed down, dug into hips. Short, tattered breaths settled into a long sigh and a groan.

“You spooked them.”

Drew glanced to the side. There sat a world-weary figure, perched on a bench, eyes glinting with mirth. “Didn’t mean to spook them, just trying to get where I need to go.”

With a nod and a chuckle, a wrinkled hand patted the bench. “Sit a minute, a moment or two won’t hurt.”

Drew eyed the trail ahead. A long, tangled mass of greenery in the woods encroached on the path. “A moment.” With a wince, Drew eased down onto the stone bench and shot out a hand. “Name’s Drew. You are?”

“Hey, pal, nice to meet you. Some people call me Socrates. Friends call me Jay, though.”

Drew eyed the bushy white hair and raised an eyebrow. “Jay? Okay, nice to meet you, Jay.”

“You seem to be in quite the hurry, Drew. What’s the rush?”

“I really don’t have time to waste. Have to disrupt myself before somebody else disrupts me.”

Jay smirked. “Sounds exciting! What exactly are you disrupting?”

“Big content management system. Drupal. Open source. PHP-based.” Drew paused, looked back and forth as if worrying somebody would overhear. “At least for now.”

“Oh yes! I’ve heard of Drupal. Come for the code, stay for the community. Good folks!” Jay looked up to the sky. Eyes fluttered shut as sun broke through the branches above and washed over them. “You just released Drupal 8, right? I thought that was a big disruption. Nearly a complete rewrite of the backend. Moving to object-oriented code. Symfony 2. Getting off the island. Right?”

Drew’s brow furrowed. “I guess you are familiar with Drupal. Yes, we certainly had some big changes.”

“And now you want to make some more big changes? What sort of changes?”

“Well.” Drew’s lips pressed together. “Some people find Drupal tricky to use. We need to make that easier, so we need to get a client-side JS framework into core. That will solve a lot of our issues, but we need to get this done fast, or we’ll be left behind.”

“Ah, yes, I think I heard WordPress did something similar recently. Calypso?”

“Well, that’s different. Calypso works with the admin side of the hosted wordpress.com service. And we were, um, totally thinking of doing this before that got announced.”

“Of course, of course. Another content management system that runs on PHP that is looking to leverage JS to improve its admin capabilities. I can see how you wouldn’t be worried about that similar effort leaving you behind, as you said.”

“No,” Drew snapped. “Nothing to do with that. People’s expectations are just getting higher and higher through the web apps they use. Gmail. Facebook. Trello. These days people can do tasks a lot more smoothly than they did in the past. Actions take place without full page refreshes. They can check a box on a form, and the page updates right away without having to wait to hear back from the server.”

“Oh yes! That’s very nice. I like it when things are nice and easy. I’ve heard some Drupal sites are using these JS frameworks sort of independently of Drupal. Taking data from Drupal’s REST API and then running the framework on a Node JS server.”

Drew’s fingers tapped, tapped, tapped on the bench. “Yes.” Tap, tap, tap. “Yes, you can do that.” With a low voice, Drew muttered, “Decoupled, headless, whatever you call it.”

“Doesn’t that allow Drupal to create these smooth single page apps you were saying were so easy to use?”

“Well that’s missing out on the best part of Drupal!” Drew’s face reddened. “That cuts out our internationalization! Our UI for field and block placement has no effect in a decoupled solution! No previews! All the work we put into accessibility! Caching improvements! Our toolbar!”

“Oh.” Jay’s hand reached out, resting on Drew’s shoulders. “That’s what you’re worried about. That decoupled solutions eliminate some of the benefits of working with Drupal.”

Drew stared at the ground. “If Drupal’s just a content store, somebody could step in and replace that layer.”

“But Drupal is really strong with content modeling, right? That would still be useful, right?”

“Even so.” Drew shook off Jay’s hand. “The tools for working with our content model need to be improved. Those need to be easy. Decoupled Drupal doesn’t change that. We need our admin tools to be just as smooth to use as other web apps.”

“Okay, fair enough. And you need a JS framework to do that?”

“Yes!” Drew shot up from the bench. “Bringing in a framework will fix everything!”

Jay’s hands raised in surrender. “I see! Well I don’t always understand these things, so could you explain how?”

Drew’s pointer finger stabbed the air. “If we want to improve our UX, we need front-end developers to do it. Bring in a framework, bring in front-end developers who are comfortable working with that framework.” A second finger joined the first. “Two. A framework allows us to get the sort of rich interactions we want without building it ourselves. We’ll get off the island with our front end.”

“Ah, I think I see.” Jay’s lips pursed a moment. “Maybe I’m remembering this wrong, but didn’t you pull in a new JS framework in Drupal 8?”

Drew’s foot traced a circle in the gravel. “Backbone. Backbone JS.”

“Oh yes, one of the big JS frameworks, right?”

“A few years ago, yes. When we picked it.”

“Did that bring in a lot of front-end developers?”

Drew kicked some of the gravel out of the way. “Listen, it’s going to be different this time. JS has changed a lot. Now there’s Virtual DOM diffing for quicker page refreshes! Universal JS, so JavaScript can render on both the server and the client for fast page loading! You should see how many stars these projects have on GitHub, JS is the wave of the future.”

“Oh! So JS frameworks have changed a lot in the last few years, and Backbone isn’t one of the popular ones now, so you need to replace it? What are the kids using these days… Angular? Ember? React?”

“Angular 2, Ember 2.x, React is one layer of a project stack, not quite the same as the others, but yes, yes.”

“So these new frameworks, you can get those into Drupal right away?”

“We can start to experiment. We can’t break backwards compatibility, so some things might need to wait until Drupal 9…”

“Which is when?”

“When it’s done,” Drew snapped.  “Probably a few years at least.”

“Okay, so JS frameworks have changed considerably in the last few years since Backbone went in to the point that Backbone needs to be replaced with something else, which will take a few years to get in. Am I getting this right?”

Drew stood there, silent.

“By your silence, I’m guessing that Backbone didn’t end up bringing in a bunch of new front-end developers, since you want to use something new now. At least in part in because JS has changed a lot since it went in. Won’t the same thing be true of any new JS framework you put into Drupal? Front-end developers can be pretty opinionated about which frameworks they prefer to work with, right? By the time it’s in, isn’t it likely that front-end developers might be using some other JS framework that doesn’t exist yet?”

Drew’s arms stretched out. “Big companies are backing these frameworks, Jay! Google backs Angular! Facebook backs React! Ember has a big community!”

“Yes! Google is a big company! They’ve made so many cool products, like Google Reader!”

“They don’t support that anymore.”

“Well I liked their image editing tool, what was it, Picasa?”

“That’s going away too.”

“Oh, maybe I should post that to my Google Plus account.”

“They’re not emphasizing that anymore.” Drew’s arms dropped. “Okay, I get it. Just because Google is big, they might not always support Angular.”

Jay smiled kindly. “Oh, you never know.”

“Right.” Drew’s nose wrinkled. “Right. So, we also have to replace Backbone because it hasn’t kept up with the times. React introduced Virtual DOM diffing, but Backbone doesn’t have that. Angular and Ember are introducing their versions. React pioneered server-side rendering. Angular and Ember are working on their versions.”

“Oh, so you’re looking at using React? They seem like they’re on the cutting edge.”

“Probably not. Their patent policies and licensing don’t match up with Drupal very well.”

“I see. Well, that does matter quite a bit.” A bird chirped in a nearby tree. Jay cocked his head, listening to the little song. “Such a cheerful bird up there, perched on its twig. Twig! You overhauled Drupal’s templates recently, right?”

Drew beamed. “One of the best new features in Drupal 8! Front-enders are very excited about it.”

“Oh, why?”

“Well, it’s much easier to make changes to markup now. We’re not mixing hard-code programming with our markup as much as before. This way is much more secure.”

“These new frameworks, do they use Twig for their markup? Or do they mix markup with this programming code stuff?”

“Well… React mixes JS and markup, typically with JSX. Ember uses HTMLBars, which is based on Handlebars. Similar to Twig, not quite the same. I did see somebody was getting Twig to work with Angular.”

“Oh, that would be nice. So you wouldn’t have to use different templates for JS and for the PHP rendering. Say, weren’t you talking about how JS frameworks can now render on the server?”

“Sure, usually with Node JS.”

“Oh, but Drupal runs on PHP, right?”

Drew’s eyes brightened. “Yes, but we have this new idea! Progressive decoupling! Most of the page renders with PHP, but the more interactive bits are rendered with a JS framework.”

“Oh neat! What other projects are doing that?”

“We’d be the first!”

“The first, oh very exciting!” Jay’s eyes crinkled. “I’m trying to remember… you had a mobile initiative for Drupal 8, right?”

“Yes, we thought it was very important for Drupal to work well no matter what device is used to access it.”

“Even for the admin interface?”

“Absolutely.”

“So if you’re using a JS framework for these interactive bits, and they’re running on the same server as Drupal… you’re not doing server-side JS rendering, right? You said that was usually done with Node JS. So the JS framework will render on the client side. I thought that could be pretty slow.”

Drew’s mouth shot open, then slowly shut. Quietly, Drew said. “We might need to run this framework on Node JS to get the full benefits of server-side rendering.”

“Oh! That would be a big change! You’d have to rewrite Drupal’s rendering layer in JS?”

“Maybe.” Drew eyed the branches overhead, trying not to make eye contact. “Maybe parts of it. Some of that might need to wait until Drupal 9.”

“Couldn’t that extend how long it takes to release Drupal 9?”

“There’s no need to spread fear, uncertainty and doubt!”

“Well, with Drupal 8, you started with just a few components of Symfony 2, and that expanded over time, right? And led to more object oriented PHP?”

“Yes, I suppose so.”

“What effect did that have on the Drupal community?”

“It did bring in more people to Drupal from more object oriented languages.”

“That’s great! What about existing developers?”

“Many have adapted.”

“But not all.”

“Well… some forked Drupal and created Backdrop.”

“In part because of a really big change to the way Drupal worked that was intimidating and would require a great deal of change?”

“Sure, in part.”

“What effect did that have on Drupal businesses?”

“There was a lot of uncertainty. Drupal 8 took a long time to come out, sure, but we had really great features. And Drupal clients wanted to wait on new sites until they could take advantage of how great Drupal 8 was!”

“Did that hurt Drupal-based businesses when clients held off on getting new sites?”

“Maybe. Hard to say. They got through it.”

“And that was a big change that just involved changing the flavor of the same language, PHP?”

“I guess you could put it that way.”

Jay’s fingers curled around the edge of the park bench. “And now you want to switch large parts of Drupal away from PHP altogether?”

Drew looked to the side.

“Is it possible that the amount of Drupal that you need to convert to JS might be more than you initially expect, just as Symfony led to more changes than you initially expected.”

“We…” Drew’s nose twitched. “We can’t just wait. Until we get disrupted.”

“Okay, I understand that.” Jay stretched and slowly uncurled from the bench. Fingers reached towards the sky as Jay stood. “So I understand you want to improve the experience of Drupal. It sounds like avoiding full page reloads is important to you. Getting feedback without waiting from the server.”

“Optimistic feedback.”

“Right. Okay, well let me ask you this. Do you think you’ll use the full capabilities of a JS framework? Don’t a lot of those have routing as a big component? Do you think you’ll use the framework to handle routing?”

“We haven’t really talked about that.”

“It sure does sound like JS will be needed for some of the interactions you want. Are there ways to provide those without a framework?”

“I’m not sure we can build those on our own. We’d need more JS developers working within Drupal, and those have been hard to come by. And we’re trying to get off the island by not developing everything ourselves.”

“That’s fair. It does sound like if you want to meet all your goals, frameworks could take quite a lot of work to implement with the amount that you’d need to overhaul in Drupal. Are there smaller ways you can improve Drupal’s UX?”

“Iteration? I don’t think we can make big enough changes just through iteration.”

“What about PHP? There seems to be a lot of movement in the PHP world these days. PHP7 is out. I keep hearing about a PHP Renaissance.”

Drew’s hand flew up into the air. “Come on, everybody is using JS these days! PHP is old hat and going away!”

“Oh. Okay. I thought a pretty large percentage of the web was still powered by PHP.”

Drew shrugged.

“Are there some new PHP technologies that might help speed up Drupal’s interactions? I think somebody mentioned long-running PHP daemons that could help avoid full bootstraps of Drupal when JS interacts with the server.”

“Apache isn’t optimized to handle long-running PHP daemons! That will exhaust memory quickly! We’d need to make big changes to the server requirements for Drupal?”

“Bigger changes than running Drupal on Node JS?”

“Well.”

“If part of Drupal, the back end, is still running on PHP, while the front end runs on Node JS… that sounds like a pretty complicated setup. How would part of Drupal run on PHP while the other part runs on Node?”

“Well…”

“Let me ask you this, Drew. Let’s say you go forward with converting big chunks of Drupal to JS. However deep that goes, that’s going to be a lot of work, right?”

Drew frowned. “Yeah. A fair amount.”

“Is it possible there are some things you might not be able to improve in Drupal because of this big overhaul? You just made some pretty big changes in Drupal 8. There’s probably a lot of follow-up changes you’d like to see with that, along with UX improvements?”

“Sure. Much of Drupal is object-oriented now, but there are probably some consistencies and efficiencies we could work on. Our Form API and Render API could probably be made more object oriented. If we still used PHP for rendering.”

“And a lot of the JS frameworks would require some pretty robust REST API support on the admin side, right?”

“Sure, we’re looking at GraphQL. There’s probably REST work necessary no matter what.”

Jay put an arm around Drew. “I don’t know very much about these things, but it seems like you have two paths forward. You could move forward with a big JS overhaul of Drupal. It could seriously lengthen the time it takes for the next release of Drupal, and the big changes could end up being pretty intimidating for the community. It might bring in new developers, but given the pace of the JS world, it’s possible you could go through all that work only to have an outdated framework that’s more deeply embedded in Drupal than Backbone ever was.”

Jay and Drew looked down the long tangled path ahead.

“Or.” Jay guided Drew back to the bench, and the two eased back onto the stone. They both took a deep breath. For the first time, Drew saw the multitude of birds in this little grove. The undergrowth was a bit overgrown. The fresh smell of pine needles in the crisp air comforted Drew. Jay smiled again. “Or you could hang out here a while. Help me clean up this grove a bit. It’s quite lovely. I bet we could make this an even nicer spot if we worked together.”

“But disruption…”

“Sometimes iteration is disruption. A few changes here and there, and then a few more, and on and on. That can make a big difference over time.”

“But we need more front-end developers…”

“Are your existing front-end developers in the Drupal community asking for this JS overhaul?”

“Well, some are pushing back on the idea.”

“Is the outside JS community begging you to do this?”

“There may be some interest from a few people.”

“Maybe if you want front-end developers to help make Drupal better, you could start by listening to existing front-end developers who care about Drupal and trust what they have to say?”

“That’s not fair, there are definitely front-end developers who want to move forward with this.”

“Most of them?”

“Well…”

“Do you think you could still make solid improvements to Drupal’s UX with some engaged front-end developers excited about using newer JS and PHP tools to provide richer interactions even if they’re not using a framework?”

“Maybe?”

Drew and Jay sat on the bench for some time, listening to the birds sing.