Skip to Content

Planet Drupal

Syndicate content
Drupal.org - aggregated feeds in category Planet Drupal
Updated: 1 day 10 hours ago

Makak Media: Taking BackDrop For A Test Drive

16 February 2015 - 8:41am

So the first BackDrop release is out there in the wild ready for a quick test drive! We're excited to see where this fork of Drupal 7 leads as we believe it to be a good complementary system to Drupal with a long term future.

First off we checked under the hood to get things configured and found the settings.php file in the root folder, which makes for easier access. Also all those txt files have been removed including the CHANGELOG.txt file, which we remove by default, as it supplies useful info to any hacker out there!

Naturally the installation process is very similar to Drupal but with a few less settings giving it a simpler feel.

Upon installation you're presented with a responsive admin menu with a slightly different structure to the standard Drupal menu. Responsiveness out of the box is great and the new menu again has a simpler look.

read more

Categories: Drupal

Acquia: Development based on Drupal's Fundamental Particles - Brad Czerniak

16 February 2015 - 4:34am
Language Undefined

Presenter Brad Czerniak caught my eye with a blog post entitled "10 things I learned using Drupal at a hackathon," based on his experiences taking part in the #hackDPL (Detroit Public Library) competitive hackathon. In our podcast interview we talk about that – before moving on to Brad's session about the Drupal development best practices he and his team use at Commercial Progression in Michigan.

Categories: Drupal

Annertech: Enlightening - The Dark Art of Solr Search with Drupal

16 February 2015 - 3:41am
Enlightening - The Dark Art of Solr Search with Drupal Why this blog post?

Often when I add a search function to a Drupal website using Apache Solr, I'm amazed at how complex some people think this is. Many developers/site builders are of the belief that this is some kind of very-hard-to-master black art. They could not be more wrong.

So what I want to contribute back to the Drupal community is an understanding of how Solr works, why/how it differs from Drupal Core Search module, and the benefits Solr has over core search.

Categories: Drupal

lakshminp.com: The Drupal 8 plugin system - part 2

16 February 2015 - 2:38am

We saw in part 1 how plugins help us in writing reusable functionality in Drupal 8. There are a lot of concepts which plugins share in common with services, like:

  1. limited scope. Do one thing and do it right.
  2. PHP classes which are swappable.

Which begs the question, how exactly are plugins different from services?
If your interface expects implementations to yield the same behaviour, then go for services. Otherwise, you should write it as a plugin. This needs some explaining.
For instance, if you are creating an interface to store data in a persistent system, like MySQL or MongoDB, then it would be implemented as a service. The save() function in your interface interface will be implemented differently for both the services, but the behaviour will be the same, i.e., it takes data as input parameters, stores them in the respective data store and returns a success message.

On the other hand, if you are creating an image effect, it needs to be a plugin. (It already is. Check image effects as plugins). The core concept of image plugins is to take in an image, apply an effect on it and return the modified image. Different image effects yield different behaviours. An image scaling effect might not produce the same behaviour as that of an image rotating effect. Hence, each of these effects need to be implemented as a plugin. If any module wants to create a new image effect, it needs to write a new plugin by extending the ImageEffectBase class.

Plugins used in core

Let's take a look at the major plugin types provided by Drupal 8 core. An example plugin of each plugin types will be the subjects of future blog posts.

  1. Blocks
    Drupal 8 finally got blocks right. Custom blocks can be created from the BlockBase class.

  2. Field Types, Field Widgets and Field Formatters
    Check part 1 for how this is done in Drupal 8.

  3. Actions
    Drupal 8 allows module developers to perform custom actions by implementing the ActionBase class. Blocking a user, unpublishing a comment, making a node sticky etc. are examples of actions.

  4. Image Effects
    Image effects are plugins which manipulate an image. You can create new image effects by extending ImageEffectBase. Examples of core image effects are CropImageEffect and ScaleImageEffect.

  5. Input filters
    User submitted input is passed through a series of filters before it is persisted in the database or output in HTML. These filters are implemented as plugins by implementing the FilterBase class.

  6. Entity Types
    In Drupal parlance, entities are objects that persist content or configuration in the database. Each entity is an instance of an entity type. New entity types can be defined using the annotation discovery mechanism.

  7. Views related plugins
    A large collection of different plugin types are employed by views during the querying, building and rendering stages.

Plugin Discovery

Plugin discovery is the process by which Drupal finds plugins written in your module. Drupal 8 has the following plugin discovery mechanisms:

  1. Annotation based. Plugin classes are annotated and have a directory structure which follows the PSR-4 notation.

  2. Hooks. Plugin modules need to implement a hook to tell the manager about their plugins.

  3. YAML files. Plugins are listed in YAML files. Drupal Core uses this method for discovering local tasks and local actions.

  4. Static. Plugin classes are registered within the plugin manager class itself. This is useful if other modules should not create new plugins of this type.

Annotation based discovery is the most popular plugin discovery method in use. We will briefly look at how we create a new plugin type using this method in the next part.

Categories: Drupal

Drupal core announcements: Drupal core security release window on Wednesday, February 18

15 February 2015 - 7:37pm
Start:  2015-02-18 (All day) America/New_York Online meeting (eg. IRC meeting) Organizers:  David_Rothstein

The monthly security release window for Drupal 6 and Drupal 7 core will take place on Wednesday, February 18.

This does not mean that a Drupal core security release will necessarily take place on that date for either the Drupal 6 or Drupal 7 branches, only that you should prepare to look out for one (and be ready to update your Drupal sites in the event that the Drupal security team decides to make a release).

There will be no bug fix release on this date; the next window for a Drupal core bug fix release is Wednesday, March 4.

For more information on Drupal core release windows, see the documentation on release timing and security releases, and the discussion that led to this policy being implemented.

Categories: Drupal

Drupalpress, Drupal in the Health Sciences Library at UVA: two new drupal distros – one for voting, one for 3d printing e-commerce

15 February 2015 - 12:29pm

Two new drupal distributions available on github

** https://github.com/alibama/cvillecouncilus is the distribution behind https://www.cvillecouncil.us - it’s an attempt to run a political campaign through a virtual proxy…

** https://github.com/alibama/rapid-prototyping-ecommerce-drupal – this is the code behind http://rpl.mae.virginia.edu/ it’s an e-commerce solution for 3d printing… A lot of this is implemented in rules and other well-standardized code thanks to Joe Pontani - a talented developer here in Virginia.  Joe integrated several third party tools, and set up the UVa payment gateway through Nelnet.

Both sites are getting updates over the next few months – the Charlottesville Council website also has a drupalgap implementation on it – absolutely awesome toolset…

18F API compliance is another feature I’m pretty stoked about… I got most of that done with the oauth2 server, views datasource, services and a couple of great notification features done with rules + views  i’ll get that feature out asap = it’s really convenient – matching a profile2 taxonomy field onto content taxonomy fields for notifications with new content.

any questions – please drop a line in the comments below

Categories: Drupal

DrupalOnWindows: Bypassing Form Validations and Required Fields in Drupal: the BFV module.

14 February 2015 - 10:00pm
Language English

Required or not required? To validate or not to validate? That is the question. So you've setup (the site builder's way, no custom forms) your required fields and custom validations for Node types, just to get this feedback from the customer:

That field we defined as mm..... as required (something trivial and not really critical such as an image file) is actually not always required. Users X and Y should be able to bypass that restriction.

More articles...
Categories: Drupal

Drupal @ Penn State: A window into our Community

14 February 2015 - 8:22am
Intro

Something that inspired me recently to write about DUG, are the efforts of MediaCurrent. Media Current has recently been pushing forward a series of postings talking about how they are giving back and being a lot more open about use of time to give back (which is awesome).

Categories: Drupal

Angie Byron: Webchick's "plain Drupal English" Guide to the Remaining Drupal 8 Critical Issues: DrupalCon Bogotá Edition

14 February 2015 - 1:09am

(Apologies for the atrocious state of the HTML that follows; this content is originally from this Google Doc.)

Webchick's "plain Drupal English" Guide to the Remaining Drupal 8 Critical Issues: DrupalCon Bogotá Edition

DrupalCon Bogotá just finished up, and critical issue-wise we've managed to stay in the 50s for a few days (down from a high of 150 last summer!), so now seems like as good a time as any to write down what's left to ship Drupal 8!

This post will attempt to document all of the remaining 55 criticals (as of this writing), and attempt to offer a somewhat "plain English" (or at least "Drupal English" ;)) description of each, loosely categorized into larger areas in which we could really use extra help. There are over 2,600 contributors to Drupal 8 at this time, please join us!

(Note: These descriptions might not be 100% accurate; this is my best approximation based on the issue summary and last few comments of each issue. If I got the description of your pet issue wrong, please update your issue summary. ;))

Table of contents

Quick vocabulary lesson

Current state of critical issues

Security

Security Parity with Drupal 7

Session and User Authentication API

REST

New security improvements

Performance

Profiling

Fix regressions relative to Drupal 7

Entity Field API

Views

Configuration system

"Fix it, or else"

General house-keeping

Other

Thrilling conclusion! (also known as "TL;DR")

Quick vocabulary lesson

Within this list, there are numerous "markers" used to signify that some of the issues in this list are more important to fix ASAP. These are:

  • D8 upgrade path: An issue tagged D8 upgrade path (currently, 13) means it blocks a beta-to-beta upgrade path for Drupal 8, generally because they materially impact the data schema or they impact security. Once we resolve all of these blockers, early adopters will no longer need to reinstall Drupal between beta releases, but can just run the update.php script as normal. This is currently our biggest priority.
  • Blocker: An issue tagged blocker (currently, 5) means it blocks other issues from being worked on. This is currently our second-biggest priority (or 0th priority in the case an issue blocks a D8 upgrade path issue :D). I've noted these as "sub-bullets" of the issues that are blocking them.
  • Postponed: Issues that are marked postponed (currently, 9) are either currently blocked by one of the "Blocker" issues, or we've deliberately chosen to leave off until later.
  • >30 days: These patches have a patch more than 30 days old, and/or were last meaningfully commented on >30 days ago. If you're looking for a place to start, re-rolling these is always helpful!
  • No patch: This issue doesn't have a patch yet. Oh the humanity! Want to give it a shot?

Other weird core issue nomenclature:

  • "meta" means a discussion/planning issue, with the actual patch action happening in related/child issues.
  • "PP-3" means "this issue is postponed on 3 other issues" (PP-1 means 1 other issue; you get the drift).
Current state of critical issues

Sections roughly organized from "scariest" to "least scary" in terms of how likely they are to make Drupal 8 take a longer time to come out.

Security

Because Drupal 8 hasn't shipped yet, it's not following Drupal's standard Security Advisory policy, so there are still outstanding, public security issues (13 as of this writing). We need to resolve most of these prior to providing a Drupal 8 beta-to-beta upgrade path, as this is the time when we signal to early adopters that it's an OK time to start cautiously building real sites on Drupal 8.

Skills needed: Various

Security Parity with Drupal 7

This class of security issue is to ensure that when Drupal 8 ships, it won't have any regressions security-wise relative to Drupal 7.

  • Port SA-CONTRIB-2013-096 to D8 (D8 upgrade path) Here's one such issue for Entity Reference module. SA-CONTRIB-2013-096 addressed a relatively esoteric remote access bypass bug, and the patch needs to be forward-ported to Drupal 8.
  • Port SA-CONTRIB-2015-039 to D8 (D8 upgrade path)  SA-CONTRIB-2015-039 addressed two issues in Views module, a redirect and default permissions for disabled views. The first was fixed in D8, but access checks are still missing from a few views for the second.

Session and User Authentication API

Because of various intricate dependencies, the authentication part of Drupal 8 isn't yet converted to object-oriented code, and prevents us from further optimizing bootstrap. This set of issues fixes various problems with this part of the code, and ensures these important security APIs are complete and ready to ship.

REST
  • REST user updates bypass tightened user account change validation (D8 upgrade path) Since Drupal 7, when you edit your user account, you have to provide the existing password when you want to change the password or e-mail. This security feature is currently by-passed by REST user updates as you can change the password or e-mail without providing the password.
  • External caches mix up response formats on URLs where content negotiation is in use (>30 days) Drupal 8's request processing system is currently based on content negotiation (which allows you to serve multiple versions of a document at the same URI based on what headers are sent e.g. Accept: text/html or Accept: application/json). This is generally considered the "right way" to do REST. However, various external caches and CDNs have trouble with this mechanism, and can mix them up and can send random formats back. The issue proposes changing from content negotiation to separate, distinct paths such as /node/1.json.

New security improvements

These issues affect new security improvements we want to make over and above what Drupal 7 does.

  • [meta] Document or remove every SafeMarkup::set() call One of the big security improvements in Drupal 8 is the introduction of Twig's autoescape feature, which ensures that all output to the browser is escaped by default. However, this is quite a big change that requires all of the code that was previously escaping content to stop doing that, else it gets double-escaped (so you start seeing < and " and whatnot in the UI). We originally introduced the ability to manually mark markup safe with SafeMarkup::set(), but the recommended approach is actually to use Twig everywhere, so this issue is to ensure that all remaining instances of the manual way are fixed, or at least documented to explain why they're using the non-recommended method.
  • Passing in #markup to drupal_render is problematic (>30 days) Another issue in the Twig autoescape space, we need to ensure that markup set by the "#markup" in e.g. form definitions is properly escaped.
  • Limit PDO MySQL to executing single statements if PHP supports it Remember SA-CORE-2014-005? Yeah, so do we. ;) This issue is to make sure that if another SQL injection vulnerability is ever found again, the damage it can do is more limited by eliminating the ability for MySQL to execute multiple queries per PDO statement.

Performance

Tied with security, 13 of the remaining issues are tagged Performance. While it may seem odd/scary to have this be a big chunk of the work left, it's a common practice to avoid premature optimization, and instead focus on optimization once all of the foundations are in place.

Skills needed: Profiling, caching, optimization, render API

Profiling

Here are a sub-set of issues where we need performance profiling to determine what gives us the biggest bang for our effort.

Fix regressions relative to Drupal 7
  • [meta] Resolve known performance regressions in Drupal 8 This is the main tracking issue in this space. During the 8.x cycle we've introduced several known performance regressions compared to Drupal 7 (sometimes to make progress on features/functionality, other times because we introduced changes that we hoped would buy us better scalability down the line), which we need to resolve before release so that Drupal 8 isn't slower than Drupal 7. The performance team meets weekly and tracks their progress in a detailed spreadsheet.
Entity Field API

Tracked under the Entity Field API tag (currently 6 issues).

Skills needed: Entity/Field API, Form API, Schema API

  • Schema for newly defined entity types is never created (D8 upgrade path) When you first install a module that defines an entity type (for example, Comment), its database tables are correctly generated. However, if an entity definition is later added by a developer to an already-installed module, the related database schema won't get created, nor will it be detected in update.php as an out-of-date update to run.
  • FileFormatterBase should extend EntityReferenceFormatterBase (D8 upgrade path) Entity Reference fields define a EntityReferenceFormatterBase class, which contains logic about which entities to display in the lookup, including non-existing entities and autocreated entities. File field's FileFormatterBase class currently duplicates that logic, except it misses some parts, including access checking, which makes this a security issue. The issue proposes to simply make File field's base class a sub-class of Entity Reference's, removing the need of "sort of but not quite the same" code around key infrastructure.
  • FieldTypePluginManager cannot instantiate FieldType plugins, good thing TypedDataManager can instantiate just about anything Currently, you get a fatal error if you attempt to use Drupal 8's Plugin API to create a new instance of a field type. The current code in core is avoiding this problem by going roundabout via the Typed Data API instead. This issue's critical because these are two of the most central APIs in Drupal 8, and they should work as expected.
  • [META] Untie content entity validation from form validation Despite all the work to modernize Drupal 8 into a first-class REST server, there still remain places where validation is within form validation functions, rather as part of the proper entity validation API, which means REST requests (or other types of workflows that bypass form submissions) are missing validation routines. This meta issue tracks progress of moving the logic to its proper place.
  • Entity forms skip validation of fields that are edited without widgets (>30 days) If a field can be edited with a form element that is not a Field API widget, we do not validate its value at the field-level (i.e., check it against the field's constraints). Fixing this issue requires ensuring that all entity forms only use widgets for editing field values.
  • Entity forms skip validation of fields that are not in the EntityFormDisplay (No patch, >30 days) Drupal 8 has a new feature called "form modes" (basically analogous to "view modes" in Drupal 7, except allowing you to set up multiple forms for a given entity instead). Currently, we're only validating fields that are displayed on a given form mode, even though those fields might have validation constraints on other fields that are not displayed. Critical because it could present a security issue.
Views

Views issues are generally tracked with the VDC tag. There are currently 6 criticals at this point which touch on Views (some already covered in earlier sections).

Configuration system

The configuration system is remarkably close to being shippable! Only 4 critical issues left. We're now working on finalizing the niggly bits around edge cases that involve configuration that depends on other configuration.

Skills needed: Configuration system, Entity Field API, Views

"Fix it, or else"

This subset of issues are things that are part of core currently, and we would really like to keep, but are willing to make some hard choices in the event they are among the last remaining criticals blocking release. The "postponed" among this list means "postponed until we're down to only a handful of criticals left." If these issues end up remaining in the list, we will move their functionality to contrib, and hope to add it back to core in a later point release if it gets fixed up.

Skills required: Various, but mainly low-level infrastructure and non-MySQL database skills.

  • [meta] Drupal.org (websites/infra) blockers to a Drupal 8 release (Blocker) This issue contains a "grab bag" of Drupal.org blockers that prevent an optimal Drupal 8 release, including things like semantic versioning support, testing support for multiple PHP/database versions, and support for Composer-based installations. If this issue is one of the last remaining criticals, we might choose to ship Drupal 8 anyway, and jettison one or more features in the process, such as…
  • [Meta] Make Drupal 8 work with PostgreSQL The meta/planning issue for fixing PostgreSQL (both in terms of functionality and in terms of failing tests). bzrudi71 is predominantly leading the charge here and making steady progress, but more hands would be greatly appreciated.
  • [meta] Database tests fail on SQLite (>30 days) Same deal as PostgreSQL but for SQLite. Unlike PostgreSQL though, this one doesn't have anyone leading the charge at this time, and it's also a lot harder to punt this to contrib, since we use it for various things such as testbot. Help wanted!

General house-keeping

These are all basic things we need to keep on top of between now and release, to ensure that when we're down to only a handful of criticals, we're ready to ship a release candidate. The good news is, these are also all generally really easy patches to make, and often also to test.

Skills needed: Basic patch rolling / reviewing / testing skills. (good for newbies!)

  • [meta] Ship minified versions of external JavaScript libraries (Postponed) Basically, in the Gilded Mobile Age™ we want to ensure that we're sending as little over the wire as possible, so scrunching various JS libraries down to the smallest possible file size needs to be the default. Separate issue from above because it needs to happen for both updated and existing JS libraries. Postponed because there'll be less work to do once all of the out-of-date JS libraries are updated and minified at the same time.
Other

I couldn't figure out a nice heading for these, so here's the rest.

  • Remove _system_path from $request->attributes Symfony provides a $request object, which has an "attributes" property for the purpose of storing various contextual bits. But the problem with $request->attributes->get('_MAGIC_KEY') is that the values are undocumented, there's no IDE autocompletion, and it's not clear which are internal vs. public properties, so we have an issue at [meta] Stop using $request->attributes->get(MAGIC_KEY) as a public API. to try and stop doing that.

    However, _system_path in particular is used a ton, since it's very common to want to know the path of the current request. The patch exposes a "CurrentPath" service instead, which eliminates all of those issues.
  • Potential data loss: concurrent node edits leak through preview Because the temp store that Drupal 8's new node preview system employs uses an entity's ID as the key, rather than something uniquely identifiable to a user, if two users are editing the same node and hit preview at the same time, one of them is going to lose data due to a race condition.
  • Ajax file uploads fail on IE 9 Pretty much exactly what it says on the tin. :P
Thrilling conclusion! (also known as "TL;DR")

Well, not so thrilling, but at least a conclusion. :)

  • Anywhere you see a blocker issue, attack it with fire. Those are holding other criticals up.
  • The biggest area of focus right now is D8 upgrade path blockers. Many of them are security issues.
  • Another big area is Performance, both fixing existing regressions, and profiling to determine where our biggest wins are.
  • Views and Entity Field API are tied in third place for number of remaining criticals. Let's have a race, shall we? ;)
  • The configuration system is looking pretty good, but still has a handful of sticky issues left.
  • There are a series of important features we'll lose if they're not fixed up in time.
  • If you're looking for something somewhat easy/mundane, help yourself to one of the general house-keeping issues.
  • Don't forget about the other miscellaneous issues I was too tired to categorize.

Sorry this post was so long (and probably has its share of inaccuracies) but I hope it will be helpful to some. It's basically what I needed to get back up to speed after taking a few months off of Drupal 8, so figured I'd document my way to understanding.

Now, let's get 'er done! :D

Tags: drupal 8drupaldrupal core diaries
Categories: Drupal

3C Web Services: Introduction to the Super Login Module for Drupal 7

13 February 2015 - 3:13pm
Drupal’s default login page form is functional but does leave a lot to be desired. It’s pretty bland and, if left as-is, is always a telltale sign that your site is a Drupal website. The Super Login Module for Drupal 7 is a simple way to improve the look and functionality of Drupal's login page.
Categories: Drupal

Stanford Web Services Blog: Behat Custom Step Definition: Wait for Batch API to Finish

13 February 2015 - 2:00pm

If you're using Behat and the Drupal Extension, you might find the following code snippet helpful if you want to add a step to wait for batch jobs to finish.

If one of your Behat scenarios kicks off a batch job (e.g., a Feeds import), and you want to wait for that batch job to finish before moving on to the next step, add this step definition in your FeatureContext.php file:

Categories: Drupal

Commerce Guys: Drupal Commerce Site Spotlight: Novusbio.com

13 February 2015 - 10:06am

We're always on the lookout for great sites built with Drupal Commerce, our truly flexible software that's changing the face of eCommerce one site at a time.

Perhaps the biggest strength of Drupal Commerce is it's flexibility, and that's clearly at work on the Novus Bio web site, a niche eCommerce site that's servicing a unique need in BioTech. Novus Biologicals features a commerce suite with a multitude of products available internationally for buyers of many different languages. Not to mention they are selling "cells", How cool is that?

 

To see Drupal Commerce sites we've Spotlighted in previous weeks view the Other Spotlight Sites

Categories: Drupal

Lullabot: Front-End Fundamentals, a Book Written by Bots

13 February 2015 - 7:22am

One of the coolest things about Lullabots is their desire to teach and share their knowledge. They do this in many formats: podcasts, articles, presentations, and even writing books. Joe Fender and Carwin Young decided there was an absolute need to write a book that brings all aspects of Front-End tools, frameworks, concepts, and procedures into one place — Front-End Fundamentals.

Categories: Drupal

InternetDevels: InternetDevels+Drupal = Love

13 February 2015 - 5:07am

We are serious about Drupal. Our relationship lasts for already 7 years by now. Today is St. Valentine’s Day — a good day to express our love to Drupal. Drupal united us and allowed making new friends, so it IS awesome and incredibly cool without any doubt! So here’s few reasons we love it (just listen to it, sounds like an ode to a real loved one):

Read more
Categories: Drupal

Iztok Smolic: 4 essential tips on implementing best practices

13 February 2015 - 4:27am

Drupal community talks a lot about best practices. When I talk about best practices I mean code driven development, code reviews, SCRUM, automated tests… I immediately realised that introducing new ways of working is not going to be easy. So I figured, why not asking one of the smart people how to start. Amitai (CTO of Gizra) was very kind to have […]

The post 4 essential tips on implementing best practices appeared first on Iztok.

Categories: Drupal

OpenLucius: Dependency injection in Drupal 8, an introduction.

13 February 2015 - 1:45am
Introduction

So, like a bunch of other Drupal people, we're also experimenting with Drupal 8; for our Drupal distro OpenLucius. Us being 'less is more'-developers, one aspect we really like is dependency injection.

Categories: Drupal

Jimmy Berry: The woes of the testbot

12 February 2015 - 4:43pm

For those not familiar with me, a little research should make it clear that I am the person behind the testbot deployed in 2008 that has revolutionized Drupal core development, stability, etc. and that has been running tens of thousands of assertions with each patch submitted against core and many contributed modules for 6 years.

My intimate involvement with the testbot came to a rather abrupt and unintended end several years ago due to a number of factors (which only a select few members of this community are clearly aware). After several potholes, detours, and bumps in the road, it became clear to me the impossibility of maintaining and enhancing the testbot under the policies and constraints imposed upon me.

Five years ago we finished writing an entirely new testing system, designed to overcome the technical obstacles of the current testbot and to introduce new features that would enable an enormous improvement in resource utilization that could then be used for new and more frequent QA.

Five years ago we submitted a proposal to the Drupal Association and key members of the community for taking the testbot to the next level, built atop the new testing system. This proposal was ignored by the Association and never evaluated by the community. The latter is quite puzzling to me given:

  • the importance of the testbot
  • the pride this open source community has in openly evaluating and debating literally everything (a healthy sentiment especially in the software development world)
  • I had already freely dedicated years of my life to the project.

The remainder of this read will:

  • list some of the items included in our proposal that were dismissed with prejudice five years ago, but since have been adopted and implemented
  • compare the technical merits of the new system (ReviewDriven) with the current testbot and a recent proposal regarding "modernizing" the testbot
  • provide an indication of where the community will be in five years if it does nothing or attempts to implement the recent proposal.

This read will not cover the rude and in some cases seemingly unethical behavior that led to the original proposal being overlooked. Nor will this cover the roller coaster of events that led up to the proposal. The intent is to focus on a technical comparison and to draw attention to the obvious disparity between the systems.

About Face

Things mentioned in our proposal that have subsequently been adopted include:

  • paying for development primarily benefiting drupal.org instead of clinging to the obvious falacy of "open source it and they will come"
  • paying for machine time (for workers) as EC2 is regularly utilized
  • utilizing proprietary SaaS solutions (Mollom on groups.drupal.org)
  • automatically spinning up more servers to handle load (e.g. during code sprints) which has been included in the "modernize" proposal
Comparison

The following is a rough, high-level comparison of the three systems that makes clear the superior choice. Obviously, this comparison does not cover everything.

table#testbot-comparison td { border: 1px solid white; } table#testbot-comparison td:nth-child(2), table#testbot-comparison td:nth-child(3), table#testbot-comparison td:nth-child(4) { width: 33% } table#testbot-comparison tr:nth-child(1), table#testbot-comparison td:nth-child(1) { font-weight: bold; font-size: 120%; } table#testbot-comparison td:nth-child(2) { background-color: #FFCC00; } table#testbot-comparison td:nth-child(3) { background-color: #D46A6A; color: white; } table#testbot-comparison td:nth-child(4) { background-color: #55AA55; color: white; }

Baseline Backwards modernization True step forward System Current qa.drupal.org "Modernize" Proposal ReviewDriven Status It's been running for over 6 years Does not exist Existed 5 years ago at ReviewDriven.com Complexity Custom PHP code and Drupal Does not make use of contrib code Mish mash of languages and environments: ruby, python, bash, java, php, several custom config formats, etc.

Will butcher a variety of systems from their intended purpose and attempt to have them all communicate

Adds a number of extra levels of communication and points of failure Minimal custom PHP code and Drupal

Uses commonly understood contrib code like Views Maintainability Learning curve but all PHP Languages and tools not common to Drupal site building or maintenance

Vast array of systems to learn and the unique ways in which they are hacked Less code to maintain and all familiar to Drupal contributors Speed Known; gets slower as test suite grows due to serial execution Still serial execution and probably slower than current as each separate system will add additional communication delay An order of magnitude faster thanks to concurrent execution

Limited by the slowest test case

*See below Extensibility (Plugins) Moderately easy, does not utilize contrib code so requires knowledge of current system Several components, one on each system used

New plugins will have to be able to pass data or tweak any of the layers involved which means writing a plugin may involve a variety of languages and systems and thus include a much wider breadth of required knowledge Much easier as it heavily uses commons systems like Views

Plugin development is almost entirely common to Drupal development:
define storage: Fields
define display: Views
define execution: CTools function on worker

And all PHP Security Runs as same user as web process Many more surfaces for attack and that require proper configuration Daemon to monitor and shutdown job process, lends itself to Docker style with added security 3rd party integration Basic RSS feeds and restricted XML-RPC client API Unknown Full Services module integration for public, versioned, read API and write for authorized clients Stability When not disturbed, has run well for years, primary causes of instability include ill-advised changes to the code base

Temporary and environment reset problems easily solved by using Docker containers with current code base Unknown but multiple systems imply more points of failure Same number of components as current system

Services versioning which allows components to be updated independently

Far less code as majority depends on very common and heavily used Drupal modules which are stable

2-part daemon (master can react to misbehaving jobs)

Docker image could be added with minimal effort as system (which predates Docker) is designed with same goals as Docker Resource utilization Entire test suite runs on single box and cannot utilize multiple machines for single patch Multiple servers with unshared memory resources due to variety of language environments

Same serial execution of test cases per patch which does not optimally utilize resources An order of magnitude better due to concurrent execution across multiple machines

Completely dynamic hardware; takes full advantage of available machines.

*See below Human interaction Manually spin up boxes; reduce load by turning on additional machines Intended to include automatic EC2 spin up, but does not yet exist; more points of failure due to multiple systems Additional resources are automatically turned on and utilized Test itself Tests could be run on development setup, but not within the production testbot Unknown Yes, due to change in worker design.

A testbot inside a testbot! Recursion! API Does the trick, but custom XML-RPC methods Unknown Highly flexible input configuration is similar to other systems built later like travis-ci

All entity edits are done using Services module which follows best practices 3rd party code Able to test security.drupal.org patches on public instance Unknown, but not a stated goal Supports importing VCS credentials which allows testing of private code bases and thus supports the business aspect to provide as a service and to be self sustaining

Results and configuration permissioned per user to allow for drupal.org results to be public on the same instance as private results Implemented plugins Simpletest, coder None exist Simpletest, coder, code coverage, patch conflict detection, reroll of patch, backport patch to previous branch Interface Well known; designed to deal with display of several 100K distinct test results; lacks revision history; display uses combination of custom code and Views Unknown as being built from scratch and not begun

Jenkins can not support this interface (in Jenkins terminology multiple 100K jobs) so will have to be written from scratch (as proposal confirms and was reason for avoiding Jenkins in past)

Jenkins was designed for small instances within businesses or projects, not a large central interface like qa.drupal.org Hierarchical results navigation from project, branch, issue, patch

Context around failed assertion (like diff -u)

Minimizes clutter, focuses on results of greatest interest (e.g. failed assertions); entirely built using Views so highly customizable

Simplified to help highlight pertinent information (even icons to quickly extract status)

Capable of displaying partial results as they are concurrently streamed in from the various workers Speed and Resource Utilization

Arguably one of the most important advantages of the ReviewDriven system is concurrency. Interestingly, after seeing inside Google I can say this approach is far more similar to the system Google has in place than Jenkins or anything else.

Systems like Jenkins and especially travis-ci, which for the purpose of being generic and simpler, do not attempt to understand the workload being performed. For example Travis simply asks for commands to execute inside a VM and presents the output log as the result. Contrast that with the Drupal testbot which knows the tests being run and what they are being run against. Why is this useful? Concurrency.

Instead of running all the test cases for a single patch on one machine, the test cases for a patch may be split out into separate chunks. Each chunk is processed on a different machine and the results are returned to the system. Because the system understands the results it can reassemble the chunked results in a useful way. Instead of an endlessly growing wait time as more tests are added and instead of having nine machines sitting idle while one machine runs the entire test suite all ten can be used on every patch. The wait time effectively becomes the time required to run the slowest test case. Instead of waiting 45 minutes one would only wait perhaps 1 minute. The difference becomes more exaggerated over time as more tests are added.

In addition to the enormous improvement in turnaround time which enables the development workflow to process much faster you can now find new ways to use those machine resources. Like testing contrib projects against core commits, or compatibility tests between contrib modules, or retesting all patches on commit to related project, or checking what other patches a patch will break (to name a few). Can you even imagine? A Drupal sprint where the queue builds up an order of magnitude more slowly and runs through the queue 40x faster?

Now imagine having additional resources automatically started when the need arises. No need to imagine...it works (and did so 5 years ago). Dynamic spinning up of EC2 resources which could obviously be applied to other services that provide an API.

This single advantage and the world of possibility it makes available should be enough to justify the system, but there are plenty more items to consider which were all implemented and will not be present in the proposed initiative solution.

Five Years Later

Five years after the original proposal, Drupal is left with a testbot that has languished and received no feature development. Contrast that with Drupal having continued to lead the way in automated testing with a system that shares many of the successful facets of travis-ci (which was developed later) and is superior in other aspects.

As was evident five years ago the testbot cannot be supported in the way much of Drupal development is funded since the testbot is not a site building component placed in a production site. This fact drove the development of a business model that could support the testbot and has proven to be accurate since the current efforts continue to be plagued by under-resourcing. One could argue the situation is even more dire since Drupal got a "freebie" so to speak with me donating nearly full-time for a couple of years versus the two spare time contributors that exist now.

On top of lack of resources the current initiative, whose stated goal is to "modernize" the testbot, is needlessly recreating the entire system instead of just adding Docker to the existing system. None of the other components being used can be described as "modern" since most pre-date the current system. Overall, this appears to be nothing more than code churn.

Assuming the code churn is completed some time far in the future; a migration plan is created, developed, and performed; and everything goes swimmingly, Drupal will have exactly what it has now. Perhaps some of the plugins already built in the ReviewDriven system will be ported and provide a few small improvements, but nothing overarching or worth the decade it took to get there. In fact the system will needlessly require a much rarer skill set, far more interactions between disparate components, and complexity to be understood just to be maintained.

Contrast that with an existing system that can run the entire test suite against a patch across a multitude of machines, seamlessly stitch the results together, and post back the result in under a minute. Contrast that with having that system in place five years ago. Contrast that with the whole slew of improvements that could have also been completed in the four years hence by a passionate, full-time team. Contrast that with at the very least deploying that system today. Does this not bother anyone else?

Contrast that with Drupal being the envy of the open source world, having deployed a solution superior to travis-ci and years earlier.

Please post feedback on drupal.org issue.

Tags:
Categories: Drupal

DrupalCon News: A place for Drupal.org at DrupalCon

12 February 2015 - 3:19pm

DrupalCon Los Angeles will be the first Con where Drupal.org, home of Drupal and the Drupal community, has its very own track.

The track will feature presentations from the Drupal Association Engineering Team, where they share long and short term plans for website development, demo new and upcoming features, and gather community feedback.

A limited amount of spots are available for sessions submitted from the community. That’s where you come in.

Categories: Drupal

Mediacurrent: Efficient Drupal Development with Tmux and Tmuxinator

12 February 2015 - 2:41pm

Have you ever wished you could just type one command and load up all of the things you need to work on for a project? Wouldn’t it be nice to have your terminal set up with the correct Drush alias, tailing the watchdog, with access to your servers just a couple keystrokes away? Sounds nice, right?

Categories: Drupal

Victor Kane: Setting up a Reusable and DurableDrupal Lean Process Factory - Presentation 2/11/2015 at DrupalCon Latin America 2015

12 February 2015 - 1:52pm

[para español ver más abajo]

The purpose of the presentation was to describe how to use reusable tools and processes, tailored and in constant evolution, in order to finally defeat waterfall and guarantee delivered value in the development of websites and web applications.

The following main topics were covered in depth:

  • Kanban (not Scrum)
  • Project Inception and Vision
  • Team Kickoff
  • Development Workflow with Everything in Code
  • DevOps, Server Provisioning and Deployment
  • User Validation

Links to resources:

This is a huge amount of material, based on both my successful and unsuccessful experiences, and I earnestly hope it will help other web centered knowledge workers. If you have questions, please ask them on twitter @victorkane with hashtag #DurableDrupalLean. There were quite a few other fascinating and very good presentations on the subject of Process and DevOps, overlapping my own substantially and it should be very worthwhile to share them here: I greatly appreciate having had the opportunity to present at this incredibly important, fun and well-organized DrupalCon. See you all in Los Angeles and Rio de Janeiro!

read more

Categories: Drupal


Google+
about seo