Planet Drupal

Subscribe to Planet Drupal feed
Drupal.org - aggregated feeds in category Planet Drupal
Updated: 6 hours 29 min ago

Agiledrop.com Blog: Top Drupal blog posts from July 2019

5 August 2019 - 11:21pm

At the start of every month, we gather all the Drupal blog posts from the previous month that we’ve enjoyed the most. Here’s an overview of our favorite posts from July related to Drupal - enjoy the read!

READ MORE
Categories: Drupal

Agaric Collective: Using constants and pseudofields as data placeholders in the Drupal migration process pipeline

5 August 2019 - 8:00pm

So far we have learned how to write basic Drupal migrations and use process plugins to transform data to meet the format expected by the destination. In the previous entry we learned one of many approaches to migrating images. In today’s example, we will change it a bit to introduce two new migration concepts: constants and pseudofields. Both can be used as data placeholders in the migration timeline. Along with other process plugins, they allow you to build dynamic values that can be used as part of the migrate process pipeline.

Setting and using constants

In the Migrate API, a constant is an arbitrary value that can be used later in the process pipeline. They are set as direct children of  the source section. You write a constants key whose value is a list of name-value pairs. Even though they are defined in the source section, they are independent of the particular source plugin in use. The following code snippet shows a generalization for settings and using constants:

source: constants: MY_STRING: 'http://understanddrupal.com' MY_INTEGER: 31 MY_DECIMAL: 3.1415927 MY_ARRAY: - 'dinarcon' - 'dinartecc' plugin: source_plugin_name source_plugin_config_1: source_config_value_1 source_plugin_config_2: source_config_value_2 process: process_destination_1: constants/MY_INTEGER process_destination_2: plugin: concat source: constants/MY_ARRAY delimiter: ' '

You can set as many constants as you need. Although not required by the API, it is a common convention to write the constant names in all uppercase and using underscores (_) to separate words. The value can be set to anything you need to use later. In the example above, there are strings, integers, decimals, and arrays. To use a constant in the process section you type its name, just like any other column provided by the source plugin. Note that you use the constant you need to name the full hierarchy under the source section. That is, the word constant and the name itself separated by a slash (/) symbol. They can be used to copy their value directly to the destination or as part of any process plugin configuration.

Technical note: The word constants for storing the values in the source section is not special. You can use any word you want as long as it does not collide with another configuration key of your particular source plugin. A reason to use a different name is that your source actually contains a column named constants. In that case you could use defaults or something else. The one restriction is that whatever value you use, you have to use it in the process section to refer to any constant. For example:

source: defaults: MY_VALUE: 'http://understanddrupal.com' plugin: source_plugin_name source_plugin_config: source_config_value process: process_destination: defaults/MY_VALUESetting and using pseudofields

Similar to constants, pseudofields stores arbitrary values for use later in the process pipeline. There are some key differences. Pseudofields are set in the process section. The name is arbitrary as long as it does not conflict with a property name or field name of the destination. The value can be set to a verbatim copy from the source (a column or a constant) or they can use process plugins for data transformations. The following code snippet shows a generalization for settings and using pseudofields:

source: constants: MY_BASE_URL: 'http://understanddrupal.com' plugin: source_plugin_name source_plugin_config_1: source_config_value_1 source_plugin_config_2: source_config_value_2 process: title: source_column_title my_pseudofield_1: plugin: concat source: - constants/MY_BASE_URL - source_column_relative_url delimiter: '/' my_pseudofield_2: plugin: urlencode source: '@my_pseudofield_1' field_link/uri: '@my_pseudofield_2' field_link/title: '@title'

In the above example, my_pseudofield_1 is set to the result of a concat process transformation that joins a constant and a column from the source section. The result value is later used as part of a urlencode process transformation. Note that to use the value from my_pseudofield_1 you have to enclose it in quotes (') and prepend an at sign (@) to the name. The new value obtained from URL encode operation is stored in my_pseudofield_2. This last pseudofield is used to set the value of the URI subfield for field_link. The example could be simplified, for example, by using a single pseudofield and chaining process plugins. It is presented that way to demonstrate that a pseudofield could be used as direct assignments or as part of process plugin configuration values.

Technical note: If the name of the subfield can be arbitrary, how can you prevent name clashes with destination property names and field names? You might have to look at the source for the entity and the configuration of the bundle. In the case of a node migration, look at the baseFieldDefinitions() method of the Node class for a list of property names. Be mindful of class inheritance and method overriding. For a list of fields and their machine names, look at the “Manage fields” section of the content type you are migrating into. The Field API prefixes any field created via the administration interface with the string field_. This reduces the likelihood of name clashes. Other than these two name restrictions, anything else can be used. In this case, the Migrate API will eventually perform an entity save operation which will discard the pseudofields.

Understanding Drupal Migrate API process pipeline

The migrate process pipeline is a mechanism by which the value of any destination property, field, or pseudofield that has been set can be used by anything defined later in the process section. The fact that using a pseudofield requires to enclose its name in quotes and prepend an at sign is actually a requirement of the process pipeline. Let’s see some examples using a node migration:

  • To use the title property of the node entity, you would write @title
  • To use the field_body field of the Basic page content type, you would write @field_body
  • To use the my_temp_value pseudofield, you would write @my_temp_value

In the process pipeline, these values can be used just like constants and columns from the source. The only restriction is that they need to be set before being used. For those familiar with the rewrite results feature of Views, it follows the same idea. You have access to everything defined previously. Anytime you use enclose a name in quotes and prepend it with an at sign, you are telling the migrate API to look for that element in the process section instead of the source section.

Migrating images using the image_import plugin

Let’s practice the concepts of constants, pseudofields, and the migrate process pipeline by modifying the example of the previous entry. The Migrate Files module provides another process plugin named image_import that allows you to directly set all the subfield values in the plugin configuration itself.

As in previous examples, we will create a new module and write a migration definition file to perform the migration. It is assumed that Drupal was installed using the standard installation profile. The code snippets will be compact to focus on particular elements of the migration. The full code is available at https://github.com/dinarcon/ud_migrations The module name is UD Migration constants and pseudofields and its machine name is ud_migrations_constants_pseudofields. The id of the example migration is udm_constants_pseudofields. Refer to this article for instructions on how to enable the module and run the migration. Make sure to download and enable the Migrate Files module. Otherwise, you will get an error like: “In DiscoveryTrait.php line 53: The "file_import" plugin does not exist. Valid plugin IDs for Drupal\migrate\Plugin\MigratePluginManager are:...”. Let’s see part of the source definition:

source: constants: BASE_URL: 'https://agaric.coop' PHOTO_DESCRIPTION_PREFIX: 'Photo of' plugin: embedded_data data_rows: - unique_id: 1 name: 'Michele Metts' photo_url: 'sites/default/files/2018-12/micky-cropped.jpg' photo_width: '587' photo_height: '657'

Only one record is presented to keep snippet short, but more exist. In addition to having a unique identifier, each record includes a name, a short profile, and details about the image. Note that this time, the photo_url does not provide an absolute URL. Instead, it is a relative path from the domain hosting the images. In this example, the domain is https://agaric.coop so that value is stored in the BASE_URL constant which is later used to assemble a valid absolute URL to the image. Also, there is no photo description, but one can be created by concatenating some strings. The PHOTO_DESCRIPTION_PREFIX constant stores the prefix to add to the name to create a photo description. Now, let’s see the process definition:

process: title: name psf_image_url: plugin: concat source: - constants/BASE_URL - photo_url delimiter: '/' psf_image_description: plugin: concat source: - constants/PHOTO_DESCRIPTION_PREFIX - name delimiter: ' ' field_image: plugin: image_import source: '@psf_image_url' reuse: TRUE alt: '@psf_image_description' title: '@title' width: photo_width height: photo_height

The title node property is set directly to the value of the name column from the source. Then, two pseudofields. psf_image_url stores a valid absolute URL to the image using the BASE_URL constant and the photo_url column from the source. psf_image_description uses the PHOTO_DESCRIPTION_PREFIX constant and the name column from the source to store a description for the image.

For the field_image field, the image_import plugin is used. This time, the subfields are not set manually. Instead, they are assigned using plugin configuration keys. The absence of the id_only configuration allows for this. The URL to the image is set in the source key and uses the psf_image_url pseudofield. The alt key allows you to set the alternative attribute for the image and in this case the psf_image_description pseudofield is used. For the title subfield sets the text of a subfield with the same name and in this case it is assigned the value of the title node property which was set at the beginning of the process pipeline. Remember that not only psedufields are available. Finally, the width and height configuration uses the columns from the source to set the values of the corresponding subfields.

What did you learn in today’s blog post? Did you know you can define constants in your source as data placeholders for use in the process section? Were you aware that pseudofields can be created in the process section to store intermediary data for process definitions that come next? Have you ever wondered what is the migration process pipeline and how it works? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your colleagues.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Eelke Blok: Creating fields and other configuration during updates in Drupal

5 August 2019 - 1:31pm

When deploying changes to a Drupal environment, you should be running database updates (e.g. drush updb, or through update.php) first, and only then import new configuration. The reason for this is that update hooks may want to update configuration, which will fail if that configuration is already structured in the new format (you can't do without updates either; update hooks don't just deal with configuration, but may also need to change the database schema and do associated data migrations). So, there really is no discussion; updates first, config import second. But sometimes, you need...

Categories: Drupal

James Oakley: Drupal 8 Development / Deployment Workflow

5 August 2019 - 4:52am

I've been putting off learning how to build sites in Drupal 8, and migrating my existing Drupal 7 sites over to Drupal 8. Why? Drupal 8 uses a lot of new tools. I want to learn how to set up a Drupal 8 site in the "right" (optimal) way so that I don't incur technical debt for myself later on. That means I have a lot of tools to learn. That takes time, which I don't have a lot of. So I've procrastinated.

Blog Category: Drupal Planet
Categories: Drupal

OSTraining: Create a Simple Faceted Search in Your Drupal 8 Site

5 August 2019 - 2:40am

Creating a faceted search in Drupal implies some configuration steps. This can be overwhelming to people new to Drupal.

The MixItUp Views Drupal 8 module allows you to create a simplified version of a faceted search based on the taxonomies of the content type. It also provides a nice animation, that makes the user experience even better.

Categories: Drupal

Lullabot: Behind the Screens: Behind the Screens with Brendan Blaine

5 August 2019 - 12:00am

I catch up with Brendan Blaine, a developer for the Drupal Association, to find out what it takes to run events.drupal.org, why the conferences run so smoothly, and always remember to use a coaster.

Categories: Drupal

Srijan Technologies: 5 Reasons To Consider Serverless AWS For Drupal

4 August 2019 - 10:43pm

Using cloud is about leveraging its agility among other benefits. For the Drupal-powered website, a right service provider can impact how well the website performs and can affect the business revenue.

Categories: Drupal

Agaric Collective: Migrating data into Drupal subfields

4 August 2019 - 5:30pm

In the previous entry, we learned how to use process plugins to transform data between source and destination. Some Drupal fields have multiple components. For example, formatted text fields store the text to display and the text format to apply. Image fields store a reference to the file, alternative, and title text, width, and height. The migrate API refers to a field’s component as a subfield. Today we will learn how to migrate into them and know which subfields are available.

Getting the example code

Today’s example will consist of migrating data into the `Body` and `Image` fields of the `Article` content type that are available out of the box. This assumes that Drupal was installed using the `standard` installation profile. As in previous examples, we will create a new module and write a migration definition file to perform the migration. The code snippets will be compact to focus on particular elements of the migration. The full code snippet is available at https://github.com/dinarcon/ud_migrations The module name is `UD Migration Subfields` and its machine name is `ud_migrations_subfields`. The `id` of the example migration is `udm_subfields`. Refer to this article for instructions on how to enable the module and run the migration.

source: plugin: embedded_data data_rows: - unique_id: 1 name: 'Michele Metts' profile: 'freescholar on Drupal.org' photo_url: 'https://agaric.coop/sites/default/files/2018-12/micky-cropped.jpg' photo_description: 'Photo of Michele Metts' photo_width: '587' photo_height: '657'

Only one record is presented to keep snippet short, but more exist. In addition to having a unique identifier, each record includes a name, a short profile, and details about the image.

Migrating formatted text

The `Body` field is of type `Text (formatted, long, with summary)`. This type of field has three components: the full text (value) to present, a summary text, and a text format. The Migrate API allows you to write to each component separately defining subfields targets. The next code snippets shows how to do it:

process: field_text_with_summay/value: source_value field_text_with_summay/summary: source_summary field_text_with_summay/format: source_format

The syntax to migrate into subfields is the machine name of the field and the subfield name separated by a slash (/). Then, a colon (:), a space, and the value. You can set the value to a source column name for a verbatim copy or use any combination of process plugins. It is not required to migrate into all subfields. Each field determines what components are required, so it is possible that not all subfields are set. In this example, only the value and text format will be set.

process: body/value: profile body/format: plugin: default_value default_value: restricted_html

The `value` subfield is set to the `profile` source column. As you can see in the first snippet, it contains HTML markup. An `a` tag to be precise. Because we want the tag to be rendered as a link, a text format that allows such tag needs to be specified. There is no information about text formats in the source, but Drupal comes with a couple we can choose from. In this case, we use the `Restricted HTML` text format. Note that the `default_value` plugin is used and set to `restricted_html`. When setting text formats, it is necessary to use its machine name. You can find them in the configuration page for each text format. For `Restricted HTML` that is /admin/config/content/formats/manage/restricted_html.

Note: Text formats are a whole different subject that even has security implications. To keep the discussion on topic, we will only give some recommendations. When you need to migrate HTML markup, you need to know which tags appear in your source, which ones you want to allow in Drupal, and select a text format that accepts what you have whitelisted and filter out any dangerous tags like `script`. As a general rule, you should avoid setting the `format` subfield to use the `Full HTML` text format.

Migrating images

There are different approaches to migrating images. Today, we are going to use the Migrate Files module. It is important to note that Drupal treats images as files with extra properties and behavior. Any approach used to migrate files can be adapted to migrate images.

process: field_image/target_id: plugin: file_import source: photo_url reuse: TRUE id_only: TRUE field_image/alt: photo_description field_image/title: photo_description field_image/width: photo_width field_image/height: photo_height

When migrating any field you have to use their machine in the mapping section. For the `Image` field, the machine name is `field_image`. Knowing that, you set each of its subfields:

  • `target_id` stores an integer number which Drupal uses as a reference to the file.
  • `alt` stores a string that represents the alternative text. Always set one for better accessibility.
  • `title` stores a string that represents the title attribute.
  • `width` stores an integer number which represents the width in pixels.
  • `height` stores an integer number which represents the height in pixels.

For the `target_id`, the plugin `file_import` is used. This plugin requires a `source` configuration value with a URL to the file. In this case, the `photo_url` column from the source section is used. The `reuse` flag indicates that if a file with the same location and name exists, it should be used instead of downloading a new copy. When working on migrations, it is common to run them over and over until you get the expected results. Using the `reuse` flag will avoid creating multiple references or copies of the image file, depending on the plugin configuration. The `id_only` flag is set so that the plugin only returns that file identifier used by Drupal instead of an entity reference array. This is done because each subfield is being set manually. For the rest of the subfields (`alt`, `title`, `width`, and `height`) the value is a verbatim copy from the source.

Note: The Migrate Files module offers another plugin named `image_import`. That one allows you to set all the subfields as part of the plugin configuration. An example of its use will be shown in the next article. This example uses the `file_import` plugin to emphasize the configuration of the image subfields.

Which subfields are available?

Some fields have many subfields. Address fields, for example, have 13 subfields. How can you know which ones are available? The answer is found in the class that provides the field type. Once you find the class, look for the `schema` method. The subfields are contained in the `columns` array of the value returned by the `schema` method. Let’s see some examples:

  • The `Text (plain)` field is provided by the StringItem class.
  • The `Number (integer)` field is provided by the IntegerItem class.
  • The `Text (formatted, long, with summary)` is provided by the TextWithSummaryItem class.
  • The `Image` field is provided by the ImageItem class.

The `schema` method defines the database columns used by the field to store its data. When migrating into subfields, you are actually migrating into those particular database columns. Any restriction set by the database schema needs to be respected. That is why you do not use units when migrating width and height for images. The database only expects an integer number representing the corresponding values in pixels. Because of object-oriented practices, sometimes you need to look at the parent class to know all the subfields that are available.

Another option is to connect to the database and check the table structures. For example, the `Image` field stores its data in the `node__field_image` table. Among others, this table has five columns named after the field’s machine name and the subfield:

  • field_image_target_id
  • field_image_alt
  • field_image_title
  • field_image_width
  • field_image_height

Looking at the source code or the database schema is arguably not straightforward. This information is included for reference to those who want to explore the Migrate API in more detail. You can look for migrations examples to see what subfields are available. I might even provide a list in a future blog post. ;-)

Tip: You can use Drupal Console for code introspection and analysis of database table structure. Also, many plugins are defined by classes that end with the string `Item`. You can use your IDEs search feature to find the class using the name of the field as hint.

Default subfields

Every Drupal field has at least one subfield. For example, `Text (plain)` and `Number (integer)` defines only the `value` subfield. The following code snippets are equivalent:

process: field_string/value: source_value_string field_integer/value: source_value_integer process: field_string: source_value_string field_integer: source_value_integer

In examples from previous days, no subfield has been manually set, but Drupal knows what to do. As we have mentioned, the Migrate API offers syntactic sugar to write shorter migration definition files. This is another example. You can safely skip the default subfield and manually set the others as needed. For `File` and `Image` fields, the default subfield is `target_id`. How does the Migrate API know what subfield is the default? You need to check the code again.

The default subfield is determined by the return value of `mainPropertyName` method of the class providing the field type. Again, object oriented practices might require looking at the parent classes to find this method. In the case of the `Image` field, it is provided by ImageItem which extends FileItem which extends EntityReferenceItem. It is the latter that contains the `mainPropertyName` returning the string `target_id`.

What did you learn in today’s blog post? Were you aware of the concept of subfields? Did you ever wonder what are the possible destination targets (subfields) for each field type? Did you know that the Migrate API finds the default subfield for you? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your colleagues.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Agaric Collective: Using process plugins for data transformation in Drupal migrations

3 August 2019 - 10:09am

In the previous entry, we wrote our first Drupal migration. In that example, we copied verbatim values from the source to the destination. More often than not, the data needs to be transformed in some way or another to match the format expected by the destination or to meet business requirements. Today we will learn more about process plugins and how they work as part of the Drupal migration pipeline.

Syntactic sugar

The Migrate API offers a lot of syntactic sugar to make it easier to write migration definition files. Field mappings in the process section are an example of this. Each of them requires a process plugin to be defined. If none is manually set, then the get plugin is assumed. The following two code snippets are equivalent in functionality.

process: title: creative_title process: title: plugin: get source: creative_title

The get process plugin simply copies a value from the source to the destination without making any changes. Because this is a common operation, get is considered the default. There are many process plugins provided by Drupal core and contributed modules. Their configuration can be generalized as follows:

process: destination_field: plugin: plugin_name config_1: value_1 config_2: value_2 config_3: value_3

The process plugin is configured within an extra level of indentation under the destination field. The plugin key is required and determines which plugin to use. Then, a list of configuration options follows. Refer to the documentation of each plugin to know what options are available. Some configuration options will be required while others will be optional. For example, the concat plugin requires a source, but the delimiter is optional. An example of its use appears later in this entry.

Providing default values

Sometimes, the destination requires a property or field to be set, but that information is not present in the source. Imagine you are migrating nodes. As we have mentioned, it is recommended to write one migration file per content type. If you know in advance that for a particular migration you will always create nodes of type Basic page, then it would be redundant to have a column in the source with the same value for every row. The data might not be needed. Or it might not exist. In any case, the default_value plugin can be used to provide a value when the data is not available in the source.

source: ... process: type: plugin: default_value default_value: page destination: plugin: 'entity:node'

The above example sets the type property for all nodes in this migration to page, which is the machine name of the Basic page content type. Do not confuse the name of the plugin with the name of its configuration property as they happen to be the same: default_value. Also note that because a (content) type is manually set in the process section, the default_bundle key in the destination section is no longer required. You can see the latter being used in the example of writing your Drupal migration blog post.

Concatenating values

Consider the following migration request: you have a source listing people with first and last name in separate columns. Both are capitalized. The two values need to be put together (concatenated) and used as the title of nodes of type Basic page. The character casing needs to be changed so that only the first letter of each word is capitalized. If there is a need to display them in all caps, CSS can be used for presentation. For example: FELIX DELATTRE would be transformed to Felix Delattre.

Tip: Question business requirements when they might produce undesired results. For instance, if you were to implement this feature as requested DAMIEN MCKENNA would be transformed to Damien Mckenna. That is not the correct capitalization for the last name McKenna. If automatic transformation is not possible or feasible for all variations of the source data, take notes and perform manual updates after the initial migration. Evaluate as many use cases as possible and bring them to the client’s attention.

To implement this feature, let’s create a new module ud_migrations_process_intro, create a migrations folder, and write a migration definition file called udm_process_intro.yml inside it. Follow the instructions in this entry to find the proper location and folder structure or download the sample module from https://github.com/dinarcon/ud_migrations It is the one named UD Process Plugins Introduction and machine name udm_process_intro. For this example, we assume a Drupal installation using the standard installation profile which comes with the Basic Page content type. Let’s see how to handle the concatenation of first an last name.

id: udm_process_intro label: 'UD Process Plugins Introduction' source: plugin: embedded_data data_rows: - unique_id: 1 first_name: 'FELIX' last_name: 'DELATTRE' - unique_id: 2 first_name: 'BENJAMIN' last_name: 'MELANÇON' - unique_id: 3 first_name: 'STEFAN' last_name: 'FREUDENBERG' ids: unique_id: type: integer process: type: plugin: default_value default_value: page title: plugin: concat source: - first_name - last_name delimiter: ' ' destination: plugin: 'entity:node'

The concat plugin can be used to glue together an arbitrary number of strings. Its source property contains an array of all the values that you want put together. The delimiter is an optional parameter that defines a string to add between the elements as they are concatenated. If not set, there will be no separation between the elements in the concatenated result. This plugin has an important limitation. You cannot use strings literals as part of what you want to concatenate. For example, joining the string Hello with the value of the first_name column. All the values to concatenate need to be columns in the source or fields already available in the process pipeline. We will talk about the latter in a future blog post.

To execute the above migration, you need to enable the ud_migrations_process_intro module. Assuming you have Migrate Run installed, open a terminal, switch directories to your Drupal docroot, and execute the following command: drush migrate:import udm_process_intro Refer to this entry if the migration fails. If it works, you will see three basic pages whose title contains the names of some of my Drupal mentors. #DrupalThanks

Chaining process plugins

Good progress so far, but the feature has not been fully implemented. You still need to change the capitalization so that only the first letter of each word in the resulting title is uppercase. Thankfully, the Migrate API allows chaining of process plugins. This works similarly to unix pipelines in that the output of one process plugin becomes the input of the next one in the chain. When the last plugin in the chain completes its transformation, the return value is assigned to the destination field. Let’s see this in action:

id: udm_process_intro label: 'UD Process Plugins Introduction' source: ... process: type: ... title: - plugin: concat source: - first_name - last_name delimiter: ' ' - plugin: callback callable: mb_strtolower - plugin: callback callable: ucwords destination: ...

The callback process plugin pass a value to a PHP function and returns its result. The function to call is specified in the callable configuration option. Note that this plugin expects a source option containing a column from the source or value of the process pipeline. That value is sent as the first argument to the function. Because we are using the callback plugin as part of a chain, the source is assumed to be the last output of the previous plugin. Hence, there is no need to define a source. So, we concatenate the columns, make them all lowercase, and then capitalize each word.

Relying on direct PHP function calls should be a last resort. Better alternatives include writing your own process plugins which encapsulates your business logic separate of the migration definition. The callback plugin comes with its own limitation. For example, you cannot pass extra parameters to the callable function. It will receive the specified value as its first argument and nothing else. In the above example, we could combine the calls to mb_strtolower() and ucwords() into a single call to mb_convert_case($source, MB_CASE_TITLE) if passing extra parameters were allowed.

Tip: You should have a good understanding of your source and destination formats. In this example, one of the values to want to transform is MELANÇON. Because of the cedilla (ç) using strtolower() is not adequate in this case since it would leave that character uppercase (melanÇon). Multibyte string functions (mb_*) are required for proper transformation. ucwords() is not one of them and would present similar issues if the first letter of the words are special characters. Attention should be given to the character encoding of the tables in your destination database.

Technical note: mb_strtolower is a function provided by the mbstring PHP extension. It does not come enabled by default or you might not have it installed altogether. In those cases, the function would not be available when Drupal tries to call it. The following error is produced when trying to call a function that is not available: The "callable" must be a valid function or method. For Drupal and this particular function that error would never be triggered, even if the extension is missing. That is because Drupal core depends on some Symfony packages which in turn depend on the symfony/polyfill-mbstring package. The latter provides a polyfill) for mb_* functions that has been leveraged since version 8.6.x of Drupal.

What did you learn in today’s blog post? Did you know that syntactic sugar allows you to write shorter plugin definitions? Were you aware of process plugin chaining to perform multiple transformations over the same data? Had you considered character encoding on the source and destination when planning your migrations? Are you making your best effort to avoid the callback process plugin? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your colleagues.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Centarro: Decoupled Days 2019

2 August 2019 - 5:01pm

Decoupled Days 2019 last month, the third edition of the conference, was fantastic. I had the privilege to speak and attend last year, as well. The conference has quickly risen to be one of my favorite conference of the year.

Not familiar with Decoupled Days? Spawned in the “Hallway Track” of DrupalCon between its founders, the conference originated as Decoupled Drupal Days in 2017. Last year saw the phasing out of the word “Drupal” as the conference became focused on decoupling in general, not just Drupal. That is one reason it has quickly become a favorite event. It is an engineering and design conference. The act of decoupling in a system requires specific system design and presents engineering challenges. The organizers identify it as:

The only conference on the future of CMS, headless CMS, and decoupled CMS.

Categories: Drupal

Hook 42: Meetings Recap - July 29th -31st, 2019

2 August 2019 - 11:27am
Meetings Recap - July 29th -31st, 2019 Will Thurston-… Fri, 08/02/2019 - 18:27
Categories: Drupal

Agaric Collective: Writing your first Drupal migration

2 August 2019 - 10:16am

In the previous entry, we learned that the Migrate API is an implementation of an ETL framework. We also talked about the steps involved in writing and running migrations. Now, let’s write our first Drupal migration. We are going to start with a very basic example: creating nodes out of hardcoded data. For this, we assume a Drupal installation using the standard installation profile which comes with the Basic Page content type.

As we progress through the series, the migrations will become more complete and more complex. Ideally, only one concept will be introduced at a time. When that is not possible, we will explain how different parts work together. The focus of today's lesson is learning the structure of a migration definition file and how to run it.

Writing the migration definition file

The migration definition file needs to live in a module. So, let’s create a custom one named ud_migrations_first and set Drupal core’s migrate module as dependencies in the *.info.yml file.

Now, let’s create a folder called migrations and inside it a file called udm_first.yml. Note that the extension is yml, not yaml. The content of the file will be:

type: module name: UD First Migration description: 'Example of basic Drupal migration. Learn more at https://understanddrupal.com/migrations.' package: Understand Drupal core: 8.x dependencies: - drupal:migrate

The final folder structure will look like:

id: udm_first label: 'UD First migration' source: plugin: embedded_data data_rows: - unique_id: 1 creative_title: 'The versatility of Drupal fields' engaging_content: 'Fields are Drupal''s atomic data storage mechanism...' - unique_id: 2 creative_title: 'What is a view in Drupal? How do they work?' engaging_content: 'In Drupal, a view is a listing of information. It can a list of nodes, users, comments, taxonomy terms, files, etc...' ids: unique_id: type: integer process: title: creative_title body: engaging_content destination: plugin: 'entity:node' default_bundle: page

YAML is a key-value format with optional nesting of elements. They are very sensitive to white spaces and indentation. For example, they require at least one space character after the colon symbol (:) that separates the key from the value. Also, note that each level in the hierarchy is indented by two spaces exactly. A common source of errors when writing migrations is improper spacing or indentation of the YAML files.

A quick glimpse at the file reveals the three major parts: source, process, and destination. Other keys provide extra information about the migration. There are more keys that the ones shown above. For example, it is possible to define dependencies among migrations. Another option is to tag migrations so they can be executed together. We are going to learn more about these options in future entries.

Let’s review each key-value pair in the file. For id, it is customary to set its value to match the filename containing the migration definition, but without the .yml extension. This key serves as an internal identifier that Drupal and the Migrate API use to execute and keep track of the migration. The id value should be alphanumeric characters, optionally using underscores to separate words. As for the label key, it is a human readable string used to name the migration in various interfaces.

In this example we are using the embedded_data source plugin. It allows you to define the data to migrate right inside the definition file. To configure it, you define a data_rows key whose value is an array of all the elements you want to migrate. Each element might contain an arbitrary number of key-value pairs representing “columns” of data to be imported.

A common use case for the embedded_data plugin is testing of the Migrate API itself. Another valid one is to create default content when the data is known in advance. I often present introduction to Drupal workshops. To save time, I use this plugin to create nodes which are later used in the views creation explanation. Check this repository for an example of this. Note that it uses a different directory structure to define the migrations. That will be explained in future blog posts.

For the destination we are using the entity:node plugin which allows you to create nodes of any content type. The default_bundle key indicates that all nodes to be created will be of type “Basic page”, by default. It is important to note that the value of the default_bundle key is the machine name of the content type. You can find it at /admin/structure/types/manage/page In general, the Migrate API uses machine names for the values. As we explore the system, we will point out when they are used and where to find the right ones.

In the process section you map columns from the source to node properties and fields. The keys are entity property names or the field machine names. In this case, we are setting values for the title of the node and its body field. You can find the field machine names in the content type configuration page: /admin/structure/types/manage/page/fields. Values can be copied directly from the source or transformed via process plugins. This example makes a verbatim copy of the values from the source to the destination. The column names in the source are not required to match the destination property or field name. In this example they are purposely different to make them easier to identify.

You can download the example code from https://github.com/dinarcon/ud_migrations The example above is actually in a submodule in that repository. The same repository will be used for many examples throughout series. Download the whole repository into the ./modules/custom directory of the Drupal installation and enable the “UD First Migration” module.

Running the migration

Let’s use Drush to run the migrations with the commands provided by Migrate Run. Open a terminal, switch directories to Drupal’s webroot, and execute the following commands.

$ drush pm:enable -y migrate migrate_run ud_migrations_first $ drush migrate:status $ drush migrate:import udm_first

The first command enables the core migrate module, the runner, and the custom module holding the migration definition file. The second command shows a list of all migrations available in the system. Only one should be listed with the migration ID udm_first. The third command executes the migration. If all goes well, you can visit the content overview page at /admin/content and see two basic pages created. Congratulations, you have successfully run your first Drupal migration!!!

Or maybe not? Drupal migrations can fail in many ways and sometimes the error messages are not very descriptive. In upcoming blog posts we will talk about recommended workflows and strategies for debugging migrations. For now, let’s mention a couple of things that could go wrong with this example. If after running the drush migrate:status command you do not see the udm_first migration, make sure that the ud_migrations_first module is enabled. If it is enabled, and you do not see it, rebuild the cache by running drush cache:rebuild.

If you see the migration, but you get a yaml parse error when running the migrate:import command check your indentation. Copying and pasting from GitHub to your IDE/editor might change the spacing. An extraneous space can break the whole migration so pay close attention. If the command reports that it created the nodes, but you get a fatal error when trying to view one, it is because the content type was not set properly. Remember that the machine name of the “Basic page” content type is page, not basic_page. This error cannot be fixed from the administration interface. What you have to do is rollback the migration issuing the following command: drush migrate:rollback udm_first, then fix the default_bundle value, rebuild the cache, and import again.

Note: Migrate Tools could be used for running the migration. This module depends on Migrate Plus. For now, let’s keep module dependencies to a minimum to focus on core Migrate functionality. Also, skipping them demonstrates that these modules, although quite useful, are not hard requirements for running migration projects. If you decide to use Migrate Tools make sure to uninstall Migrate Run. Both provide the same Drush commands and conflict with each other if the two are enabled.

What did you learn in today’s blog post? Did you know that Migrate Plus and Migrate Tools are not hard requirements for Drupal migrations projects? Did you know you can place your YAML files in a migrations directory? What advice would you give to someone writing their first migration? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your friends and colleagues.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Electric Citizen: Mastering Drupal 8 Multilingual: Part 1 of 3

2 August 2019 - 9:08am

The web is constantly growing, evolving and—thankfully—growing more accessible and inclusive.

It is becoming expected that a user can interact with a website solely via keyboard or have the option to browse in their native language. There are many ways to serve the needs of non-native-language users, but one of the more robust is Drupal Multilingual.

Unlike 3rd party translation plugins like Google Translate or browser translation tools, Drupal's suite of core Multilingual tools allows you to write accurate and accessible translated content in the same manner as you write in your default language content. With no limit on the number languages, settings for right-to-left content, and the ability to translate any and all of your content, Drupal 8 can create a true multi-language experience like never before.

There is, however, a bit of planning and work involved.

Hopefully, this blog series will help smooth the path to truly inclusive content by highlighting some project management, design, site building, and development gotchas, as well as providing some tips and tricks to make the multilingual experience better for everyone. Part one will help you decide if you need multilingual as well as provide some tips on how to plan and budget for it.

Categories: Drupal

Agiledrop.com Blog: Recap of Acquia's webinar on the Digital Experience Platform

2 August 2019 - 12:27am

We wrote up a recap of Acquia's webinar on digital experience platforms held on July 31st. Check it out to get a head start on the future digital experience of your brand or company.

READ MORE
Categories: Drupal

Tandem's Drupal Blog: Drupal 8 Search API Solr + Lando + Platform.sh

1 August 2019 - 5:00pm
August 02, 2019 Setting up Solr is quick and fairly painless with Lando. This guide shall bring you to the promise land. Why Use Solr? We have a large enterprise client that has a tens of thousands of nodes. We originally setup the site to use Search API via the database. This worked but it was painfully slow on most searches. I knew that we ...
Categories: Drupal

Palantir: Leading Children’s Hospital in the Southwest

1 August 2019 - 4:54pm
Simplifying the complexities of access to family healthcare On

Our client is one of the largest children’s hospitals in the country. With a medical staff of nearly 1,000 pediatric specialists, this hospital provides world-class inpatient, outpatient, trauma, emergency and urgent care allowing for the most comprehensive pediatric services available in their state. This healthcare delivery system and network provides health services across more than 75 pediatric specialties and is poised for continued growth in quality patient care, research and medical education.

The hospital’s former website was extremely information-dense with thousands of pages of mostly text-based content and complex and somewhat non-intuitive navigation. Important information was buried deep in the site. The search functionality needed to be improved to sort, filter, and categorize critical results. Additionally, there were many practice sites and microsites that had been developed by different hospital departments over time that needed to be brought back under a unified architecture.

The Vision

The vision for this redesign project was clear: to simplify the complex pediatric medical care process by providing families and people in need with clear and concise information about the hospital’s health care services.

Palantir planned our approach based on the following goals:

  • Increase engagement via an improved patient experience.
  • Reinforce the hospital’s credibility and their national reputation for excellence in specialty care.
  • Provide trustworthy health information within a convenient environment.
  • Provide educational materials for anyone needing to learn about a disease or diagnosis.
  • Improve content management efficiency.
Frictionless Access to Healthcare

A key objective of this site is to help visitors easily get access to care for their families. All types of content on the site should have strong calls to action for a visitor to request an appointment or make other decisions that could lead to them getting the care they or their family need. Less friction in the process allows for a larger number of people to choose the hospital for treatment.

Our client was able to optimize the content on their site and take the site from 10,000 pages down to 900 pages. Paired with their new increased search functionality, site visitors in need of finding critical healthcare information can now do so more quickly and with greater ease.

The redesign has led to increased conversion for:
  • Online Appointment Form Submission Requests
  • Patient Referrals
  • The Find A Doctor Journey
  • Usage of the Patient Portal
  • Subscriptions to Newsletters
The Palantir Approach

All of the overall goals for our client’s redesign tied back to one thing: optimizing their content and navigation. This project was broken out into two phases to address content strategy and modeling first that could then set the path for the design and build phase.

Phase 1: Strategic Focus, Content Audit, and Content Modeling

An important part of the strategy phase was to complete a massive site audit in partnership with another agency who helped out with copywriting. The old site had over 10,000 pages and many of those pages had low engagement. The overall plan was to analyze our client’s existing content and remove any content that site users weren’t looking for or finding relevant.

The first phase took about 10 weeks and resulted in a content model for key page types. Each page type included an evaluation of the content on the site as it was, as well as strategic recommendations for how the page type would be used in the future. Four of these page types were then wireframed. By building out content types directly in their content management system, Drupal (as opposed to organizing content in a spreadsheet), the client team was able to see how their content would be displayed.

Phase 2: UX, Design, and Build

As soon as the first phase was complete, the second phase began with solidifying the content model, creating and implementing a design system, as well as building and launching the site.

Our goal was to design a solution that showcases the cutting-edge research and innovation at the hospital. We did this with the use of modern design treatments that takes a visual spin on words with dynamic shapes, creating a captivating first impression, and setting them apart from their competition.

The look and feel captures the vibrant and playful spirit of the hospital with an energetic and uplifting color palette that creates a sense of hope for patients and families and elevates our client’s online brand presence. The use of authentic, non-stock imagery communicates a warm and caring environment that is paramount to their mission of providing the best possible care for patients and their loved ones.

The design components are simple and clean but unique in composition and layout. Though there are bold colors and images, the information is communicated in a clear and concise way which allows users to take action along their intended journey. The use of subtle interactions highlights content that educates and instructs, driving more patient engagement.

Find a Doctor

Researching and finding the right doctor to see or treat your child is a crucial step in ensuring good care for your child. Our client wanted to make its database of physicians available for the public to search and aggregate physician information with ratings and other key pieces of information to help empower patients and parents.

Palantir built a framework for merging in complex data from multiple sources in Drupal to provide an intuitive, sleek search interface that allows users to sift through huge quantities of data to find exactly what they’re looking for. By pulling key pieces of information in from externally-managed systems, we were able to lower the workload on the marketing team necessary to keep the physician directory up to date and accurate as physicians, credentials, insurance, and location information change over time.

Search Strategy

Palantir defined personas for the various site audiences. The site needed to be able to surface relevant content for teens, parents, physicians, and people in crisis situations. We tested wireframes in-depth and performed chalkmark tests around the menu structure, both of which helped make sure pathways were simple and straightforward for all audiences.

For general site search, we implemented Solr-based Acquia Search, which provided more advanced capabilities than the standard Drupal search functionality. Palantir added recommended results, so if there is something our client wants to bump to the top of search results based on a specific keyword, they now have that ability. For example, if a user were to search for the term “cancer,” our client can now make sure that results for the oncology department get bumped to the top of the results list.

Connection and Conversion

Our client is passionate about connecting its collaborative network of providers to patients and parents who are just as eager to find the right kind of care. Through this network, they are able to achieve their mission. Their new digital presence will allow them to iterate and expand on the ways they are able to meet the needs of their community for years to come.

Major Outcomes
  • An increase in awareness and conversion through “Find a Provider” flows.
  • Users can make effective choices as they weigh convenience, cost, and quality by displaying available appointment times, making pricing transparent, and using geolocation to surface nearby facilities.
  • New or returning patients can easily request an appointment by surfacing contextual calls to action and enabling system-wide scheduling.
  • Simple forms eliminate existing pain points, minimizing the amount of information initially required as input.
  • Robust provider profiles inspire confidence and trust. When available, we included personal statements, videos, ratings, and testimonials that appeal to customers’ emotions and values.
Categories: Drupal

Ramsalt Lab: Annoy only EU citizens with your GDPR cookie banner

1 August 2019 - 12:49pm

Written by Sven Berg Ryen, Leader of the GDPR audit team at Ramsalt Lab

EU Cookie Compliance, one of the top 100 Drupal modules, is a Drupal module that offers a cookie consent banner with various features, making it more convenient for your site to become GDPR compliant. GDPR is the new data privacy regulation that came into effect on 25 May 2018 and it sets out to bolster the rights citizens of the EU have over their data which is held by companies. Ramsalt Lab is currently supporting the module development as part of our GDPR audit services.

According to GDPR, if you have any traffic from EU citizens on your site, you need to ask for consent before you, or third party scripts, process any of their personal data.

This is all very well, you can ask for consent first, and then only use the visitor’s private data if they consent, but under GDPR you’re required to do so only when the visitor is an EU or EEC citizen. That still leaves billions residents outside of the EU where privacy laws may not require consent (one could argue whether this is good or bad) for storing cookies that identify individuals. Wouldn’t it be nice if you can comply with EU regulations and at the same time not pester those outside of the area where GDPR is enforced?

Luckily, EU Cookie Compliance has a feature to the rescue. It can first check whether the user resides in the countries that GDPR affects, and then display the banner accordingly, only when applicable.

So the technical parts

To achieve this, you need an additional addon; either the Smart IP or geoIP modules - or the geoIP PHP library. It may be easiest to use the module route, since adding the PHP library may not be feasible on your hosted server or cloud solution.

We will here use Smart IP, since that’s the only module that the Drupal 8 version of EU Cookie Compliance supports. There is now also a beta version of GeoIP available for Drupal 8, so at some point, EU Cookie Compliance may support GeoIP also in the 8.x module version. You can follow this issue for the progress.

The option to show the banner only to EU countries can be found near the bottom of the module settings page. A notification can be seen when the Smart IP module is not enabled.

Enabling and setting up the Smart IP module

Install and enable the Smart IP module, using your preferred technique (such as composer/drush or direct download from drupal.org). In Drupal 8, you also have to enable a Smart IP data source module.

The Drupal 8 module gives you to the following geolocation lookup options:

  • Free and licensed geolocation files from ip2location.com (signup required to get access to the free version). For the purposes of this module, you only need the DB1 database, with coverage of countries. Attribution is required when you use the free database.
  • Geolocation service from ipinfodb. A free API is available. You are however limited to 2 requests per second, and will be blacklisted if you exceed that limit. Also, the service limits you to lookup requests from one server IP only, which may not be ideal if you’re planning to test the service from your localhost. Note that the module utilizes the ip-city endpoint, and not the faster ip-country one. Sign up to get an API key.
  • MaxMind GeoIP2. A free database is available, updated on the first Tuesday of each month. No signup is required to use the free version, though attribution is required.
  • MaxMind GeoIP2 Precision API service offering lookup at the country level at $0.0001 per request. A free trial is available.

Some fallback options are available, and will be accessible if the headers exist in the web page query when you open the configuration page (which means they may not be available on your localhost, but could be available on your server).

  • Cloudflare headers
  • The mod_geoip module in Apache
  • Nginx headers

The Drupal 7 version of Smart IP offers all of the above and in addition some legacy lookup services.

We will be using the Smart IP MaxMind GeoIP2 binary database, because it has a free version of the database that will automatically be updated once a month on cron run. In other words, you need to enable the smart_ip_maxmind_geoip2_bin_db submodule (part of smart_ip).

Configuring Smart IP for GDPR

After having enabled the required modules, head over to /admin/config/people/smart_ip.

Select the “Use MaxMind GeoIP2 binary database” option to see the configuration for the service. Choose the Lite database version, the Country level edition and make sure that Yes is chosen under Automatic updates.

Further down, in the second pane, configure your settings to allow geolocation lookup for all desired user roles. Then, since I guess you care about privacy, either opt to not save the user’s geolocation on account creation, or enable the feature to prevent storing location details from GDPR countries.

Scroll all the way to the bottom and press “Save configuration”. If you get an error at this point, you need to set a private file path in settings.php.

After having configured Smart IP, you need to head over to MaxMind’s website and download the GeoLite2 Country file in DB format. Then, expand the archive, grab just the file labeled “GeoLite2-Country.mmdb” and drop it into “[PATH_TO_PRIVATE_FOLDER]/smart_ip”. After you add this file manually once, the Smart IP module will take care of the automatic monthly updates.

Note: In Drupal 7, the GeoLite 2 country database is downloaded automatically when configuring the module, so there’s no need for a manual download.

Configuring EU Cookie Compliance

Next, head back to the settings for EU Cookie Compliance at admin/config/system/eu-cookie-compliance and enable the “Only display banner in EU countries” option. If your site uses any caching at all, you’ll want to enable the Javascript based option.

After enabling this feature, you will need to rebuild Drupal cache, in order for Drupal to pick up the new path that is used to determine if the user is in the EU.

Testing

Note: If you’re on an EU Cookie Compliance version prior to 8.x-1.7, you need the patch from this EU Cookie Compliance issue in order for the debug feature in Smart IP to work. The Drupal 7 version of EU Cookie Compliance doesn’t have this problem (though you should always make sure that your version is up-to-date to get the latest bug fixes and features).

This feature involves a few moving parts, so to ensure everything has been set up correctly, there’s a handy debug feature in Smart IP that can be used. This way, you can check that you are indeed displaying the banner only to European countries where GDPR legislation apply. The easiest way to check if the settings are correct is to temporarily set up debugging in Smart IP for the Anonymous user and open an Incognito window. This way you can ensure that no existing cookies are giving false assurance that the feature is working.

Try using a value such as 151.101.2.217 (which at the time of this article is one of the IPs for the drupal.org server, situated in the US). Notice that no banner is shown when you debug smart IP with this value.

Try 185.91.65.150 (the IP for the server where drupalnorge.no is hosted, which is in Norway) and the banner should appear.

After testing is completed, remember to disable debugging for the anonymous user by clearing the value on the Smart IP configuration page.

Conclusion

A little work is required to set up EU Cookie Compliance to display the GDPR cookie banner only to countries and territories where the law requires one. Resulting from this, you will hopefully have happier users.

If you need help setting up your GDPR cookie banner, or have questions about how your site can become GDPR compliant, you can always get in touch with us at Ramsalt Lab through our contact page.

Written by Sven Berg Ryen
Developer and Leader of the GDPR audit team at Ramsalt Lab


Categories: Drupal

Dries Buytaert: Acquia a leader in 2019 Gartner Magic Quadrant for Web Content Management

1 August 2019 - 10:36am

For the sixth year in a row, Acquia has been recognized as a leader in the Gartner Magic Quadrant for Web Content Management. Acquia first entered the Web Content Management Magic Quadrant back in 2012 as a Visionary, and since then we've moved further than any other vendor to cement our leadership position.

As I've written before, analyst reports like the Gartner Magic Quadrant are important because they introduce organizations to Acquia and Drupal. As I've put if before If you want to find a good coffee place, you use Yelp. If you want to find a nice hotel in New York, you use TripAdvisor. Similarly, if a CIO or CMO wants to spend $250,000 or more on enterprise software, they often consult an analyst firm like Gartner..

In 2012, Gartner didn't fully understand the benefits of Acquia being the only WCM company who embraced both Open Source and cloud. Just seven years later, our unique approach has forever changed web content management. This year, Acquia moved up again in both of the dimensions that Gartner uses to rank vendors: Completeness of Vision and Ability to Execute. You'll see in the Magic Quadrant graphic that Acquia has tied Sitecore for the first time:

Acquia recognized as a leader, next to Adobe, Sitecore and Episerver, in the 2019 Gartner Magic Quadrant for Web Content Management.

I believe we would have placed even higher had our Mautic acquisition completed a bit earlier.

In mature markets like Web Content Management, there is almost always a single proprietary leader and a single Open Source leader. There is Oracle and MongoDB. Splunk and Elastic. VMWare and Docker. Gitlab and Github. That is why I believe that next year it will be Acquia and Adobe at the very top of the WCM Magic Quadrant. Sitecore and Episerver will continue to fight for third place among companies who prefer a Microsoft-centric approach. I was not surprised to see Sitecore move down this year as they work to overcome technical product debt and cloud transition, leading to strange decisions like acquiring a services company.

You can read the complete report on Acquia.com. Thank you to everyone who contributed to this result!

Categories: Drupal

Drupal Association blog: It's Time To Vote - Community Elections 2019

1 August 2019 - 10:09am

Voting is now open for the 2019 At-Large Board positions for the Drupal Association! If you haven't yet, check out the candidate’s profiles. Get to know your candidates, and then go vote.

Cast Your Vote!

Voting is open to all individuals who had a Drupal.org account by the time nominations opened and who have logged in at least once in the past year. You do not have to have a Drupal Association membership.

To vote, you will rank candidates in order of your preference (1st, 2nd, 3rd, etc.). The results will be calculated using an "instant runoff" method. For an accessible explanation of how instant runoff vote tabulation works, see videos linked in this discussion.

Election voting is from 1 August, 2019 through 16 August, 2019. During this period, you can continue to review and comment on the candidate profiles.

Have questions? Please contact me: Rachel Lawson.

Categories: Drupal

Agaric Collective: Drupal migrations: Understanding the ETL process

1 August 2019 - 7:12am

The Migrate API is a very flexible and powerful system that allows you to collect data from different locations and store them in Drupal. It is in fact a full-blown extract, transform, and load (ETL) framework. For instance, it could produce CSV files. Its primarily use, thought, is to create Drupal content entities: nodes, users, files, comments, etc. The API is thoroughly documented and their maintainers are very active in the #migration slack channel for those needing assistance. The use cases for the Migrate API are numerous and vary greatly. Today we are starting a blog post series that will cover different migrate concepts so that you can apply them to your particular project.

Understanding the ETL process

Extract, transform, and load (ETL) is a procedure where data is collected from multiple sources, processed according to business needs, and its result stored for later use. This paradigm is not specific to Drupal. Books and frameworks abound on the topic. Let’s try to understand the general idea by following a real life analogy: baking bread. To make some bread you need to obtain various ingredients: wheat flour, salt, yeast, etc. (extracting). Then, you need to combine them in a process that involves mixing and baking (transforming). Finally, when the bread is ready you put it into shelves for display in the bakery (loading). In Drupal, each step is performed by a Migrate plugin:

The extract step is provided by source plugins.
The transform step is provided by process plugins.
The load step is provided by destination plugins.

As it is the case with other systems, Drupal core offers some base functionality which can be extended by contributed modules or custom code. Out of the box, Drupal can connect to SQL databases including previous versions of Drupal. There are contributed modules to read from CSV files, XML documents, JSON and SOAP feeds, WordPress sites, LibreOffice Calc and Microsoft Office Excel files, Google Sheets, and much more.

The list of core process plugins is impressive. You can concatenate strings, explode or implode arrays, format dates, encode URLs, look up already migrated data, among other transform operations. Migrate Plus offers more process plugins for DOM manipulation, string replacement, transliteration, etc.

Drupal core provides destination plugins for content and configuration entities. Most of the time targets are content entities like nodes, users, taxonomy terms, comments, files, etc. It is also possible to import configuration entities like field and content type definitions. This is often used when upgrading sites from Drupal 6 or 7 to Drupal 8. Via a combination of source, process, and destination plugins it is possible to write Commerce Product Variations, Paragraphs, and more.

Technical note: The Migrate API defines another plugin type: `id_map`. They are used to map source IDs to destination IDs. This allows the system to keep track of records that have been imported and roll them back if needed.

Drupal migrations: a two step process

Performing a Drupal migration is a two step process: writing the migration definitions and executing them. Migration definitions are written in YAML format. These files contain information about the how to fetch data from the source, how to process the data, and how to store it in the destination. It is important to note that each migration file can only specify one source and one destination. That is, you cannot read form a CSV file and a JSON feed using the same migration definition file. Similarly, you cannot write to nodes and users from the same file. However, you can use as many process plugins as needed to convert your data from the format defined in the source to the format expected in the destination.

A typical migration project consists of several migration definition files. Although not required, it is recommended to write one migration file per entity bundle. If you are migrating nodes, that means writing one migration file per content type. The reason is that different content types will have different field configurations. It is easier to write and manage migrations when the destination is homogeneous. In this case, a single content type will have the same fields for all the elements to process in a particular migration.Once all the migration definitions have been written, you need to execute the migrations. The most common way to do this is using the Migrate Tools module which provide Drush commands and a user interface (UI) to run migrations. Note that the UI for running migrations only detect those that have been defined as configuration entities using the Migrate Plus module. This is a topic we will cover in the future. For now, we are going to stick to Drupal core’s mechanisms of defining migrations. Contributed modules like Migrate Scheduler, Migrate Manifest, and Migrate Run offer alternatives for executing migrations.

 

This blog post series is made possible thanks to these generous sponsors. Contact Understand DRupal if your organization would like to support this documentation project, whether the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Pages