Planet Drupal

Subscribe to Planet Drupal feed
Drupal.org - aggregated feeds in category Planet Drupal
Updated: 21 hours 34 min ago

ADCI Solutions: How much does it cost to build a website with Drupal in 2019?

26 August 2019 - 12:29am

Almost every day we are asked the same question: “How much does it cost to build a website with Drupal?”. We understand that customers are tired of hearing that everything depends only on the complexity of the development, so we decided to answer this question in more detail. Of course, we didn't take into account the specific wishes of customers, but in this article, your client will find the information about factors that influence website development cost, where it is better to look for development teams and how long it takes to develop a website. 

Read this article and share it with your audience to make their knowledge of this issue more complete.

 

">

Categories: Drupal

aleksip.net: Component-based theming with Layout Builder

25 August 2019 - 11:31am
Component-based theming with Layout Builder I have previously written about using Drupal’s definition files in component-based theming and how it is possible to have component-specific layout and asset library definition files. Now that Layout Builder is stable it is time to have a look at how it can be used with theme components. aleksip Sun, 08/25/2019 - 21:31
Categories: Drupal

Code Karate: Drupal 8 Shield Module

25 August 2019 - 10:08am
Episode Number: 222

The Drupal 8 Shield module allows you to protect your site using a simple htaccess authentication. It’s great for sites that you are working on that you don’t want the world (or Google) to see yet. This way you can send the site to a client or anyone really to test and just provide them the username/password to view the site. Once it’s ready to go, you can launch the site and remove this module.

If you have ever wanted to password protect your Drupal site, the Shield module will help with that!

Tags: DrupalDrupal 8Site BuildingDrupal Planet
Categories: Drupal

DrupalEasy: DrupalEasy Podcast 221 - Greg Anderson - Composer Support in Drupal 8 Core Initiative

24 August 2019 - 9:31am

Direct .mp3 file download.

Greg Anderson, Open Source Contributions Engineer at Pantheon joins Mike Anello to talk about the Drupal Community's [Composer Support in Core Initiative][(https://www.drupal.org/about/strategic-initiatives/composer).

Discussion DrupalEasy News Upcoming Events Sponsors
  • Drupal Aid - Drupal support and maintenance services. Get unlimited support, monthly maintenance, and unlimited small jobs starting at $99/mo.
  • WebEnabled.com - devPanel.
Follow us on Twitter Subscribe

Subscribe to our podcast on iTunes, Google Play or Miro. Listen to our podcast on Stitcher.

If you'd like to leave us a voicemail, call 321-396-2340. Please keep in mind that we might play your voicemail during one of our future podcasts. Feel free to call in with suggestions, rants, questions, or corrections. If you'd rather just send us an email, please use our contact page.

Categories: Drupal

Agaric Collective: Workflows and benefits of managing Drupal migrations as configuration entities

23 August 2019 - 8:00pm

In the last blog post we were introduced to managing migration as configuration entities using Migrate Plus. Today, we will present some benefits and potential drawbacks of this approach. We will also show a recommended workflow for working with migration as configuration. Let’s get started.

What is the benefit of managing migration as configurations?

At first sight, there does not seem to be a big difference between defining migrations as code or configuration. You can certainly do a lot without using Migrate Plus’ configuration entities. The series so far contains many examples of managing migrations as code. So, what are the benefits of adopting s configuration entities?

The configuration management system is one of the major features that was introduced in Drupal 8. It provides the ability to export all your site’s configuration to files. These files can be added to version control and deployed to different environments. The system has evolved a lot in the last few years, and many workflows and best practices have been established to manage configuration. On top of Drupal core’s incremental improvements, a big ecosystem has sprung in terms of contributed modules. When you manage migrations via configuration, you can leverage those tools and workflows.

Here are a few use cases of what is possible:

  • When migrations are managed in code, you need file system access to make any changes. Using configuration entities allows site administrators to customize or change the migration via the user interface. This is not about rewriting all the migrations. That should happen during development and never on production environments. But it is possible to tweak certain options. For example, administrators could change the location to the file that is going to be migrated, be it a local file or on a remote server.
  • When writing migrations, it is very likely that you will work on a subset of the data that will eventually be used to get content into the production environment.  Having migrations as configuration allow you to override part of the migration definition per environment. You could use the Configuration Split module to configure different source files or paths per environment. For example, you could link to a small sample of the data in development, a larger sample in staging, and the complete dataset in production.
  • It would be possible to provide extra configuration options via the user interface. In the article about adding HTTP authentication to fetch remote JSON and XML files, the credentials were hardcoded in the migration definition file. That is less than ideal and exposes sensitive information. An alternative would be to provide a configuration form in the administration interface for the credentials to be added. Then, the submitted values could be injected into the configuration for the migration. Again, you could make use of contrib modules like Configuration Split to make sure those credentials are never exported with the rest of your site’s configuration.
  • You could provide a user interface to upload migration source files. In fact, the Migrate source UI module does exactly this. It exposes an administration interface where you have a file field to upload a CSV file. In the same interface, you get a list of supported migrations in the system. This allows a site administrator to manually upload a file to run the migration against. Note: The module is supposed to work with JSON and XML migrations. It did not work during my tests. I opened this issue to follow up on this.

These are some examples, but many more possibilities are available. The point is that you have the whole configuration management ecosystem at your disposal. Do you have another example? Please share it in the comments.

Are there any drawbacks?

Managing configuration as configuration adds an extra layer of abstraction in the migration process. This adds a bit of complexity. For example:

  • Now you have to keep the uuid and id keys in sync. This might not seem like a big issue, but it is something to pay attention to.
  • When you work with migrations groups (explained in the next article), your migration definition could live in more file.
  • The configuration management system has its own restrictions and workflows that you need to follow, particularly for updates.
  • You need to be extra careful with your YAML syntax, especially if syncing configuration via the user interface. It is possible to import invalid configuration without getting an error. It is until the migration fails that you realize something is wrong.

Using configuration entities to define migrations certainly offers lots of benefits. But it requires being extra careful managing them.

Workflow for managing migrations as configuration entities

The configuration synchronization system has specific workflows to make changes to configuration entities. This imposes some restrictions in the way you make updates to the migration definitions. Explaining how to manage configuration could use another 31 days blog post series. ;-) For now, only a general overview will be presented. The general approach is similar to managing configuration as code. The main difference is what needs to be done for changes to the migration files to take effect.

You could use the “Configuration synchronization” administration interface at /admin/config/development/configuration. In it you have the option to export  or import a “full archive” containing all your site’s settings or a “single item” like a specific migration. This is one way to manage migrations as configuration entities which lets you find their UUIDs if not set initially. This approach can be followed by site administrators without requiring file system access. Nevertheless, it is less than ideal and error-prone. This is not the recommended way to manage migration configuration entities.

Another option is to use Drush or Drupal Console to synchronize your site’s configuration via the command line. Similarly to the user interface approach, you can export and import your full site configuration or only single elements. The recommendation is to do partial configuration imports so that only the migrations you are actively working on are updated.

Ideally, your site’s architecture is completed before the migration starts. In practice, you often work on the migration while other parts of the sites are being built. If you were to export and import the entire site’s configuration as you work on the migrations, you might inadvertently override unrelated pieces of configurations. For instance, this can lead to missing content types, changed field settings, and lots of frustration. That is why doing partial or single configuration imports is recommended. The following code snippet shows a basic Drupal workflow for managing migrations as configuration:

# 1) Run the migration. $ drush migrate:import udm_config_json_source_node_local # 2) Rollback migration because the expected results were not obtained. $ drush migrate:rollback udm_config_json_source_node_local # 3) Change the migration definition file in the "config/install" directory. # 4a) Sync configuration by folder using Drush. $ drush config:import --partial --source="modules/custom/ud_migrations/ud_migrations_config_json_source/config/install" # 4b) Sync configuration by file using Drupal Console. $ drupal config:import:single --file="modules/custom/ud_migrations/ud_migrations_config_json_source/config/install/migrate_plus.migration.udm_config_json_source_node_local.yml" # 5) Run the migration again. $ drush migrate:import udm_config_json_source_node_local

Note the use of the --partial and --source flags in the migration import command. Also, note that the path is relative to the current working directory from where the command is being issued. In this snippet, the value of the source flag is the directory holding your migrations. Be mindful if there are other non-migration related configurations in the same folder. If you need to be more granular, Drupal Console offers a command to import individual configuration files as shown in the previous snippet.

Note: Uninstalling and installing the module again will also apply any changes to your configuration. This might produce errors if the migration configuration entities are not removed automatically when the module is uninstalled. Read this article for details on how to do that.

What did you learn in today’s blog post? Did you know the know the benefits and trade-offs of managing migrations as configuration? Did you know what to do for changes in migration configuration entities to take effect? Share your answers in the comments. Also, I would be grateful if you share this blog post with others.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Agaric Collective: Defining Drupal migrations as configuration entities with the Migrate Plus module

23 August 2019 - 3:28pm

Today, we are going to talk about how to manage migrations as configuration entities. This functionality is provided by the Migrate Plus module. First, we will explain the difference between managing migrations as code or configuration. Then, we will show how to convert existing migrations. Finally, we will talk about some important options to include in migration configuration entities. Let’s get started.

Drupal migrations: code or configuration?

So far, we have been managing migrations as code. This is functionality provided out of the box. You write the migration definition file in YAML format. Then, you place it in the migrations directory of your module. If you need to update the migration, you make the modifications to the files and then rebuild caches. More details on the workflow for migrations managed in code can be found in this article.

Migrate Plus offers an alternative to this approach. It allows you to manage migrations as configuration entities. You still use YAML files to write the migration definition files, but their location and workflow is different. They need to be placed in a config/install directory. If you need to update the migration,  you make the modifications to the files and then sync the configuration again. More details on this workflow can be found in this article.

There is one thing worth emphasizing. When managing migrations as code you need access to the file system to update and deploy the changes to the file. This is usually done by developers.  When managing migrations as configuration, you can make updates via the user interface as long as you have permissions to sync the site’s configuration. This is usually done by site administrators. You might still have to modify files depending on how you manage your configuration. But the point is that file system access to update migrations is optional. Although not recommended, you can write, modify, and execute the migrations entirely via the user interface.

Transitioning to configuration entities

To demonstrate how to transition from code to configuration entities, we are going to convert the JSON migration example. You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD config JSON source migration whose machine name is udm_config_json_source. It comes with four migrations: udm_config_json_source_paragraph, udm_config_json_source_image, udm_config_json_source_node_local, and udm_config_json_source_node_remote.

The transition to configuration entities is a two step process. First, move the migration definition files from the migrations folder to a config/install folder. Second, rename the files so that they follow this pattern: migrate_plus.migration.[migration_id].yml. For example: migrate_plus.migration.udm_config_json_source_node_local.yml. And that’s it! Files placed in that directory following that pattern will be synced into Drupal’s active configuration when the module is installed for the first time (only). Note that changes to the files require a new synchronization operation for changes to take effect. Changing the files and rebuilding caches does not update the configuration as it was the case with migrations managed in code.

If you have the Migrate Plus module enabled, it will detect the migrations and you will be able to execute them. You can continue using the Drush commands provided the Migrate Run module. Alternatively, you can install the Migrate Tools module which provides Drush commands for running both types of migrations: code and configuration. Migrate Tools also offers a user interface for executing migrations. This user interface is only for migrations defined as configuration though. It is available at /admin/structure/migrate. For now, you can run the migrations using the following Drush command: drush migrate:import udm_config_json_source_node_local --execute-dependencies.

Note: For executing migrations in the command line, choose between Migrate Run or Migrate Tools. You pick one or the other, but not both as the commands provided by the two modules have the same name. Another thing to note is that the example uses Drush 9. There were major refactorings between versions 8 and 9 which included changes to the name of the commands.

UUIDs for migration configuration entities

When managing migrations as configuration, you can set extra options. Some are exposed by Migrate Plus while others come from Drupal’s configuration management system. Let’s see some examples.

The most important new option is defining a UUID for the migration definition file. This is optional, but adding one will greatly simplify the workflow to update migrations. The UUID is used to keep track of every piece of configuration in the system. When you add new configuration, Drupal will read the UUID value if provided and update that particular piece of configuration. Otherwise, it will create a UUID on the fly, attach it to the configuration definition, and then import it. That is why you want to set a UUID value manually. If changes need to be made, you want to update the same configuration, not create a new one. If no UUID was originally set, you can get the automatically created value by exporting the migration definition. The workflow for this is a bit complicated and error prone so always include a UUID with your migrations. This following snippet shows an example UUID:

uuid: b744190e-3a48-45c7-97a4-093099ba0547 id: udm_config_json_source_node_local label: 'UD migrations configuration example'

The UUID a string of 32 hexadecimal digits displayed in 5 groups. Each is separated by hyphens following this pattern: 8-4-4-4-12. In Drupal, two or more pieces of configuration cannot share the same value. Drupal will check the UUID and the type of configuration in sync operations. In this case the type is signaled by the migrate_plus.migration. prefix in the name of the migration definition file.

When using configuration entities, a single migration is identified by two different options. The uuid is used by the Drupal’s configuration system and the id is used by the Migrate API. Always make sure that this combination is kept the same when updating the files and syncing the configuration. Otherwise you might get hard to debug errors. Also, make sure you are importing the proper configuration type. The latter should not be something to worry about unless you utilize the user interface to export or import single configuration items.

Tip: If you do not have a UUID in advance for your migration, you can copy one from another piece of configuration and change some of the hexadecimal digits. Keep in mind that this could lead to a duplicated UUID if you happen to make the exact changes to the UUID in two separate files. Another option is to use a tool to generate the values. Searching online for UUID generators will yield many  tools for this.

Automatically deleting migration configuration entities

By default, configuration remains in the system even if the module that added it gets uninstalled. This can cause problems if your migration depends on custom migration plugins provided by your module. It is possible to enforce that migration entities get removed when your custom module is uninstalled. To do this, you leverage the dependencies option provided by Drupal’s configuration management system. The following snippet shows how to do it:

uuid: b744190e-3a48-45c7-97a4-093099ba0547 id: udm_config_json_source_node_local label: 'UD migrations configuration example' dependencies: enforced: module: - ud_migrations_config_json_source

You add the machine name of your module to dependencies > enforced > module array. This adds an enforced dependency on your own module. The effect is that the migration will be removed from Drupal’s active configuration when your custom module is uninstalled. Note that the top level dependencies array can have others keys in addition to enforced. For example: config and module. Learning more about them is left as an exercise for the curious reader.

It is important not to confuse the dependencies and migration_dependencies options. The former is provided by Drupal’s configuration management system and was just explained. The latter is provided by the Migrate API and is used to declare migrations that need be imported in advance. Read this article to know more about this feature. The following snippet shows an example:

uuid: b744190e-3a48-45c7-97a4-093099ba0547 id: udm_config_json_source_node_local label: 'UD migrations configuration example' dependencies: enforced: module: - ud_migrations_config_json_source migration_dependencies: required: - udm_config_json_source_image - udm_config_json_source_paragraph optional: []

What did you learn in today’s blog post? Did you know that you can manage migrations in two ways: code or configuration? Did you know that file name and location as well as workflows need to be adjusted depending on which approach you follow? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Workflows and benefits of managing Drupal migrations as configuration entities

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Srijan Technologies: Integrating Drupal with AWS Machine Learning

23 August 2019 - 1:00am

With enterprises looking for ways to stay ahead of the curve in the growing digital age, machine learning is providing them with the needed boost for seamless digital customer experience.

Categories: Drupal

Agiledrop.com Blog: Interview with pinball wizard Greg Dunlap, Senior Digital Strategist at Lullabot

22 August 2019 - 11:19pm

This time we talked with Greg Dunlap, pinball wizard and Lullabot's senior digital strategist. We spoke of how satisfying it is to work on interesting things with the right people, the importance of the Drupal Diversity and Inclusion initiative, and the close similarities between the Drupal community and Greg's local pinball community.

READ MORE
Categories: Drupal

Freelock : Better Care Network

22 August 2019 - 2:15pm
Better Care Network Don Dill Thu, 08/22/2019 - 14:15

More to come soon....

Categories: Drupal

Dropsolid: Contributing to Open Source, what's your math?

22 August 2019 - 11:58am
22 Aug

1xINTERNET made a great post about calculating how much you are giving back and contributing to Open Source. Open Source makes our business model possible and for some people in the company, and certainly yours truly, made a career out of it. Our CEO even started Dropsolid as he thought we could approach this differently. If this makes a difference for us, why not for some of our employees as well, or could it even be a big impact in recruitment?

Delivering Open Digital Experiences with Open Source components on our own Dropsolid Platform is also is very interesting for clients. We are able to deliver quality projects for a budget someone once could only dream of. The downside is that you are not alone that has this magical ability, so it is a competitive landscape. The question then is how you can stand out, how can you make that difference.

Let's honor 1xINTERNET by doing our math.

How do we contribute

Our contribution to the Drupal project can be divided in the same 3 areas:

  • Community work
  • Event, Sponsorships, and memberships
  • Source code
Community work

During the last year and even this year we spent a lot of time helping the Belgian Drupal Community. We organised Drupal User Group meetings, helped with organising Drupalcamp Belgium and are currently actively involved in organising Drupal Developer Days. Next to that is yours truly also active with the European Splash Awards as an organisational member and active as Chair of the Belgian Drupal Community. We also had several speakers from Dropsolid on all these events and I've also volunteered as Drupalcon Track Chair or Co-Chair for the DevOps track, both Drupal Europe and Drupalcon Amsterdam. Attending those meetings during the day or evening takes time, but is well worth it. And next to that we gave back to the community by going to Drupalcamp Kiev and Drupal Mountain Camp to share the knowledge.

When taking all of the above, time spent on community activities adds up to around 1 FTE. This includes time spent to organise it, give back with speaker preparations and time spent to organise those sponsorships and facilitate our employees to attend. For your information, in 2018 we had on average 65 people working for Dropsolid.

Sponsorships and memberships

Since 2018 we invested in the following events and memberships

  • Silver Sponsor for Drupal Europe 2018
  • Gold Sponsor for Drupalcamp Belgium
  • Gold Drupaljam XL 2019
  • Diamond Sponsor Drupalcon Amsterdam 2019
  • Organisation Member of Belgian Drupal Community
  • Organisation Member of the Drupal Association
  • Supporting Partner of the Drupal Association
  • Donation to the Promote Drupal Fund
  • Employ a Provisional member of the Drupal Security Team

in total we spent close to 1% of our total yearly spent in sponsorships, travel to and memberships related to the Drupal project.

Drupal.org contributions

Without a doubt we actively contribute back to the Drupal.org source code. Some contributions are also meetings from the community events and other important milestones that matter for the community. All efforts matter, not just code.

We contributed our complete install profile Dropsolid Rocketship and all our paragraphs that come with it. Dropsolid has a dedicated R&D team that consists out of 8 people. This team has a Drupal Backend & Frontend R&D Engineer, a couple DevOps engineers, 2 machine learning geniuses and myself. Next to that we supported one of our employees in his personal pet project drupalcontributions.org. Let's take a look at that site to come up with some numbers.

We support over 30 projects, have over 207 credits, with just 58 credits in the last 3 months and 181 comments in the last 3 months. I dare to say that we are ramping it up.

We currently rank nr. 42 of the organisations that contribute most back to Drupal in the last 3 months. Compared to 1xINTERNET we have a bigger headcount, but nevertheless I am equally proud, just as they are, to see this happen and to make an impact. 

Not only do we contribute with our own agenda, we also contract Swentel to spend just 1 day a month on Drupal contributions without any agreed agenda. By just supporting 1 day a month, this accounted to 17 comments in the last 30 days alone. To make a point, just counting credits isn't fair. It's how you come together and find a common solution. Communication is key and we support this with hard earned money.

How much did we contribute since 2018?

So, given that from the 8 R&D persons, we at least contribute back 1 FTE to the Drupal ecosystem. Combined with contributions that happen back from the projects by our development teams and the core contribution sponsorships for Swentel we easily get to 2 FTE's to contribute back on a total headcount of 65. 

If we add up the efforts, next to the payroll, in community work, sponsorships and memberships Dropsolid contributes an equivalent of +- 4.5% of our annual yearly spent in 2018. With the Drupalcon Amsterdam in 2019 happening as a Diamond Sponsor, this will be even bigger. We're aiming to send more than 20 payroll employees to Drupalcon and have 3 selected sessions! We're also not cutting back on our efforts to contribute back, on the contrary!

War on Talent

Being visible in the ecosystem with code contributions, event contributions, sponsorships helps to be a company people want to work for. Obviously this is not the only motivation but given that in our technical teams we employ around 1/4th of the employees as remote developers, a good company brand is important. We see that this works and this helps us to attract and retain highly skilled employees. We now have employees in countries such as Portugal, Slovenia, Poland, Ukraine, Romania, Bulgaria, ...

 

Lead, don't just follow

By actively participating, whether it is code or events or anything that decides the future of Drupal or for that matter Digital Experiences, you are ahead of the curve. This allows for more competitive offers towards clients and this helps you win deals that as a follower of the pack might not be possible. Dropsolid believes in taking the lead when it comes to being the best Drupal partner in Greater Europe, and is proud to believe we are. Up to you to believe us or not ;-)

Are you a client or looking for a job and are interested in hearing more, don't hesitate to contact us.

Nick Veenhof
Categories: Drupal

Bounteous.com: Drush 8 to Drush 9 Migration Path

22 August 2019 - 11:28am
With the release of Drupal 9 coming soon, the need to update form Drush 8 to Drush 9 is imminent.
Categories: Drupal

Amazee Labs: Contribution and Client Projects: Part Two

22 August 2019 - 4:23am
The first part of this article described why and how the stakeholders of a project can contribute to Drupal. This developer-oriented article is a summary of the Drupal.org documentation for new code contributors. We will cover: how to work on the issue queue, how to publish a project, and how to approach this process with Drupal 9 in mind. 
Categories: Drupal

Agaric Collective: Migrating Microsoft Excel and LibreOffice Calc files into Drupal

21 August 2019 - 3:03pm

Today we will learn how to migrate content from LibreOffice Calc and Microsoft Excel files into Drupal using the Migrate Spreadsheet module. We will give instructions on getting the module and its dependencies. Then, we will present how to configure the module for spreadsheets with or without a header row. There are two example migrations: images and paragraphs. Let’s get started.

Getting the code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD Google Sheets, Microsoft Excel, and LibreOffice Calc source migration whose machine name is ud_migrations_sheets_sources. It comes with four migrations: udm_google_sheets_source_node.yml, udm_libreoffice_calc_source_paragraph.yml, udm_microsoft_excel_source_image.yml, and udm_backup_csv_source_node.yml. The image migration uses a Microsoft Excel file as source. The paragraph migration uses a LibreOffice Calc file as source. The CSV migration is a backup in case the Google Sheet is not available. To execute the last one you would need the Migrate Source CSV module.

You can get the Migrate Google Sheets module using composer: composer require drupal/migrate_spreadsheet:^1.0. This module depends on the PHPOffice/PhpSpreadsheet library and many PHP extensions including ext-zip. Check this page for a full list of dependencies. If any required extension is missing the installation will fail. If your Drupal site is not composer-based, you will not be able to use Migrate Spreadsheet, unless you go around a lot of hoops.

Understanding the example set up

This migration will reuse the same configuration from the introduction to paragraph migrations example. Refer to that article for details on the configuration. The destinations will be the same content type, paragraph type, and fields. The source will be changed in today's example, as we use it to explain Microsoft Excel and LibreOffice Calc migrations. The end result will again be nodes containing an image and a paragraph with information about someone’s favorite book. The major difference is that we are going to read from different sources.

Note: You can literally swap migration sources without changing any other part of the migration.  This is a powerful feature of ETL frameworks like Drupal’s Migrate API. Although possible, the example includes slight changes to demonstrate various plugin configuration options. Also, some machine names had to be changed to avoid conflicts with other examples in the demo repository.

Understanding the source document and plugin configuration

In any migration project, understanding the source is very important. For Microsoft Excel and LibreOffice Calc migrations, the primary thing to consider is whether or not the file contains a row of headers. Also, a workbook (file) might contain several worksheets (tabs). You can only migrate from one worksheet at a time. The example documents have two worksheets: UD Example Sheet and Do not peek in here. We are going to be working with the first one.

The spreadsheet source plugin exposes seven configuration options. The values to use might change depending on the presence of a header row, but all of them apply for both types of document. Here is a summary of the available configurations:

  • file is required. It stores the path to the document to process. You can use a relative path from the Drupal root, an absolute path, or stream wrappers.
  • worksheet is required. It contains the name of the one worksheet to process.
  • header_row is optional. This number indicates which row containing the headers. Contrary to CSV migrations, the row number is not zero-based. So, set this value to 1 if headers are on the first row, 2 if they are on the second, and so on.
  • origin is optional and defaults to A2. It indicates which non-header cell contains the first value you want to import. It assumes a grid layout and you only need to indicate the position of the top-left cell value.
  • columns is optional. It is the list of columns you want to make available for the migration. In case of files with a header row, use those header values in this list. Otherwise, use the default title for columns: A, B, C, etc. If this setting is missing, the plugin will return all columns. This is not ideal, especially for very large files containing more columns than needed for the migration.
  • row_index_column is optional. This is a special column that contains the row number for each record. This can be used as unique identifier for the records in case your dataset does not provide a suitable value. Exposing this special column in the migration is up to you. If so, you can come up with any name as long as it does not conflict with header row names set in the columns configuration. Important: this is an autogenerated column, not any of the columns that come with your dataset.
  • keys is optional and, if not set, it defaults to the value of row_index_column. It contains an array of column names that uniquely identify each record. For files with a header row, you can use the values set in the columns configuration. Otherwise, use default column titles like A, B, C, etc. In both cases, you can use the row_index_column column if it was set. Each value in the array will contain database storage details for the column.

Note that nowhere in the plugin configuration you specify the file type. The same setup applies for both Microsoft Excel and LibreOffice Calc files. The library will take care of detecting and validating the proper type.

Migrating spreadsheet files with a header row

This example is for the paragraph migration and uses a LibreOffice Calc file. The following snippets shows the UD Example Sheet worksheet and the configuration of the source plugin:

book_id, book_title, Book author B10, The definitive guide to Drupal 7, Benjamin Melançon et al. B20, Understanding Drupal Views, Carlos Dinarte B30, Understanding Drupal Migrations, Mauricio Dinarte source: plugin: spreadsheet file: modules/custom/ud_migrations/ud_migrations_sheets_sources/sources/udm_book_paragraph.ods worksheet: 'UD Example Sheet' header_row: 1 origin: A2 columns: - book_id - book_title - 'Book author' row_index_column: 'Document Row Index' keys: book_id: type: string

The name of the plugin is spreadsheet. Then you use the file configuration to indicate the path to the file. In this case, it is relative to the Drupal root. The UD Example Sheet is set as the worksheet to process. Because the first row of the file contains the header rows, then header_row is set to 1 and origin to A2.

Then specify which columns to make available to the migration. In this case, we listed all of them so this setting could have been left unassigned. It is better to get into the habit of being explicit about what you import. If the file were to change and more columns were added, you would not have to update the file to prevent unneeded data to be fetched. The row_index_column is not actually used in the migration, but it is set to show all the configuration options in the example. The values will be 1, 2, 3, etc.  Finally, the keys is set the column that serves as unique identifiers for the records.

The rest of the migration is almost identical to the CSV example. Small changes were made to prevent machine name conflicts with other examples in the demo repository. For reference, the following snippet shows the process and destination sections for the LibreOffice Calc paragraph migration.

process: field_ud_book_paragraph_title: book_title field_ud_book_paragraph_author: 'Book author' destination: plugin: 'entity_reference_revisions:paragraph' default_bundle: ud_book_paragraphMigrating spreadsheet files without a header row

Now let’s consider an example of a spreadsheet file that does not have a header row. This example is for the image migration and uses a Microsoft Excel file. The following snippets shows the UD Example Sheet worksheet and the configuration of the source plugin:

P01, https://agaric.coop/sites/default/files/pictures/picture-15-1421176712.jpg P02, https://agaric.coop/sites/default/files/pictures/picture-3-1421176784.jpg P03, https://agaric.coop/sites/default/files/pictures/picture-2-1421176752.jpg source: plugin: spreadsheet file: modules/custom/ud_migrations/ud_migrations_sheets_sources/sources/udm_book_paragraph.ods worksheet: 'UD Example Sheet' header_row: 1 origin: A2 columns: - book_id - book_title - 'Book author' row_index_column: 'Document Row Index' keys: book_id: type: string

The plugin, file, amd worksheet configurations follow the same pattern as the paragraph migration. The difference for files with no header row is reflected in the other parameters. header_row is set to null to indicate the lack of headers and origin is to A1. Because there are no column names to use, you have to use the ones provided by the spreadsheet. In this case, we want to use the first two columns: A and B. Contrary to CSV migrations, the spreadsheet plugin does not allow you to define aliases for unnamed columns. That means that you would have to use A, B in the process section to refer to these columns.

row_index_column is set to null because it will not be used. And finally, in the keys section, we use the A column as the primary key. This might seem like an odd choice. Why use that value if you could use the row_index_column as the unique identifier for each row? If this were an isolated migration, that would be a valid option. But this migration is referenced from the node migration explained in the previous example. The lookup is made based on the values stored in the A column. If we used the index of the row as the unique identifier, we would have to update the other migration or the lookup would fail. In many cases, that is not feasible nor desirable.

Except for the name of the columns, the rest of the migration is almost identical to the CSV example. Small changes were made to prevent machine name conflicts with other examples in the demo repository. For reference, the following snippet shows part of the process and destination section for the Microsoft Excel image migration.

process: psf_destination_filename: plugin: callback callable: basename source: B # This is the photo URL column. destination: plugin: 'entity:file'

Refer to this entry to know how to run migrations that depend on others. In this case, you can execute them all by running: drush migrate:import --tag='UD Sheets Source'. And that is how you can use Microsoft Excel and LibreOffice Calc files as the source of your migrations. This example is very interesting because each of the migration uses a different source type. The node migration explained in the previous post uses a Google Sheet. This is a great example of how powerful and flexible the Migrate API is.

What did you learn in today’s blog post? Have you migrated from Microsoft Excel and LibreOffice Calc files before? If so, what challenges have you found? Did you know the source plugin configuration is not dependent on the file type? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Agaric Collective: Migrating Google Sheets into Drupal

21 August 2019 - 10:05am

Today we will learn how to migrate content from Google Sheets into Drupal using the Migrate Google Sheets module. We will give instructions on how to publish them in JSON format to be consumed by the migration. Then, we will talk about some assumptions made by the module to allow easier plugin configurations. Finally, we will present the source plugin configuration for Google Sheets migrations. Let’s get started.

Getting the code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD Google Sheets, Microsoft Excel, and LibreOffice Calc source migration whose machine name is ud_migrations_sheets_sources. It comes with four migrations: udm_google_sheets_source_node.yml, udm_libreoffice_calc_source_paragraph.yml, udm_microsoft_excel_source_image.yml, and udm_backup_csv_source_node.yml. The last one is a backup in case the Google Sheet is not available. To execute it you would need the Migrate Source CSV module.

You can get the Migrate Google Sheets module and its dependency using composer: composer require drupal/migrate_google_sheets:^1.0'. It depends on Migrate Plus. Installing via composer will get you both modules.  If your Drupal site is not composer-based, you can download them manually.

Understanding the example set up

This migration will reuse the same configuration from the introduction to paragraph migrations example. Refer to that article for details on the configuration. The destinations will be the same content type, paragraph type, and fields. The source will be changed in today's example, as we use it to explain Google Sheets migrations. The end result will again be nodes containing an image and a paragraph with information about someone’s favorite book. The major difference is that we are going to read from different sources. In the next article, two of the migrations will be explained. They read from Microsoft Excel and LibreOffice Calc files.

Note: You can literally swap migration sources without changing any other part of the migration.  This is a powerful feature of ETL frameworks like Drupal’s Migrate API. Although possible, the example includes slight changes to demonstrate various plugin configuration options. Also, some machine names had to be changed to avoid conflicts with other examples in the demo repository.

Migrating nodes from Google Sheets

In any migration project, understanding the source is very important. For Google Sheets, there are many details that need your attention. First, the module works on top of Migrate Plus and extends its JSON data parser. In fact, you have to publish your Google Sheet and consume it in JSON format. Second, you need to make the JSON export publicly available. Third, you must understand the JSON format provided by Google Sheets and the assumptions made by the module to configure your fields properly. Specific instructions for Google Sheets migrations will be provided. That being said, everything explained in the JSON migration example is applicable in this case too.

Publishing a Google Sheet in JSON format

Before starting the migration, you need the source from where you will extract the data. For this, create a Google Sheet document. The example will use this one:

https://docs.google.com/spreadsheets/d/1YVJt9isPNjkUNHf3YgoTx38r04TwqRYnp1LFrik3TAk/edit#gid=0

The 1YVJt9isPNjkUNHf3YgoTx38r04TwqRYnp1LFrik3TAk value is the worksheet ID which will be used later. Once you are done creating the document, you need to publish it so it can be consumed by the Migrate API. To do this, go to the File menu and then click on Publish to the web. A modal window will appear where you can configure the export. Note that it is possible to publish the Entire document or only some of the worksheets (tabs). The example document has two: UD Example Sheet and Do not peek in here. Make sure that all the worksheets that you need are published or export the entire document. Unless multiple urls are configured, a migration can only import from one worksheet at a time. If you fetch from multiple urls they need to have homogeneous structures. When you click the Publish button, a new URL will be presented. In the example it is:

https://docs.google.com/spreadsheets/d/e/2PACX-1vTy2-CGzsoTBkmvYbolFh0UDWenwd9OCdel55j9Qa37g_earT1vA6y-6phC31Xkj8sTWF0o6mZTM90H/pubhtml

The previous URL will not be used. Publishing a document is a required step, but the URL that you get should be ignored. Note that you do not have to share the document. It is fine that the document is private to you as long as it is published. It is up to you if you want to make it available to Anyone with the link or Public on the web and potentially grant edit or comment access. The Share setting does not affect the migration. The final step is getting the JSON representation of the document. You need to assemble a URL with the following pattern:

http://spreadsheets.google.com/feeds/list/[workbook-id]/[worksheet-index]/public/values?alt=json

Replace the [workbook-id] by worksheet ID mentioned at the beginning of this section, the one that is part of the regular document URL, not the published URL. The worksheet-index is an integer number starting that represents the order in which worksheets appear in the document. Use 1 for the first, 2 for the second, and so on. This means that changing the order of the worksheets will affect your migration. At the very least, you will have to update the path to reflect the new index. In the example migration, the UD Example Sheet worksheet will be used. It appears first in the document so worksheet index is 1. Therefore, the exported JSON will be available at the following URL:

http://spreadsheets.google.com/feeds/list/1YVJt9isPNjkUNHf3YgoTx38r04TwqRYnp1LFrik3TAk/1/public/values?alt=json

Understanding the published Google Sheet JSON export

Take a moment to read the JSON export and try to understand its structure. It contains much more data than what you need. The records to be imported can be retrieved using this XPath expression: /feed/entry. You would normally have to assign this value to the item_selector configuration of the Migrate Plus’ JSON data parser. But, because the value is the same for all Google Sheets, the module takes care of this automatically. You do not have to set that configuration in the source section. As for the data cells, have a look at the following code snippet to see how they appear on the export:

{ "feed": { "entry": [ { "gsx$uniqueid": { "$t": "1" }, "gsx$name": { "$t": "One Uno Un" }, "gsx$photo-file": { "$t": "P01" }, "gsx$bookref": { "$t": "B10" } } ] } }

Tip: Firefox includes a built-in JSON document viewer which helps a lot in understanding the structure of the document. If your browser does not include a similar tool out of the box, look for one in their extensions repository. You can also use a file formatter to pretty print the JSON output.

The following is a list of headers as they appear in the Google Sheet compared to how they appear in the JSON export:

  • unique_id appears like gsx$uniqueid.
  • name appears like gsx$name.
  • photo-file appears like gsx$photo-file.
  • Book Ref appears like gsx$bookref.

So, the header name from Google Sheet gets transformed in the JSON export. They get a prefix of gsx$ and the header name is transformed to all lowercase letters with spaces and most special characters removed. On top of this, the actual cell value, that you will eventually import, is in a $t property one level under the header name. Now, you should create a list of fields to migrate using XPath expressions as selectors. For example, for the Book Ref header, the selector would be gsx$bookref/$t. But that is not the way to configure the Google Sheets data parser. The module makes some assumptions to make the selector clearer. So, the gsx$ prefix and /$t hierarchy are assumed. For the selector, you only need to use the transformed name. In this case: uniqueid, name, photo-file, and bookref.

Configuring the Migrate Google Sheets source plugin

With the JSON export of the Google Sheet and the list of transformed header names, you can proceed to configure the plugin. It will be very similar to configuring a remote JSON migration. The following code snippet shows source configuration for the node migration:

source: plugin: url data_fetcher_plugin: http data_parser_plugin: google_sheets urls: 'http://spreadsheets.google.com/feeds/list/1YVJt9isPNjkUNHf3YgoTx38r04TwqRYnp1LFrik3TAk/1/public/values?alt=json' fields: - name: src_unique_id label: 'Unique ID' selector: uniqueid - name: src_name label: 'Name' selector: name - name: src_photo_file label: 'Photo ID' selector: photo-file - name: src_book_ref label: 'Book paragraph ID' selector: bookref ids: src_unique_id: type: integer

You use the url plugin, the http fetcher, and the google_sheets parser. The latter is provided by the module. The urls configuration is set to the exported JSON link. The item_selector is not configured because the /feed/entry value is assumed. The fields are configured as in the JSON migration with the caveat of using the transformed header values for the selector. Finally, you need to set the ids key to a combination of fields that uniquely identify each record.

The rest of the migration is almost identical to the JSON example. Small changes were made to prevent machine name conflicts with other examples in the demo repository. For reference, the following snippet shows part of the process, destination, and dependencies section for the Google Sheets migration.

process: field_ud_image/target_id: plugin: migration_lookup migration: udm_microsoft_excel_source_image source: src_photo_file destination: plugin: 'entity:node' default_bundle: ud_paragraphs migration_dependencies: required: - udm_microsoft_excel_source_image - udm_libreoffice_calc_source_paragraph optional: []

Note that the node migration depends on an image and paragraph migration. They are already available in the example. One uses a Microsoft Excel file as the source while the other a LibreOffice Calc document. Both of these migrations will be explained in the next article. Refer to this entry to know how to run migrations that depend on others. For example, you can run: drush migrate:import --tag='UD Sheets Source'.

What did you learn in today’s blog post? Have you migrated from Google Sheets before? If so, what challenges have you found? Did you know the procedure to export a sheet in JSON format? Did you know that the Migrate Google Sheets module is an extension of Migrate Plus? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Read more and discuss at agaric.coop.

Categories: Drupal

Lullabot: Running and testing Drupal 8 migrations in CircleCI

21 August 2019 - 6:54am

This is the second article in a series on Drupal 8 migrations which started with An Overview for Migrating Drupal Sites to 8. In this article, you will see a sample setup of a Drupal 7 to 8 migration where we provide the front and back-end teams with a daily database that has the latest configuration and content changes, plus a means for the migration team to test migrations.

Categories: Drupal

Specbee: How to make Interactive Websites and why you need one?

21 August 2019 - 4:56am
How to make Interactive Websites and why you need one? Shefali Shetty 21 Aug, 2019 Top 10 best practices for designing a perfect UX for your mobile app

Do you like people who are warm and friendly or cold and hostile? You’ve got it right! I’m comparing Interactive to Non-interactive (static) websites here. In this increasingly digital generation, it isn’t sufficient to place some content on your website and wait for it to work its magic. Providing a web User experience without interactivity is like opening a store filled with inventory without a salesperson to interact with. 
When you create an interactive website, you are forming a connection with your audience. It propels a two-way communication on a medium where you cannot directly interact with a user. Studies have proven that people are more likely to convert on, return to or recommend websites that are interactive. Drupal CMS offers a wide variety of interactive themes and modules that can be easily adapted to your website and further customized.

What is an interactive website?

Put simply, an interactive website is a website that communicates and allows for interaction with users. And by interaction, we don’t just mean allowing users to “click” and “scroll”. Offering users with content that is amusing, collaborative and engaging is the essential objective of an interactive website. An interactive website design will not just display attractive content, it will exhibit interactive content. Content that will compel users to communicate and deeply engage with the website. 
 

 
Interactive Website Designs Communicate & Engage with users 

 

Why do you need one?

Today, all businesses in the digital market are racing to expand their audience. Most of them, however, forget that increasing traffic is simply not enough. Retaining and engaging users is what converts. Engaging your users should be your prime motive and for this you will first need an interactive business website. 

  • Drives more engagement. Interactive business websites can make your website less boring, thus garnering more action. 
  • Users will spend more time on a website that interacts with them. This increases your conversion rate, decreases bounce rate and can boost the SEO of your website.
  • Develops a more personalized user experience that can result in happy users. 
  • Engaged users are more likely to maintain a long-term relationship with websites.
  • Interactive website designs can create lasting effects in user’s minds. This improves your brand awareness and reach. 
  • Interactive websites encourages users to recommend your website and link back to it.
  • More conversions means you have a better chance in making a sale!
How to make interactive websites? 

Creating an interactive website from scratch is easier and more effective as you envision and plan the customer journey from day one. Nevertheless, if you already have a website that you think is static or needs more interactive website features, it is never too late. The first step is to define your business objectives and then identify various touch points from where you can interact with your customers.
If budgets and timelines are constraints you could also look at HTML5 interactive website templates (not recommended if you need customizations).
There are various interactive website features that can increase user engagement but you should pick the ones that suit your business goals. For example, if you are sell financial services, having an interest calculator in your website can prove to be very useful. Nonetheless, the most essential interactive feature that you just cannot ignore is responsiveness. Users will respond to your website on various devices only when it looks and feels presentable.
So what kind of interactive website features or elements can you utilize for your benefit?

  • Social Media Applications

There is no denying that Social media marketing can give you the visibility like no other marketing programs if done right. Provide your users with an option to like and share your content on social media platforms like LinkedIn Twitter or Facebook. Or just to be able to follow your page. You can also display live feed from your social media page to keep users updated. 

  • Simple Interactive Tools

Offer your users with simple interactive tools like Quizzes, short Games, math tools, tax calculators, etc. connected to your business objectives. Integrating simple software tools that can provide your users with instant results have proven to boost user engagement. 

  • Interactive Page Elements

You can enhance your page elements by adding something interesting and attractive to it. For example, colourful and dynamic hover-states on links or images, on-scroll or on-click loading/animation, navigation with clicks on image stories, and much more. Add videos or animations to say more about your business in an interactive way.

  • Forms and Feedback

Allowing users to get in touch with you via a contact form is a great way to connect with them. Not only does it let you increase your database of leads, it is a nice way of saying “We care”. Feedback forms lets you identify your strengths and weaknesses via the best source – your audience! 

  • Chat Widgets

What’s better than a live person chatting with you, answering all your questions about the products or services being offered?! That’s probably the highest level of interactivity you can offer in an interactive business website. If live chat sounds like too much commitment, you could also opt for Chat bots that can be configured to answer predictive questions.

  • User-generated Content

Letting users add their content on your website is a great way to improve interactivity. This can be done in the form of Comments (in your blogs/articles section), inviting them to write guest posts, submit images or even creating a small discussion Forum.

  • Other interactive website Features

You can get creative with the interactive features you want for your audience but here’s a short list of commonly used interactive elements –

  • Google Maps makes you a more trust-worthy brand and provide a great way to improve interaction especially when they are clickable.
  • Newsletters can keep your users coming back to your website for more updates.
  • Voting and showing them results of previous polls helps increase engagement.
  • Search functionalities eases the user from the pain of navigating through your website.
  • Ratings can be a quick and interactive method of getting instant feedback that can improve your products/services/work.
  • Slideshows offer a great way to engage users and can make them want to keep going to the next image.
     
           Interactive Website Features and Elements 
                                         
Drupal for Interactive Websites

When you build your website with Drupal, you will come across multiple options in the form of modules and features that can instantly turn your static website into an interactive one. With Drupal 8, responsiveness comes out of the box. Which means that you don’t need any additional modules to make your Drupal website look great irrespective of the devices. In addition there are a variety of modules that encourage interactivity like the Search API, Contact forms module, Social Media module, Slideshow module,  SimpleNews (or newsletters) and much more! 

Rapid increase in internet speeds are one of the many reasons that have caused a nightmarish drop in the attention spans of consumers. Couple that with highly competitive digital business owners and your plain-Jane static website could be completely abandoned. To keep your users engaged and engrossed you will need to create an interactive website. A website that compels users to respond, communicate and hang in there for longer. 
Get in touch with expert Drupal developers to help you create interactive websites.

Drupal Planet Shefali ShettyApr 05, 2017 Subscribe For Our Newsletter And Stay Updated Subscribe Shefali ShettyApr 05, 2017 Recent Posts Image How to make Interactive Websites and why you need one? Image AMP It Up! The Why and How of Drupal AMP (And what it can do to your website) Image Setup Responsive Images in Drupal 8 - A Step-by-Step Guide Explore Our Drupal Services TAKE ME THERE Featured Success Stories

Know more about our technology driven approach to recreate the content management workflow for [24]7.ai

link

Find out how we transformed the digital image of world’s largest healthcare provider, an attribute that defined their global presence in the medical world.

link

Develop an internal portal aimed at encouraging sellers at Flipkart to obtain latest insights with respect to a particular domain.

link
Categories: Drupal

Srijan Technologies: My Experience with Progressive Decoupled Blocks

20 August 2019 - 11:41pm

The JS frameworks have changed quite a lot in Drupal especially with API-first concept adding to the scenario. It is only expected that developers are inclined towards learning more about JS and related possibilities.

Categories: Drupal

Web Wash: Using Code Generators in Drupal 8

20 August 2019 - 10:00pm

Code generators in Drupal are great as a productivity tool. If you need to create a module, you could easily run a few commands and have a module generated. Then if you need to create a custom block, you could run another command which will generate all the boilerplate code and add the block into a module.

If you want to create a new event subscriber, form, service, etc… There’s always a bit of boilerplate code required to get things going. For example, making sure you extend the right class and inject the correct services. A code generator makes this process quick and easy.

Most of the popular frameworks, Laravel, Symfony, Rails just to name a few, utilize code generators which create scaffolding code.

In this tutorial, you’ll learn three ways you can generate code in Drupal 8 using Drupal Console, Drush and Module Builder.

Categories: Drupal

Agaric Collective: Migrating XML files into Drupal

20 August 2019 - 7:18pm

Today we will learn how to migrate content from a XML file into Drupal using the Migrate Plus module. We will show how to configure the migration to read files from the local file system and remote locations. We will also talk about the difference between two data parsers provided the module. The example includes node, images, and paragraphs migrations. Let’s get started.

Note: Migrate Plus has many more features. For example, it contains source plugins to import from JSON files and SOAP endpoints. It provides many useful process plugins for DOM manipulation, string replacement, transliteration, etc. The module also lets you define migration plugins as configurations and create groups to share settings. It offers a custom event to modify the source data before processing begins. In today’s blog post, we are focusing on importing XML files. Other features will be covered in future entries.

Getting the code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD XML source migration whose machine name is ud_migrations_xml_source. It comes with four migrations: udm_xml_source_paragraph, udm_xml_source_image, udm_xml_source_node_local, and udm_xml_source_node_remote.

You can get the Migrate Plus module using composer: composer require 'drupal/migrate_plus:^5.0'. This will install the 8.x-5.x branch where new development will happen. This branch was created to introduce breaking changes in preparation for Drupal 9. As of this writing, the 8.x-4.x branch has feature parity with the newer branch. If your Drupal site is not composer-based, you can download the module manually.

Understanding the example set up

This migration will reuse the same configuration from the introduction to paragraph migrations example. Refer to that article for details on the configuration: the destinations will be the same content type, paragraph type, and fields. The source will be changed in today's example, as we use it to explain XML migrations. The end result will again be nodes containing an image and a paragraph with information about someone’s favorite book. The major difference is that we are going to read from XML. In fact, three of the migrations will read from the same file. The following snippet shows a reduced version of the file to get a sense of its structure:

<?xml version="1.0" encoding="UTF-8" ?> 1 Michele Metts P01 B10 ... ... B10 The definite guide to Drupal 7 Benjamin Melançon et al. ... ... P01 https://agaric.coop/sites/default/files/pictures/picture-15-1421176712.jpg 240 351 ... ...

Note: You can literally swap migration sources without changing any other part of the migration.  This is a powerful feature of ETL frameworks like Drupal’s Migrate API. Although possible, the example includes slight changes to demonstrate various plugin configuration options. Also, some machine names had to be changed to avoid conflicts with other examples in the demo repository.

Migrating nodes from a XML file

In any migration project, understanding the source is very important. For XML migrations, there are two major considerations. First, where in the XML tree hierarchy lies the data that you want to import. It can be at the root of the file or several levels deep in the hierarchy. You use an XPath expression to select a set of nodes from the XML document. In this article, the term element when referring to an XML document node to distinguish it from a Drupal node.  Second, when you get to the set of elements that you want to import, what child elements are going to be made available to the migration. It is possible that each element contains more data than needed. In XML imports, you have to manually include the child elements that will be required for the migration. The following code snippet shows part of the local XML file relevant to the node migration:

<?xml version="1.0" encoding="UTF-8" ?> 1 Michele Metts P01 B10 ... ...

The set of elements containing node data lies two levels deep in the hierarchy. Starting with data at the root and then descending one level to udm_people. Each element of this array is an object with four properties:

  • unique_id is the unique identifier for each element returned by the data/udm_people hierarchy.
  • name is the name of a person. This will be used in the node title.
  • photo_file is the unique identifier of an image that was created in a separate migration.
  • book_ref is the unique identifier of a book paragraph that was created in a separate migration.

The following snippet shows the configuration to read a local XML file for the node migration:

source: plugin: url # This configuration is ignored by the 'xml' data parser plugin. # It only has effect when using the 'simple_xml' data parser plugin. data_fetcher_plugin: file # Set to 'xml' to use XMLReader https://www.php.net/manual/en/book.xmlreader.php # Set to 'simple_xml' to use SimpleXML https://www.php.net/manual/en/ref.simplexml.php data_parser_plugin: xml urls: - modules/custom/ud_migrations/ud_migrations_xml_source/sources/udm_data.xml # XPath expression. It is common that it starts with a slash (/). item_selector: /data/udm_people fields: - name: src_unique_id label: 'Unique ID' selector: unique_id - name: src_name label: 'Name' selector: name - name: src_photo_file label: 'Photo ID' selector: photo_file - name: src_book_ref label: 'Book paragraph ID' selector: book_ref ids: src_unique_id: type: integer

The name of the plugin is url. Because we are reading a local file, the data_fetcher_plugin  is set to file and the data_parser_plugin to xml. The urls configuration contains an array of file paths relative to the Drupal root. In the example we are reading from one file only, but you can read from multiple files at once. In that case, it is important that they have a homogeneous structure. The settings that follow will apply equally to all the files listed in urls.

Technical note: Migrate Plus provides two data parser plugins for XML files. xml uses XMLReader while simple_xml uses SimpleXML. The parser to use is configured in the data_parser_plugin configuration. Also note that when you use the xml parser, the data_fetcher_plugin setting is ignored. More details below.

The item_selector configuration indicates where in the XML file lies the set of elements to be migrated. Its value is an XPath expression used to traverse the file hierarchy. In this case, the value is /data/udm_people. Verify that your expression is valid and select the elements you intend to import. It is common that it starts with a slash (/).

fields has to be set to an array. Each element represents a field that will be made available to the migration. The following options can be set:

  • name is required. This is how the field is going to be referenced in the migration. The name itself can be arbitrary. If it contained spaces, you need to put double quotation marks (") around it when referring to it in the migration.
  • label is optional. This is a description used when presenting details about the migration. For example, in the user interface provided by the Migrate Tools module. When defined, you do not use the label to refer to the field. Keep using the name.
  • selector is required. This is another XPath-like string to find the field to import. The value must be relative to the subtree specified by the item_selector configuration. In the example, the fields are direct children of the elements to migrate. Therefore, the XPath expression only includes the element name (e.g., unique_id). If you had nested elements, you could use a slash (/) character to go deeper in the hierarchy. This will be demonstrated in the image and paragraph migrations.

Finally, you specify an ids array of field names that would uniquely identify each record. As already stated, the unique_id field servers that purpose. The following snippet shows part of the process, destination, and dependencies configuration of the node migration:

process: field_ud_image/target_id: plugin: migration_lookup migration: udm_xml_source_image source: src_photo_file destination: plugin: 'entity:node' default_bundle: ud_paragraphs migration_dependencies: required: - udm_xml_source_image - udm_xml_source_paragraph optional: []

The source for the setting the image reference is src_photo_file. Again, this is the name of the field, not the label nor selector. The configuration of the migration lookup plugin and dependencies point to two XML migrations that come with this example. One is for migrating images and the other for migrating paragraphs.

Migrating paragraphs from a XML file

Let’s consider an example where the elements to migrate have many levels of nesting. The following snippets show part of the local XML file and source plugin configuration for the paragraph migration:

<?xml version="1.0" encoding="UTF-8" ?> B10 The Definitive Guide to Drupal 7 Benjamin Melançon et al. ... ... source: plugin: url # This configuration is ignored by the 'xml' data parser plugin. # It only has effect when using the 'simple_xml' data parser plugin. data_fetcher_plugin: file # Set to 'xml' to use XMLReader https://www.php.net/manual/en/book.xmlreader.php # Set to 'simple_xml' to use SimpleXML https://www.php.net/manual/en/ref.simplexml.php data_parser_plugin: xml urls: - modules/custom/ud_migrations/ud_migrations_xml_source/sources/udm_data.xml # XPath expression. It is common that it starts with a slash (/). item_selector: /data/udm_book_paragraph fields: - name: src_book_id label: 'Book ID' selector: book_id - name: src_book_title label: 'Title' selector: book_details/title - name: src_book_author label: 'Author' selector: book_details/author ids: src_book_id: type: string

The plugin, data_fetcher_plugin, data_parser_plugin and urls configurations have the same values as in the node migration. The item_selector and ids configurations are slightly different to represent the path to paragraph elements and the unique identifier field, respectively.

The interesting part is the value of the fields configuration. Taking data/udm_book_paragraph as a starting point, the records with paragraph data have a nested structure. Particularly, the book_details element has two children: title and author. To refer to them, the selectors are book_details/title and book_details/author, respectively. Note that you can go as many level deeps in the hierarchy to find the value that should be assigned to the field. Every level in the hierarchy could be separated by a slash (/).

In this example, the target is a single paragraph type. But a similar technique can be used to migrate multiple types. One way to configure the XML file is having two children. paragraph_id would contain the unique identifier for the record. paragraph_data would contain a child element to specify the paragraph type. It would also have an arbitrary number of extra child elements with the data to be migrated. In the process section, you would iterate over the children to map the paragraph fields.

The following snippet shows part of the process configuration of the paragraph migration:

process: field_ud_book_paragraph_title: src_book_title field_ud_book_paragraph_author: src_book_authorMigrating images from a XML file

Let’s consider an example where the elements to migrate have more data than needed. The following snippets show part of the local XML file and source plugin configuration for the image migration:

  P01 https://agaric.coop/sites/default/files/pictures/picture-15-1421176712.jpg       240       351             ...         ...   source: plugin: url # This configuration is ignored by the 'xml' data parser plugin. # It only has effect when using the 'simple_xml' data parser plugin. data_fetcher_plugin: file # Set to 'xml' to use XMLReader https://www.php.net/manual/en/book.xmlreader.php # Set to 'simple_xml' to use SimpleXML https://www.php.net/manual/en/ref.simplexml.php data_parser_plugin: xml urls: - modules/custom/ud_migrations/ud_migrations_xml_source/sources/udm_data.xml # XPath expression. It is common that it starts with a slash (/). item_selector: /data/udm_photos fields: - name: src_photo_id label: 'Photo ID' selector: photo_id - name: src_photo_url label: 'Photo URL' selector: photo_url ids: src_photo_id: type: string

 

The following snippet shows part of the process configuration of the image migration:

process: psf_destination_filename: plugin: callback callable: basename source: src_photo_url

The plugin, data_fetcher_plugin, data_parser_plugin and urls configurations have the same values as in the node migration. The item_selector and ids configurations are slightly different to represent the path to image elements and the unique identifier field, respectively.

The interesting part is the value of the fields configuration. Taking data/udm_photos as a starting point, the elements with image data have extra children that are not used in the migration. Particularly, the photo_dimensions element has two children representing the width and height of the image. To ignore this subtree, you simply omit it from the fields configuration. In case you wanted to use it, the selectors would be photo_dimensions/width and photo_dimensions/height, respectively.

XML file location

Important: What is described in this section only applies when you use either (1) the xml data parser or (2) the simple_xml parser with the file data fetcher.

When using the file data fetcher plugin, you have three options to indicate the location to the XML files in the urls configuration:

  • Use a relative path from the Drupal root. The path should not start with a slash (/). This is the approach used in this demo. For example, modules/custom/my_module/xml_files/example.xml.
  • Use an absolute path pointing to the XML location in the file system. The path should start with a slash (/). For example, /var/www/drupal/modules/custom/my_module/xml_files/example.xml.
  • Use a fully-qualified URL to any built-in wrapper like http, https, ftp, ftps, etc. For example, https://understanddrupal.com/xml-files/example.xml.
  • Use a custom stream wrapper.

Being able to use stream wrappers gives you many more options. For instance:

Migrating remote XML files

Important: What is described in this section only applies when you use the http data fetcher plugin.

Migrate Plus provides another data fetcher plugin named http. Under the hood, it uses the Guzzle HTTP Client library. You can use it to fetch files using any protocol supported by curl like http, https, ftp, ftps, sftp, etc. In a future blog post we will explain this data fetcher in more detail. For now, the udm_xml_source_node_remote migration demonstrates a basic setup for this plugin. Note that only the data_fetcher_plugin, data_parser_plugin, and urls configurations are different from the local file example. The following snippet shows part of the configuration to read a remote XML file for the node migration:

source: plugin: url data_fetcher_plugin: http # 'simple_xml' is configured to be able to use the 'http' fetcher. data_parser_plugin: simple_xml urls: - https://sendeyo.com/up/d/478f835718 item_selector: /data/udm_people fields: ... ids: ...

And that is how you can use XML files as the source of your migrations. Many more configurations are possible when you use the simple_xml parser with the http fetcher. For example, you can provide authentication information to get access to protected resources. You can also set custom HTTP headers. Examples will be presented in a future entry.

XMLReader vs SimpleXML in Drupal migrations

As noted in the module’s README file, the xml parser plugin uses the XMLReader interface to incrementally parse XML files. The reader acts as a cursor going forward on the document stream and stopping at each node on the way. This should be used for XML sources which are potentially very large. On the other than, the simple_xml parser plugin uses the SimpleXML interface to fully parse XML files. This should be used for XML sources where you need to be able to use complex XPath expressions for your item selectors, or have to access elements outside of the current item element via XPath.

What did you learn in today’s blog post? Have you migrated from XML files before? If so, what challenges have you found? Did you know that you can read local and remote files? Did you know that the data_fetcher_plugin configuration is ignored when using the xml data parser? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

This blog post series is made possible thanks to these generous sponsors. Contact us if your organization would like to support this documentation project, whether it is the migration series or other topics.

Next: Adding HTTP request headers and authentication to remote JSON and XML in Drupal migrations

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Read more and discuss at agaric.coop.

Categories: Drupal

Pages