Newsfeeds

Drupal 8 migration from a SOAP API mikeryan Tuesday, May 15, 2018 - 10:12am

Returning from my sabbatical, as promised I’m catching up on blogging about previous projects. For one such project, I was contracted by Acquia to provide migration assistance to a client of theirs [redacted, but let’s call them Acme]. This project involved some straightforward node migrations from CSV files, but more interestingly required implementing two ongoing feeds to synchronize external data periodically - one a SOAP feed, and the other a JSON feed protected by OAuth-based authentication. There were a number of other interesting techniques employed on this project which I think may be broadly useful and haven’t previously blogged about - all-in-all, there was enough to write about on this project that rather than compose one big epic post, I’m going to break things down in a series of posts, spread out over several days so as not to spam Planet Drupal. In this first post of the sequence, I’ll cover migration from SOAP. The full custom migration module for this project is on Gitlab.

A key requirement of the Acme project was to implement an ongoing feed, representing classes (the kind people attend in person, not the PHP kind), from a SOAP API to “event” nodes in Drupal. The first step, of course, was to develop (in migrate_plus) a parser plugin to handle SOAP feeds, based on PHP’s SoapClient class. This class exposes functions of the web service as class methods which may be directly invoked. In WSDL mode (the default, and the only mode this plugin currently supports), it can also report the signatures of the methods it supports (via __getFunctions()) and the data structures passed as parameters and returned as results (via __getTypes()). WSDL allows our plugin to do introspection and saves the need for some explicit configuration (in particular, it can automatically determine the property to be returned from within the response).

migrate_example_advanced (a submodule of migrate_plus) demonstrates a simple example of how to use the SOAP parser plugin - the .yml is well-documented, so please review that for a general introduction to the configuration. Here’s the basic source configuration for this specific project:

source: plugin: url # To remigrate any changed events. track_changes: true data_fetcher_plugin: http # Ignored - SoapClient does the fetching itself. data_parser_plugin: soap # The method to invoke via the SOAP API. function: GetClientSessionsByClientId # Within the response, the object property containing the list of events. item_selector: SessionBOLExternal # Indicates that the response will be in the form of a PHP object. response_type: object # You won’t find ‘urls’ and ‘parameters’ in the source .yml file (they are inserted # by a web UI - the subject of a future post), but for demonstration purposes # this is what they might look like. urls: http://services.example.com/CFService.asmx?wsdl parameters: clientId: 1234 clientCredential: ClientID: 1234 Password: service_password startDate: 08-31-2016 # Unique identifier for each event (section) to be imported, composed of 3 columns. ids: ClassID: type: integer SessionID: type: integer SectionID: type: integer fields: - name: ClientSessionID label: Session ID for the client selector: ClientSessionID ...

Of particular note is the three-part source ID defined here. The way this data is structured, a “class” contains multiple “sessions”, which each have multiple “sections” - the sections are the instances that have specific dates and times, which we need to import into event nodes, and we need all three IDs to uniquely identify each unique section.

Not all of the data we need for our event nodes is in the session feed, unfortunately - we want to capture some of the class-level data as well. So, while, the base migration uses the SOAP parser plugin to get the session rows to migrate, we need to fetch the related data at run time by making direct SOAP calls ourselves. We do this in our subscriber to the PREPARE_ROW event - this event is dispatched after the source plugin has obtained the basic data per its configuration, and gives us an opportunity to retrieve further data to add to the canonical source row before it enters the processing pipeline. I won’t go into detail on how that data is retrieved since it isn’t relevant to general migration principles, but the idea is since all the class data is not prohibitively large, and multiple sessions may reference the same class data, we fetch it all on the first source row processed and cache it for reference by subsequent rows.

Community contributions

SOAP Source plugin - Despite the title (from the original feature request), it was implemented as a parser plugin.

Altering migration configuration at import time - the PRE_IMPORT event

Our event feed permits filtering by the event start date - by passing a ‘startDate’ parameter in the format 12-31-2016 to the SOAP method, the feed will only return events starting on or after that date. At any given point in time we are only interested in future events, and don’t want to waste time retrieving and processing past events. To optimize this, we want the startDate parameter in our source configuration to be today’s date each time we run the migration. We can do this by subscribing to the PRE_IMPORT event.

In acme_migrate.services.yml:

services: ... acme_migrate.update_event_filter: class: Drupal\acme_migrate\EventSubscriber\UpdateEventFilter tags: - { name: event_subscriber }

In UpdateEventFilter.php:

class UpdateEventFilter implements EventSubscriberInterface { /** * {@inheritdoc} */ public static function getSubscribedEvents() { $events[MigrateEvents::PRE_IMPORT] = 'onMigrationPreImport'; return $events; }

The migration system dispatches the PRE_IMPORT event before the actual import begins executing. At that point, we can insert the desired date filter into the migration configuration entity and save it:

/** * Set the event start date filter to today. * * @param \Drupal\migrate\Event\MigrateImportEvent $event * The import event. */ public function onMigrationPreImport(MigrateImportEvent $event) { // $event->getMigration() returns the migration *plugin*. if ($event->getMigration()->id() == 'event') { // Migration::load() returns the migration *entity*. $event_migration = Migration::load('event'); $source = $event_migration->get('source'); $source['parameters']['startDate'] = date('m-d-Y'); $event_migration->set('source', $source); $event_migration->save(); } }

Note that the entity get() and set() functions only operate directly on top-level configuration properties - we can’t get and set, for example ‘source.parameters.startDate’ directly. We need to retrieve the entire source configuration, modify our one value within it, and set the entire source configuration back on the migration.

Tags Drupal Planet Drupal Migration Use the Twitter thread below to comment on this post:

Drupal 8 migration from a SOAP API https://t.co/hf8LGiATsh

— Virtuoso Performance (@VirtPerformance) May 15, 2018

There's a lot of momentum to fix media management in Drupal 8 thanks to the Media Entity module. By using a combination of Media Entity, Entity Embed, Entity Browser and some media providers such as Media entity image you could add decent media handling in Drupal 8.

Then in Drupal 8.4, the Media Entity functionality was moved into a core module called Media. However, the core module was hidden by default. Now in Drupal 8.5 it's no longer hidden and you can install it yourself.

In this tutorial, you'll learn how to install and configure the Media module in Drupal 8 core. This tutorial is an updated version of the How to Manage Media Assets in Drupal 8 tutorial where we cover Media Entity.

Configuring Entity Embed and Entity Browser for the core Media module is essentially the same as with Media Entity. So if you have experience using Media Entity, then you'll be fine using the core Media module.

What happens when you combine a dungeon-crawling game with a party game and a dexterity game? You get Fungeon Party. It’s a new game coming from WizKids wherein players take on the role of traditional dungeon-crawl classes, like wizard and barbarian, and head into a dungeon where their skills will be put to the test […]

Dust off your saddle and get prepared to optimize your workflow. There is a lot packed into 3 days in Austin. Pull on your chaps, fasten your leathers, dig in your spurs and head on over to Texas Drupal Camp. On Thursday, make sure you check out the trainings and sprints. On Friday and Saturday, catch all of the keynotes and sessions.

Our own Ryan Bateman will be at Texas Drupal Camp to share his presentation about visual regression testing.

Texas Drupal Camp is Thursday, March 31st through Saturday, June 2nd at the Norris Conference Center in beautiful Austin, TX.

I’m woefully behind on watching certain shows that I’ve been meaning to look at for quite some time. Rick & Morty is one of them. I’ve seen a shotgun of episodes here and there, but I really mean to buckle in and just go through the series. Then, I’ll have a better handle of what’s […]

The 2nd of a 4-part series. Video game composer Winifred Phillips shares ideas from her GDC 2018 talk, Music in Virtual Reality. Part 2: 3D versus 2D, with an exploration of the role spatial delivery can play in music implementation in a VR environment.

The independent space since its inception has popularized modern retro design, but while many view it as an easy way to create a game, there is a lot more than just looking at the classics.

Growth and change come from breaking out of our loops, for games and their stories this can be hard. But not all change has to be related to the gameplay loop. There are other ways to express it in the story and characters of your game.

The network behind online mod and game development communities like GameFront, IndieDB, and ModDB has released mod.io, an API geared at making it easier for developers to include mod support in their games.  ...

The Uthu Y’llan are already a group of pretty deranged murder-machines. So you know it’s really something when a group stands out from that. The Berserkers are a whirlwind of claws that will rip any enemies they get a hold of to pieces. These terrifying new additions will be available soon for Runewars. Get yourself […]

This module exposes a simple entity that is not translatable or revisionable. This will stay like this always, so do not open a feature request for revisions or translations. You can create multiple simple content types (bundles).

Every simple content type can be embedded in a block. A block is also available to render an existing simple content entity using autocomplete to select.

Think of it as a light version of Fieldable Panels Panes for Drupal 8.

Researchers have found that visual-motor synchronicity of only the hands and feet can induce a sense of illusory ownership over an invisible body interpolated between virtual hands and feet. This active method to induce a sense of illusory ownership over an invisible body at a distance has potential applications in skill learning/transfer and the concept of body-appearance-irrelevant communication in the future.

Overview

Active Facet Pills provides a block that can be used on faceted search pages to display the active facets as pills.

Often we write and contribute module, but have you ever thought or considered how the module features can be extended? In Drupal 8, we can do so by using Plugin Manager that make our modules extendable. For this, first, you need to know what is Plugin, Plugin Type and how it works. Have a look.

So what is Plugin?

In short, Plugin is small pieces of swappable functionality.

What is Plugin Type?

Plugin type is categorization or grouping of Plugins, which perform similar functionality. Drupal 8 Plugin system has three base elements:

1. Plugin Types

   The central controlling class that defines the ways plugins of this type will be discovered, instantiated and…

One life furnished in early Tolkien.

This module is based on Paragraphs module and creates a new "Generic page" content type. The main elements used to create content are included, additional paragraphs can be easily added.

Sections

The content type is built with "Section" paragraphs. For each "Section" paragraph you can add a title, you can specify if you want to display the title on the node page and you can add a custom id (this is very useful in the case that you want to set anchor links to the section).

There's a great module called the debug module. I'd give you the link… but it doesn't exist. Or rather, it's not a module you download. It's a module you write yourself, and write again, over and over again.

Do you ever want to inspect the result of a method call, or the data you get back from a service, the result of a query, or the result of some other procedure, without having to wade through the steps in the UI, submit forms, and so on?

This is where the debug module comes in. It's just a single page which outputs whatever code you happen to want to poke around with at the time. On Drupal 8, that page is made with:

• an info.yml file
• a routing file
• a file containing the route's callback. You could use a controller class for this, but it's easier to have the callback just be a plain old function in the module file, as there's no need to drill down a folder structure in a text editor to reach it.

(You could quickly whip this up with Module Builder!)

Here's what my router file looks like:

joachim_debug: path: '/joachim-debug' defaults: _controller: 'joachim_debug_page' options: _admin_route: TRUE requirements: _access: 'TRUE'

My debug module is called 'joachim_debug'; you might want to call yours something else. Here you can see we're granting access unconditionally, so that whichever user I happen to be logged in as (or none) can see the page. That's of course completely insecure, especially as we're going to output all sorts of internals. But this module is only meant to be run on your local environment and you should on no account commit it to your repository.

I don't want to worry about access, and I want the admin theme so the site theme doesn't get in the way of debug output or affect performance.

The module file starts off looking like this:

opcache_reset(); function joachim_debug_page() { $build = [ '#markup' => “aaaaarrrgh!!!!”, ]; /* // ============================ TEMPLATE return $build; */ return $build; }

The commented-out section is there for me to quickly copy and paste a new section of code anytime I want to do something different. I always leave the old code in below the return, just in case I want to go back to it later on, or copy-paste snippets from it.

Back in the Drupal 6 and 7 days, the return of the callback function was merely a string. On Drupal 8, it has to be a proper render array. The return text used to be 'It's going wrong!' but these days it's the more expressive 'aaaaarrrgh'. Most of the time, the output I want will be the result of dsm() call, so the $build is there just so Drupal's routing system doesn't complain about a route callback not returning anything.

Here are some examples of the sort of code I might have in here.

// ============================ Route provider $route_provider = \Drupal::service('router.route_provider'); $path = 'node/%/edit'; $rs = $route_provider->getRoutesByPattern($path); dsm($rs); return $build;

Here I wanted to see the what the route provider service returns. (I have no idea why, this is just something I found in the very long list of old code in my module's menu callback, pushed down by newer stuff.)

// ============================ order receipt email $order = entity_load('commerce_order', 3); $build = [ '#theme' => 'commerce_order_receipt', '#order_entity' => $order, '#totals' => \Drupal::service('commerce_order.order_total_summary')->buildTotals($order), ]; return $build;

I wanted to work with the order receipt emails that Commerce sends. But I don't want to have to make a purchase, complete and order, and then look in the mail logger just to see the email! But this is quicker: all I have to do is load up my debug module's page (mine is at the path 'joachim-debug', which is easy to remember for me; you might want to have yours somewhere else), and vavoom, there's the rendered email. I can tweak the template, change the order, and just reload the page to see the effect.

As you can see, it's quick and simple. There's no safety checks, so if you ever put code here that does something (such as an entity_delete(), it's useful for deleting entities in bulk quickly), be sure to comment out the code once you're done with it, or your next reload might blow up! And of course, it's only ever to be used on your local environment; never on shared development sites, and certainly never on production!

I once read something about how a crucial piece of functionality required for programming, and more specifically, for ease of learning to program with a language or a framework, is being able to see and understand the outcomes of the code you are writing. In Drupal 8 more than ever, being able to understand the systems you're working with is vital. There are tools such as debuggers and the Devel and Devel Contrib modules' information pages, but sometimes quick and dirty does the job too.

• Add new comment

The concept of a global village is getting more and more real with the advancement of ‘online’ world. And online shops share a major part in this advancement. But with the elevated need of building an online store, the options offering platforms to build these stores has also elevated.

Here’s where our experience and expertise come in picture. After 500+ man hours spent over building about 10+ Ecommerce websites, we’ve come to a conclusion that Drupal is indeed the best choice for building an Ecommerce website. So, here are the 11 realistic reasons to guide you through while choosing the best platform for building an Ecommerce website for you; which is undoubtedly Drupal Commerce! 

 

1. An Array of Inbuilt Features 
Drupal is priorly loaded with all the features that are required for building a website viz., product management system, payment modes, cart management, et al.

 

2. Time-Saving 
Development time reduces since the time consumed in first developing and then custom integrating two separate systems is eliminated.
 

3. SEO Friendly 
Drupal is SEO friendly and hence, helps your website rank higher in the search engine. 

 

4. Negligible Traffic Issues 
Heavy traffic is never an issue with Drupal since it is backed by a wealthy system to support the traffic.
 

5. Social Media Integration 
Social Media platforms like Facebook, Instagram, Twitter, LinkedIn, etc comes priorly integrated with Drupal. 

 

6. High on Security 
Drupal is high on security grounds and hence, comes up with an inbuilt solution for securing your data/information on the website. 

 

7. Super Easy Data Management 
Data management becomes easy with Drupal since it is the best content management system. 

 

8. Feasible for E-Commerce Websites
Easy to built and run a Drupal-based eCommerce website, whether it is a small size enterprise or large business houses. 

 

9. Inbuilt Plugins for Visitor Analysis  
The inbuilt plugins for visitor reporting and analytics help you to easily evaluate your website without any external support. 

 

10. Customization
Drupal is flexible enough to make your website a customized one. 

 

11. Every Single Code is Free!
Drupal firmly believes in maintaining the integrity, the core of Open Source Community, where nothing is chargeable and every single code is for everyone to use. 

And you thought we’re trying to sell it just because ‘We Drupal Everyday’? Well, good that now you’re aware of the selfless efforts we make to solve your tech-related confusions! We at AddWeb are Friends of Drupal Development.

A Drupal 8 module which allows administrators to track notes on webform submissions. For example, if a user requests an appointment through a webform submission, we might want to track what we did with that appointment request, something like:

Somecentury, I will be able to make it out to Spiel Essen. And when I do, I’m gonna book a tour like the one Geek Nation Tours is doing. You get so much out of it. Sights. Events. And hanging out with cool people like Rodney from Watch it Played. While I might not be […]