New Drupal Modules

Cache warmer for Drupal

Introduction

cache_warmer is a drush command that hits a set of URIs of a drupal site based on the freshness of the content.

The main purpose is to offer a complete setup for running a mostly cached drupal site without having to deal with expiration logic and instead use microcaching.

Dealing with the expiration logic is messy and inefficient it requires additional work, be it from the drupal side of things, be it from the external cache side.

Microcaching can be used with any type of site. Be it your portfolio or personal blog or a high traffic news site. You have to tweak the cache validity (or Time To Live: TTL) for your site traffic profile.

Although microcaching is particularly useful for the Nginx filesystem based cache it can be used with other caching systems like Varnish.

It can be used also for priming any type of external cache

How it works

The drush command is quite light, it performs a drush bootstrap up to
DRUSH_BOOTSTRAP_DRUPAL_DATABASE, which is equivalent to Drupal's BOOTSTRAP_DRUPAL_DATABASE, and queries the database for content that is fresh according to a given criteria.

As example consider that we want to hit the lastest 300 nodes in the http://example.com site plus the URIs listed in hub pages file hub_pages.txt site in single threaded mode.

drush cache-warmer --latest-n=300 --hub-pages-file=hub_pages.txt http://example.com

If instead we wanted to hit the URIs for all the nodes updated in the last 2 days we do:

drush cache-warmer --updated-last='-2 days' --hub-pages file=hub_pages.txt http://example.com

The --updated-last option accepts its argument as an integer representing the number of seconds ago the content was updated, e.g., --updated-last=3600 means the content updated in the last hour. It accepts also strings in strotime format. Hence the above -2 days (latest 48 hours).

The only command argument is the base URI of the site to be hit.

If you specify both a number of nodes through the - -latest-n option and a time range through the --updated-last option. The one returning the most number of items will be used. Note that this involves doing sub-queries hence is less efficient in SQL terms.

Single threaded and Parallel operating modes

You can run cache_warmer in single threaded (the default mode) or parallel. Parallel means that the requests are made in parallel, i.e., simultaneously.

The single threaded mode uses PHP cURL extension only.

Note that in order to use drupalhttprequest requires a higher level of boostrap than BOOTSTRAP_DRUPAL_DATABASE hence it would make cache_warmer less performant. Hence the option for cURL which is pretty much standard everywhere when we're considering a dependable standards observing HTTP client library.

In single-threaded mode the request are made sequentially, i.e, the URIs are hit one after the other. Hence in each request cURL blocks up until it either times out or it gets a response. You can specify the timeout that cURL uses through the --timeout option. To specify a timeout of 15 seconds specify --timeout=15. Re-using the above example:

drush cache-warmer --updated-last='-2 days' --timeout=15 --hub pages-file=hub_pages.txt http://example.com

In parallel mode the requests are made in parallel in a specified size batch. To perform the above request in parallel with 20 requests in parallel do:

drush cache-warmer --updated-last='-2 days' --parallel=20 --crawler-service-uri=http://crawl.example.com/cache-warmer --timeout=15 --hub-pages file=hub_pages.txt http://example.com

As you can see there's a new option --crawler-service-uri that specifies the URI if the crawler to be used for hitting the list of URIs in batches of 20, i.e., making 20 parallel requests.

The parallel crawler is implemented as a web service allowing for the crawler to be pluggable. There's a default crawler provided relying on Nginx Embedded Lua module ngx.location.capture_multi directive, that allows to make an arbitrary number of requests in a non-blocking way using Nginx event driven architecture coupled with Lua speed.

The focus is on speed and simplicity. The limit of the parallelism will be at the OS and network level and not in any way inherent to this command. You can make 1000 requests in parallel if you wish. The limiting factors will be the either at the network level (the network gets saturated) or your drupal site cannot cope with such a number of simultaneous requests.

Requirements Single threaded mode

To use the single threaded mode just the cURL PHP extension and PHP 5 are required.

Parallel mode

This requires to either code your own parallel crawler or to have Nginx with the Embedded Lua module.

In Debian, for example, this module is available in the nginx-extras. I provide a bleeding edge Nginx debian package that includes this module and a few less common ones while not including mail related modules. Other option is for you to build your own nginx package or binary.

It requires the Lua Socket library.

Hub pages

Hub pages are pages in your site that function as a hub for accessing content. For example if you have a page for each taxonomy term you may specify some taxonomyterm pages in the hub pages file. Here's a simple hub pages file. It's a text file with a URI relative to the base URI of the site to be hit by the crawler on each line. The sole exception to that rule is the front page which is denoted by <front>

<front>
foobar/term1
foobar/term2
featured/users

Besides the front page, <front>, we're also hitting the hub pages with URIs /foobar/term1, /foobar/term2 and /featured/users. Note that each URI is always relative to the base URI of the site and is specified always without leading slash.

Aliases and Pathauto

By default the crawler assumes that you're using clean URIs with aliases. Usually you run something like pathauto on your drupal site. If your site doesn't use aliases you can disable the crawler default behavior with the option --no-aliases.

Installation and Usage: Single threaded mode

1. Download the command and install it in a drush aware location. The easiest option is to put it in your ~/.drush directory.

2. Create your hub pages file.

3. Specify the necessary options and you're done, if just testing or priming a cache.

4. For using microcaching in a consistent manner. Create a cronjob with a schedule adapted to your site traffic profile and cache validity. Consider the above example. The cache TTL is 5 minutes and we want to keep the last 24 hours worth of content in cache and also all the hub pages. We specify a cronjob that runs every 8 minutes.

   */8 * * * * drush cache-warmer --updated-last='-24 hours' --timeout=15 --hub-pages-file=hub_pages.txt http://example.com

5. Processing the replies. The command returns a structure of the response status, timestamp, time spent in request for each URI hit in JSON encoded format. You can pipe this output to a file for further processing and analysis of the way the cache warmer is interacting with the site. With the above example.

   */8 * * * * drush cache-warmer --updated-last='-24 hours' --timeout=15 --hub-pages-file=hub_pages.txt http://example.com >> /path/to/cache_warmer.log

Installation and Usage: Parallel mode

1. Download the command and install it in a drush aware location. The easiest option is to put it in your ~/.drush directory.

2. Create your hub pages file.

3. Create your custom parallel crawler web service or install Nginx and setup a host for functioning as the parallel crawler web service. There's a suggested config in the config subdirectory. Adapt it to your liking.

4. Specify the necessary options and you're done, if just testing or priming a cache.

5. For using microcaching in a consistent manner. Create a cronjob with a schedule adapted to your site traffic profile and cache validity. Consider the above example. The cache TTL is 5 minutes and we want to keep the last 24 hours worth of content in cache and also all the hub pages. We specify a cronjob that runs every 8 minutes.

   */8 * * * * drush cache-warmer --updated-last='-24 hours' --timeout=15 --parallel=20 --crawler-service-uri=http://crawler.example.com/cache warmer --hub-pages-file=hub_pages.txt http://example.com

6. Processing the replies. The command returns a structure of the response status, timestamp, time spent in request for each URI hit in JSON encoded format. You can pipe this output to a file for further processing and analysis of the way the cache warmer is interacting with the site. With the above example.

   */8 * * * * drush cache-warmer --updated-last='-24 hours' --timeout=15 --hub-pages-file=hub_pages.txt http://example.com >> /path/to/cache_warmer.log

100-continue responses

The parallel mode works by POSTing the URIs to be hit in a form. cURL obeys the standards and sends a Expect: 100-continue header to the server when the POST data has a size above 1024 bytes. The server should reply with a 100 status code thus instructing cURL to go ahead and send the POST data.

Therefore you could see a bunch of 100 status in the JSON array containing the responses of the crawler. That's how it should be. If you want to force cURL to POST immediately and thus avoid this additional round trip to the server then send an empty Expect header.

Here's a gist with the option you need to add to the command. I might consider making it an option to the command in the future.

Scheduling the cache warmer with precision

Vixie's cron is the default cron on most UNIX flavours. Unfortunately not happy with only having an abstruse syntax it's also very imprecise. The smallest time unit is one minute and there's no certainty that a job will be triggered precisely in the exact second of the minute, i.e., if now the job is triggered at the 10 second mark of the current minute, there's no garantuee that the next minute trigger will happen also at the 10 second mark.

For greater precision and an expressive language for scheduling jobs there's the Scheme (Guile) based scheduler mcron, that has second precision.

You can even use the abstruse Vixie's cron syntax with it if you prefer.

Microcaching configuration for Drupal

Configuring Nginx for microcaching is a little beyond the scope of this documentation. I maintain a generic Nginx configuration that includes microcaching for both anonymous and authenticated users.

TODO

1. Add support for using Nginx embedded Lua module cosocket and thus create non-blocking sockets from within Nginx and thus avoid the Lua Socket library dependence.

2. Add mcron configuration example.

3. Add a script for doing graphical analysis of the crawler responses.

4. Benchmark this approach against more usual approaches to caching like using Boost for example

TAC Redirect 403 extends the Taxonomy Access Control module by allowing you to specify a redirect URL for each taxonomy term. When a site visitor navigates to a content page that is restricted by a taxonomy access control rule, instead of Drupal's standard 403 (Access Denied) page being displayed, the visitor is redirected to the URL entered for the restricting term. This can be used to send people to custom "upsell" pages.

For example, if your site has the taxonomy terms Basic and Premium, and these are used to designate content as only available to members at the corresponding membership tier, this module lets you redirect visitors attempting to access restricted content to a signup form for purchasing the necessary membership level.

Dependencies

• Taxonomy Access Control
• Link (the redirect URL is added as a link field to the taxonomy term entities)
• ModuleField (lets modules define uninstantiated fields)

This project also includes an add-on module that simplifies the administration UI when used in conjunction with the experimental TAC Alt UI interface.

Credits

This project was sponsored by Acquia as part of the Drupal Gardens project.

Site builders

This module provides no direct functionality, so only download it if another module lists it as a dependency, or if you want to write your own module that uses its API.

Developers

Field UI and API are designed with the assumption that modules define field types, and administrators create fields and instantiate them within bundles. As part of this assumption, Field API automatically deletes fields when they're no longer in use by any bundles, and Field UI populates the add existing field dropdown only with fields that are in use by at least one bundle. This presents an annoyance for modules that want to define a field and make it available for adding to bundles, but without requiring any particular bundle to use it. ModuleField compensates for this: simply implement hook_modulefield_info() with your field definition and default instance definition, and ModuleField exposes this field within Field UI's add existing field dropdown even if the field has not yet been instantiated in any bundle.

If you're interested in seeing this capability added to Drupal 8 core, please see #1426804: Provide a declarative API for modules to define fields, not just field types.

Examples

This module is used by TAC Redirect 403, so you can inspect that module's code for a real-world example. If you know of any other modules that use ModuleField, please submit an issue for that module to be referenced from here.

Credits

This project was sponsored by Acquia as part of the Drupal Gardens project.

Image gallery with javascript slideshow.

What is the infusionsoft API module?

This module acts as a layer between Drupal and infusionsoft, which makes the process of talking between the two much easier.

Why is this module needed?

Infusionsoft is a proprietary CRM, and highly available email auto responder system. This module, makes things such as adding a user to infusionsoft, or manipulating user information being sent to infusionsoft easier, by providing default functions for common actions.

Also, this module is faster than using the Infusionsoft SDK because it uses the built in drupal version of XML-RPC instead of using it twice.

This module also includes drupal specific functions for things like adding a drupal user to infusionsoft.

Default Integrations

These integrations work if you have the modules turned on. These still have to be configured before anything will happen.

Ubercart conditional actions - provides configurable conditional actions for use within ubercart

Drupal actions - provides configurable actions which can be used by drupal triggers module, and other modules which leverage the drupal actions system.

Rules - works with the rules module to provide a tags condition, rules can also leverage drupal actions to perform various functions.

Submodules

infusionsoft_log_email - a small module when enabled will log all outgoing drupal email to the receiving contact's infusionsoft correspondence record.
infusionsoft_oneway_sync - synching an email address to infusionsoft when someone updates their profile.
infusionsoft_uc_access - conditional action and api function to sync contact data from uc_adresses to infusionsoft

Install

To get started using the API, you will need to go through an initial set-up process. The result of the setup process will be an encrypted key that you will need for authentication.

1. Log in to your Infusionsoft application with Admin privileges.
2. Hover over Setup and select Misc Settings.
3. Click Application Settings on the left navigation menu.
4. Select the Miscellaneous tab
5. Scroll down to the bottom of the page, where you will see the API section.
6. Enter a passphrase in the API passphrase field.
7. (optional) enter the Enter IP addresses of servers you want to talk to infusionsoft (this is an additional security parameter that you can use to limit access to the API from certain IP addresses. It is highly recommended that you use this field to help protected your data)
8. Click the Save button at the bottom of the screen.
9. The Encrypted Key will be automatically generated.
10. Write down this key – it will be essential to getting the API to work
11. On your drupal site fill out the information on admin/settings/infusionsoft.

Use

This module provides the following actions for use in the system through the built in trigger module, or through the rules module

* Add current user to infusionsoft database
* Add tag to drupal user in the infusionsoft database
* Remove tag from a drupal user in the infusionsoft database
* Run an action set on a user in the infusionsoft database
* Add user to a campaign in the infusionsoft database
* Remove user from a campaign in the infusionsoft database
* Pause a campaign for a user in the infusionsoft database
* Resume a campaign for a user in the infusionsoft database

Rules only:
Condition: User has a specific infusionsoft tag.

Developers
_____________

This module allows developers to leverage infusionsoft without reinventing the wheel. Do this by using the services and methods found here:
http://help.infusionsoft.com/developers/services-methods
and encapsulating them by calling the service and method as the first variable passed to infusionsoft_send_request(). All subsequent variables should be passed following this first request. You should not include the API key in this request.

Also There is an infusionsoft api built especially for drupal. It is documented inside the module in the infusionsoft.api.inc file.

You need to know which service to use. method calls look like this:
Service.method (as documented http://help.infusionsoft.com/developers/services-methods)

The code in the infusionsoft module is thoroughly documented to make it easy to use, just look through infusionsoft.api.inc to find more descriptive functions to use in your own application.

Special thanks to this thread (and its contributors)
http://drupal.org/node/922964

In late 2011, Twitter launched some new tools that allows users to embedd tweets. You can grab the code from a specific tweet from the tweet's detail page OR you can use the Twitter's oEmbed endpoint (https://dev.twitter.com/docs/embedded-tweets). This module uses the oEmbed endpoint call to generate static pre-formatted blocks to your site to be used via blocks where ever you like.

The oEmbed api can go and request the information about the tweet and generate the embedded block.

NOTE: This is not a live updating block of tweets, this leverages the twitter oEmbed using the tweetid you specified in the admin settings. The block is static unless you update the ID in the settings.

This module generates 3 blocks

• Block 1 - The proper response from the twitter API formatter by twitter. The twitter JS is required and is an option in the admin for this block.
• Block 2 - A single line representation of the tweet.
• Block 3 - A stylized version of the tweet that you can skin yourself.

There is a limit to # of API request you can do per hour. This module stashes the response in the db and only makes requests when you save the settings form.

Content is not written into the block unless an ID is specified.

The Search Query Exclude module, allows site administrators to easily control the search results by providing a query to exclude nodes from the search result.

Use Cases:

1) Exclude nodes with php code using the default query provided

select nid from node_revisions where body like '%<?php%' group by nid
2) Exclude test nodes using the query
select nid from node_revisions where title like '%test%' OR body like '%test%' group by nid

Inspired from: http://www.blinkreaction.com/blog/be-aware-nodes-with-php-code-and-drupa...

Code taken from: http://www.lullabot.com/articles/hiding-content-drupals-search-system

Bash scripts to manage development and deployment of Drupal distributions using drush_make

commerce feeds processor for commerce price table

This is the Drupal 7 version of the features module and sub-theme from my presentation on 02/01/2012 at the DrupalNYC Meetup.

I formulated this features module with the broadest audience in mind and have included all the dependencies. When the module in enabled, the user will have immediate access to Views UI and so forth.

I didn't include theme_default in this version of module because it made the blocks go away rather weirdly. Anyway, you can Set Bartik Extend as a default theme in admin/appearance.

The colorbox module requires a patch to its views/colorbox_handler_field_colorbox.inc in order to display the Replacement Patterns. I've included the patch with the module although you can easily download it at http://drupal.org/node/1280456. The patch is by the developer and it worked immediately after I applied it.

Otherwise, after enabling the module, add one or more nodes of type 'basic page', and then go to the home page.

Provides a series of field-types for storing NLM-DTD based data.

Currently supports
- NLM Contributor -- stores information about an author or contributor

Will soon support
- NLM Copyright -- stores information about copyright

For more information about the NLM-DTD see:
http://dtd.nlm.nih.gov/
http://dtd.nlm.nih.gov/publishing/tag-library/3.0/index.html

This module is under active development and is not ready for production. It will eat your data, ruin the carpets, and run off with your SO.

This module exists thanks to the generous support of HighWire Press and Stanford University

Drupal ERP. Leveraging Drupal 7 Entity and Field technologies to build on-demand ERP systems.

This is the Drupal 6 version of the features module and sub-theme from my presentation on 02/01/2012 at the DrupalNYC Meetup.

I formulated this features module with the broadest audience in mind and have included all the dependencies. When the module in enabled, the user will have immediate access to Views UI and so forth.

Although I included theme_default and theme_settings in Strongarm, you will have to drush features-revert colorbox_customfields_slideshow_d6 to get the sub-theme, Garland Extend, to become the default. Or, simply make Garland Extend the default theme in admin/build/themes.

After enabling the module, add one or more nodes of type 'page', and then go to the home page.

Module that provides a content type for viewing chess games in PGN (Portable Game Notation) format. Users with appropriate permissions can upload a pgn format file, creating a PGN Node so that the game can be played through on the screen using the Ltpgn viewer by Lutz Tautenhahn ( http://www.lutanho.net/ )

Installation Instructions

Download Ltpgn viewer from http://www.lutanho.net/ and unpack into sites/all/libraries/ltpgnviewer. Activate the module as normal.

Usage

Users with "chess pgn game" permission can create "Chess PGN Game" nodes. Any user can view these nodes. Users with "chess pgn admin" can modify settings, such as the IFrame size.

Shareaholic provides a social bookmarking services.

Latest news

Enterprise search is currently in internal beta with a public release planned for March 2012. Participation prior to March is by invite only. If you would like to be involved (pre-release), or just hear more about this project, please drop us a line by using the contact form here.

What is Enterprise Search

Enterprise Search is a module which can replace standard Drupal search (on D6 or D7) with a highly tuned and pre-configured search solution powered by SOLR.

With Enterprise Search you get access to blindingly fast advanced search functionality without any of the technical overhead you would take on if you tried to configure, customise and maintain SOLR in-house.

Enterprise Search connects to Axis12's Find service which is a pay as you go service with a free trial period available (no credit card required).

You need this module if...

• Your site has got too big for Drupal Search to handle
• You are not happy with the quality of search results you're getting
• You need more flexiblity than Drupal Search can offer
• You need more features than Drupal Search can offer
• You want to move to an Enterprise search solution but you don't have the time or the budget to do it in-house
• You want the benefits of SOLR without taking on the technical/cost overhead

Highlights of Enterprise Search module

• Drupal 6 and Drupal 7 support
• Instant index creation
• Multi-index and multi-core support
• Granular permissions (per index)
• Unlimited SOLR indexes

All the power of SOLR plus additional bespoke features

• Stemming
• Spellcheck and suggestions
• Related content/did you mean
• Industry specific synonyms
• Facets

Sponsorship

This project is sponsored by Axis12. We provide specialist consulting services in Enterprise Drupal including SOLR or anything else to do with scaling, release, or performance. Visit us at www.axis12.com or Contact us.

This will be a payment method for commerce that integrates with Realex. Eventually it will hope to integrate with all the different options that realex offers, but at the moment it works remotely with realex's realauth system.

You must have a Merchant ID and Shared secret to use this module.

Adds functionality to set teasers for a node and display in a promo area block. Each promo block area created can be displayed through a block added to the blocks page or through a function call in your templates to node_promo_picker_display_block().

A promo block area is associated to a taxonomy and will only pull in teaser nodes that are associated to the same taxonomy.

Role Field provides a field for selecting roles that exist in your site.

Please note that there is no out-of-the-box permissions shenanigans going on. It just lets you pick from a list of roles. Although, combined with Rules the possibilities are pretty interesting.

The creator of this module is Raphael Dürst.

The module generates an overview of all hours noted by employees. It helps a company to review monthly / annual timetracking.

Dependencies: http://drupal.org/project/storm