Project: Whiskey Tango Foxtrot

Floren Munteanu's post about using lsyncd to trigger csync2 cross-server filesystem replication had me at lsync, which I've successfully been using to manage a poor man's "filesystem RAID" of my twin server hard drives.  But lsyncd is up to 2.0.7, and the LUA-backed support code he published back in the 2.0.4-ish days no longer works.  Usage of "inlet" has changed in 2.0.7, as has agent.etype, as detailed here.

So, without further ado, here's my lsyncd-2.0.7-ready version of the same:

-----
-- User configuration file for lsyncd.
--
-- This example synchronizes one specific directory through multiple nodes,
-- by combining csync2 and lsyncd as monitoring tools.
-- It avoids any race conditions generated by lsyncd, while detecting and
-- processing multiple inotify events in batch, on each node monitored by
-- csync2 daemon.
--
-- @author      Floren Munteanu, Brian Shensky (2.0.7 mods)
-- @link        http://www.axivo.com/
-----
settings = {
        logident        = "lsyncd",
        logfacility     = "user",
        logfile         = "/var/log/lsyncd/lsyncd.log",
        statusFile      = "/var/log/lsyncd/status.log",
        statusInterval  = 1
}

initSync = {
        delay = 1,
        maxProcesses = 1,
        action = function(inlet)
                local config = inlet.getConfig()
                local elist = inlet.getEvents(function(event)
                        return event.etype ~= "Init"
                end)
                local directory = string.sub(config.source, 1, -2)
                local paths = elist.getPaths(function(etype, path)
                        return "\t" .. config.syncid .. ":" .. directory .. path
                end)
                log("Normal", "Processing syncing list:\n", table.concat(paths, "\n"))
                spawn(elist, "/usr/sbin/csync2", "-C", config.syncid, "-x")
        end,
        collect = function(agent, exitcode)
                local config = agent.config
                if not agent.isList and agent.etype == "Init" then
                        if exitcode == 0 then
                                log("Normal", "Startup of '", config.syncid, "' instance finished.")
                        elseif config.exitcodes and config.exitcodes[exitcode] == "again" then
                                log("Normal", "Retrying startup of '", config.syncid, "' instance.")
                                return "again"
                        else
                                log("Error", "Failure on startup of '", config.syncid, "' instance.")
                                terminate(-1)
                        end
                        return
                end
                local rc = config.exitcodes and config.exitcodes[exitcode]
                if rc == "die" then
                        return rc
                end
                if agent.isList then
                        if rc == "again" then
                                log("Normal", "Retrying events list on exitcode = ", exitcode)
                        else
                                log("Normal", "Finished events list = ", exitcode)
                        end
                else
                        if rc == "again" then
                                log("Normal", "Retrying ", agent.etype, " on ", agent.sourcePath, " = ", exitcode)
                        else
                                log("Normal", "Finished ", agent.etype, " on ", agent.sourcePath, " = ", exitcode)
                        end
                end
                return rc
        end,
        init = function(event)
                local inlet = event.inlet;
                local config = inlet.getConfig()
                log("Normal", "Recursive startup sync: ", config.syncid, ":", config.source)
                spawn(event, "/usr/sbin/csync2", "-C", config.syncid, "-x")
        end,
        prepare = function(config)
                if not config.syncid then
                        error("Missing 'syncid' parameter.", 4)
                end
                local c = "csync2_" .. config.syncid .. ".cfg"
                local f, err = io.open("/etc/" .. c, "r")
                if not f then
                        error("Invalid 'syncid' parameter: " .. err, 4)
                end
                f:close()
        end
}

local sources = {
        ["/home/mytargetfolder"] = "test1"
}
for key, value in pairs(sources) do
        sync {initSync, source=key, syncid=value}
end

That is all!

We are almost 2 weeks in, and it's funny but sad how the story is being written on the Dexter tornado tragedy.  I see it playing out in the community, and in the individuals who live here, much like the death of a close family member in so many ways.  Moms are telling tales on our neighborhood Facebook support group of how they'll just spontaneously break down and cry; pictures posted of schoolkids' artwork narrated with their "wish lists", #1 being "my old house back"; and most recently, all sorts of blowback from greedy insurance companies (glances at State Farm) and utilities (spits upon Comcast), shifty contractors telling fragile homeowners to "just sign here, we'll fill in the details later"; bewilderment about which improvements need permits and other approvals from the subdivision association, Village, insurance companies and the Township; concern for our kids playing on lawns that still might have shards of broken glass on them; pets refusing to emerge from places of safety; a lack of housing rentals for displaced families; and, oddly, many victims refusing to take help from the organizations offering it because they're "just too proud" to do so.  Never mind just going to work to make a mortgage payment on a house that may no longer exist, or is too damaged to occupy.

I tried to cheer up our friend and neighbor Saturday by taking him and his son with us to the LEGO convention at Washtenaw Community College.  He admitted some depression attributable in part to "all the damage done by the tornado".

Last night, our 9 year old son climbed in bed with us after, in his own words, "having 5 consecutive nightmares" that included "our house burning down", and "another tornado coming through our neighborhood".  Yesterday, he exhibited obvious trauma at all the banging being done by all the roofing crews that surround us.

The TV trucks are gone, the helicopters no longer hover, the cops have finally throttled back their patrols, but the hard part - the personal part - of dealing with the tragedy has just begun.

In my last post, I detailed the manner in which a data in CCK field can be encrypted when saved to the database, and decrypted when retrieved from the database.

The project that needed the encryption-at-rest also demanded that the same data never be shown in the UI - on the screen - in an unobscured or native state.  This is to prevent "over-the-shoulder" viewing of the data.

The concept I had in mind was to present the field as a password field, rather than a standard-issue CCK text field.  It would be easy enough to change the field type in the Drupal Form API from "textfield" to "password", but this had an unintended consequence: Drupal password fields never pass the field's current value from the Form API to the theming engine for HTML rendering.  This is a security feature of Drupal's Form API, as it prevents a sensitive value (like a password) from being intercepted "over the wire" or in the HTML Source despite the fact that the field would otherwise show dots or asterisks in the "password" field.

But, in this case, the presentment of that sensitive data to the "password" field was acceptable!  We just needed a way to stuff the value="sensitivedata" attribute into the <input type="password"> tag the Drupal themer generates.

This can be accomplished with some Form API trickery that leverages the "after build" hook.  I'll cut to the chase, in my custom module called "encdectest":

/**
 *
 */
function encdectest_form_alter(&$form, &$form_state, $form_id) {
  if (($form_id == 'encdectest_node_form') && ($form['type']['#value'] == 'encdectest')) {
    $form['#after_build'][] = '_encdectest_form_encdectest_alter_after_build';
  }
}

/**
 *
 */
function _encdectest_form_encdectest_alter_after_build($form, $form_state) {
  $form['field_test_field'][0]['value']['#type'] = 'password';
  $form['field_test_field'][0]['value']['#attributes']['value'] = $form['field_test_field'][0]['value']['#value'];
  return $form;
}

We require only two small functions:

• One of them officially a hook - hook_form_alter() - which sets up some #after_build post-form-build scaffolding
• The other, a custom function with the Actual Magic:
  • Set the field's #type from textfield to password - to fool the theming engine into rendering the field as an HTML <input type="password> field
  • Since a password field will not officially recognize a #value form attribute, we fake one by way of the #attributes array.  In it, set the quasi-attribute "value" to the actual value of the field - you know, what it would be if this were a normal text field.

The theming engine is not smart enough to block an HTML form element attribute of "value=something" in it.  The net result is that the field's value does come through into the password field on the browser screen. 

It's important to recognize that this is NOT SECURE.  A simple "Show HTML Source" or other network traffic interceptor will quickly reveal the actual value being obscured.  In our case, this is acceptable risk, because the server link is encrypted (via HTTPS), and we assert that the end user retains control over the workstation and browser.  The desired result - that the field remains editable while its value being obscured from over-the-shoulder eyeballs - is achieved.

Another option might be to check out the "CCK Password Field" module:
http://drupal.org/project/cckpassword
It provides double-blind data-entry verification on password-style fields.  No D7 port as of yet.

If you find this approach acceptable if only the sensitive data were NOT presented in the clear in the HTML source code, you could potentially leverage JQuery and the jquery.crypt.js library (http://www.itsyndicate.ca/jquery/) to obfuscate the sensitive value while in transit to the browser.  I'll leave that exercise to you, the reader.

A directive came down to my team to ensure that a field containing sensitive data is "encrypted at rest", meaning the data must sit in the database in an encrypted state.  If someone were to hack into the database, they'd see junk in the field.

Turns out, getting this done in Drupal 6 is easier than one might hope.  This post held the answer, and is accurate as of 15 February 2012:

http://drupal.org/node/1143106

Sure enough, all I needed to do was the following:

1. Create the specific Content Type
2. Create a new empty custom module (I called mine encdectest)
3. Create a nodeapi hook in encdectest.module - with the "presave" and "load" ops therein
4. Carefully select what kind of encryption to use.  There's DES, AES, custom, and other 2-way encryptions to choose from.  There's even some built into the MySQL database server we use - the ENCODE() and DECODE() functions - so you could theoretically issue the db_query calls to get the transformations there, with the added benefit of being able to issue direct-database queries on encrypted ("encoded") data by way of SQL statements that call DECODE() therein.  This is ultimately what we chose to do.

Here's the barebones module code:

/**
 * portal to the encryption/decryption mechanisms
 */
function encdectest_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
  $key = 'test';
  if (($node->type == "encdectest") && ($op == "presave"))  {
    $node->field_test_field[0]["value"]  = _encdectest_encrypt($node->field_test_field[0]["value"], $key);
  }
  if (($node->type == "encdectest") && ($op == "load")) {
    $node->field_test_field[0]["value"]  = _encdectest_decrypt($node->field_test_field[0]["value"], $key);
  }
}

Replace the _encdectest_encrypt() and _encdectest_decrypt() functions with your own.  That is IT!

UPDATE: Here is the crypt code we chose to use.  Note that database storage of the encrypted field requires twice the actual original length on account of the use of HEX() and UNHEX().  AFAIAC this is a small price to pay for 7-bitness of the encrypted data.

/**
 * encryption function with key
 */
function _encdectest_encrypt($string, $key) {
  $res = db_query("SELECT HEX(AES_ENCRYPT('%s','%s')) AS transformed", $string, $key);
  $row = db_fetch_object($res);
  return $row->transformed;
}

/**
 * decryption function with key
 */
function _encdectest_decrypt($string, $key) {
  $res = db_query("SELECT AES_DECRYPT(UNHEX('%s'),'%s') AS transformed", $string, $key);
  $row = db_fetch_object($res);
  return $row->transformed;
}

Be sure to check out my companion post about presenting "sensitive" data to the UI in view and edit mode, for an end-to-end solution for handling sensitive data from the users' eyes to the database record and field.

In this blog post, I shall do nothing more than declare my undying love for the MIUI ROM for Android on my 2-year-old Nexus One (N1).  I endured 2.1, tolerated Froyo despite its issues, then appreciated a reasonable refresh in Gingerbread.  But none of them - NONE - compared to the magic I experienced when I rooted my phone and installed the MIUI ROM.

In short, I always valued the freedom that my unlocked N1 running Android afforded.  Still, I silently continued to exercise a green eye toward all of my iPhone-carrying colleagues.  I'd secretly play with my wife's iPhone 3G and 4 while she was in the shower.  Why couldn't my "stock" N1 exhibit the "cool factor" of an iOS device?  Google's announcement that the N1 would NOT get an official 4.0 "ICS" update, I declared myself "released" from the unwritten contract between myself and T-Mobile and Google, and allowed myself to turn off the beaten path.

I've been running MIUI for 2 weeks now.  Here are the reasons I love MIUI, grand or trite, better or worse:

1. Stock MIUI apps work so well that they eliminate the need for many third party apps, including:
2. SMS popups - When I get an SMS message, the sender pops up with a respond dialog box, no app launching required.
3. Email app - better than stock, better than Maildroid IMO.  It just works, its elegantly designed, and does only what it's supposed to.
4. Music app - I had resorted to using Winamp for Android, but no longer.  I get a parametric equalizer, cover art, lockscreen controls, easy add-track-to-favorites-playlist, pause and resume on headphone jack changes, quick-fade in and out at the beginning and ends of tracks, and even duck-under notifications.  Thoughtful touches everywhere.
5. Data/minutes management - I had used a third-party app for this, but the MIUI folks anticipated this in ICS and jumped the gun.
6. A better stock font for better readability and management of screen real estate.  I cannot emphasize how important this is.  Readability of the display is paramount, and the original team at Google got this all wrong.  MIUI fixed it.
7. App launcher.  Yes, it's iOS like.  It's simple.  It works.  No more drawer.  Love it.
8. App launcher's "remove app" implementation  Like iOS - drag the app icon to the trash.  Done.
9. App launcher's use of subfolders - again, like iOS, and preferred by me.
10. App launcher's pagination UI effects.  I absolutely LOVE the "Falldown" effect.  Feels like I'm flipping though a set of paint color samples when I browse apps.
11. Themeing.  There are countless themes out there.  And, yes, there's one that makes MIUI look EXACTLY like iOS.  I tried it.  I liked it.  The novelty wore off after 2 days.  I went instead with a lean, mean theme that emphasizes readability above all else.  It has enhanced my "usability relationship" with the device.
12. USB management - when I plug the N1 in to a computer, MIUI asks me if I want to connect the SD card.  Thanks for asking!
13. Full-screen Browser - in a small form factor, every millimeter counts.
14. Orientation lock - BETTER than iOS, it asks if you want to lock/unlock if you rotate.
15. Browser favorites and history management - more streamlined than stock.
16. Stock lockscreen capabilities - when you unlock, the "place" on the screen where you unlock determines the destination: Phone, SMS, or main app launcher screen.  A thoughtful design.
17. More memory available due to elimination of third-party apps (above).  I gained 20Mb of internal memory as a result of apps not installed because of MIUI features.
18. Catalog of available themes - The theme browser is pretty cool, and the selection is awesome.
19. Layout and UI interface of the Phone application - again, very iOS-like.  The new layout is thoughtfully designed.
20. Notifications + Toggles.  Easy to get to, and built into the ROM.
21. Performance.  The UI doesn't hesitate as much, presumably because it swaps less when in use.  It just feels snappier.

I'll add more as I think of them.  But as far as I am concerned, MIUI is what stock Android should have been all along.  I look forward to another 2+ good years with my N1.

This morning, Reed Hastings emailed his customers to "explain" the recent price hikes.  It apparently was part of a plan to separate out the DVD delivery and streaming businesses.

I don't buy it.

With the separation streaming and DVD delivery into separate Netflix and Qwikster brands and Web sites, we lose the ability to manage streaming and DVD requests in one place.  No longer will you be able to view a single library of titles and choose between DVD and streaming delivery of a title.  There will be 2 separate sites and 2 queues, with no way to play "streaming" and "DVD snail mail" delivery experiences and against each other.

I'm no conspiracy theorist, but there's something more sinister going on here.  Reed knows it, but refuses to be forthright.  He risks additional discontent - and lost subcriptions - among the remaining Netflix faithful, by effectively removing features from the Netflix experience.  One look at his blog post comments is all that is necessary to exemplify this outrage: http://goo.gl/VM57p.

This price increase and separation of businesses at first appears to be the first manifestation of what Netflix followers have always believed: that the DVD delivery business is a ball-and-chain with a finite lifespan that acts as a drag on the streaming delivery business.  The streaming business is where the real growth is.  It's the future of distribution.  DVD delivery is a dodo.

This "revelation" in Reed's email is supposed to appease those of us who carped so loudly about the price increases that came as a result of the unbundling of streaming and DVD delivery businesses.

Reed's move amounts to a classic blackjack split - a way to guard against losses to the House by turning a mediocre hand comprised of a pair of duds into two equal chances for one of the two to yield a winning hand.  Here, though, the House is the Hollywood TV and movie distribution cartel that continues to resist content streaming in the name of preventing piracy.

If the Cartel succeeds by pricing Netflix out of existence, Reed still has Qwikster.
If the Cartel fails, Netflix blossoms and Qwikster canbe put to pasture.

This showdown at the Hollywood Corral could not have been put into play unless Reed split the business in two.

It clearly shows me that Reed has his company's long-term interests ahead of the short-term interests of his customers.  But the recent customer defections, and seeming email smooth-over, reveal that his focus on the future has blinded him to the plight of the present: an angry customer base so furious at being treated as pawns that it'd rather not play at all.  It's a theme that should sound familiar - mobile phone customers so mad at device inavailability (iPhone) and early termination fees (AT&T, Verizon and Sprint) that they'd sooner "not play" the vendor's game and instead go seek alternatives (contract-free Android phones on MNVO plans).  We want to view Netflix as the perennial underdog, but their willingness to sacrifice part of their customer base - to treat their customers as pawns in going toe to toe with Hollywood - shows it's a big dog now.  Surely Netflix will survive, but at the expense of a great deal of goodwill.

What it should reveal to you is that, at Netflix, it's all about the future.  Let's hope that the company can survive its present.

Three steps:

1. Make replicated versions of the original PNGs with the _hover in the name
2. Run the _hover files in bulk through ImageMagick to make these images "brighter"
3. Add the JQuery JS to the HTML to support the hover-over versions

That's it!

#
# give me all not-already-created PNGs. Make _hover versions of them
#
find . -name txt\*.png -not -name txt\*_hover.png -exec basename {} .png \; | \
  xargs -n 1 -I{} cp {}.png {}_hover.png
#
# increase brightness 25% for all _hover versions of the PNGs
#
find . -name \*_hover.png -exec mogrify -modulate 125,100,100 {} \;

And...in the HTML:

  <script type="text/javascript" language="JavaScript">
    $(document).ready(function(){
      $('#submit').hover(
        function(){ // Change the input image's source when we "roll on"
          $(this).attr({ src : 'images/txtSubmit_hover.png'});
        },
        function(){ // Change the input image's source back to the default on "roll off"
          $(this).attr({ src : 'images/txtSubmit.png'});             
        }
      );     
    });
  </script>

...

<input type="image" name="Submit" id="submit" value="Submit" src="{{$IMAGES}}/txtSubmit.png" />

I recently pushed a Drupal installation from a Development server to a Production server.  All seemed to migrate without issue.

On the Production server, I noticed that anonymous users had no $_SESSION data available to them.  What was different about our Production install that would allow this?

A little Googling turned up this:
http://drupal.org/node/353428

Turns out, the file transfer went well.  The MySQL migration got hosed.  The record for UID 0 got hosed in Production - it was reassigned UID 30, which was arbitrarily the next number in the AUTO_INCREMENT sequence.

Why?

MySQL's auto-incrementer will fire its assignment code if you pass "0" as the value in the INSERT statement unless you turn this behavior off!  See:
http://shrubbery.mynetgear.net/c/display/W/MySQL+dump+and+restore#MySQLdumpandrestore-InsertsintotableswithautoincrementcolumnsfailswithERROR1062Duplicatekey

The way around this is to bookend the following statements around your data migration activities:

SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO';
... inserts go here ...
SET SQL_MODE=@OLD_SQL_MODE;

This will turn the assign-on-zero-value behavior off during your data migration.

Courtesy of:
http://www.nichesoftware.co.nz/blog/200907/drupal-views-date-trap
Thanks, Bevan!

Imagine that you have a Drupal view set up for users to issue queries via exposed filters.  Now, imagine that your users tend to use this view to locate and drill down into individual records, and that it's a declared pain for the users to have to endure the rendering of the view results with a single record, forcing the user to talk action and click to drill down.

The users do NOT have to endure those extra clicks!  You can check the size of the result set, and if it's just one record, you can redirect to the node proper!  This is done via the Views API and the HOOK_views_pre_render() hook.

Throw the following into a custom module, making the requisite changes that point the code at your user-facing view.

function MYMODULE_views_pre_render(&$view) {
  // hook_pre_render for the "MYVIEWNAME" view:
  ($view->name == 'MYVIEWNAME') && MYMODULE_MYVIEWNAME_views_pre_render($view);
}

function MYMODULE_MYVIEWNAME_views_pre_render(&$view) {
  if (count($view->result)) {
    // if the resultset is only one record, just header-redirect to that node
    if (count($view->result) == 1) {
      drupal_goto('node/' . $view->result[0]->nid);
    } else {
      drupal_set_message(t('More than one entry was found that matches the data you provided.'));
    }
  } else {
    // this is one way of ensuring that we only display the not-found message when a query has been run
    // perhaps there is a better way to determine if a query has been run, but I don't know about it yet.
    if (!empty($view->display['PAGE_NAME']->handler->handlers['filter']['type']->query->table_queue['FILTER_FIELD'])) {
      drupal_set_message(t('Sorry, but we cannot locate any records with the criteria you specified.  Please try again.'), 'error');
    }
  }
}

That's it!