You are viewing bshensky

Simplify Windows Backup Management: Use "mklink" to "bundle" your target backup folders

May. 15th, 2013 | 02:49 pm


I've been using Crashplan to back up my 10Gb worth of files from my Windows 7 workstation to a local SD card.  It works nicely.  I like Crashplan.

But this DIY'er is in need of Cloud backup, and I am too much of a cheapskate to pony up for packaged set-and-forget solutions.  And I already have an Amazon S3 account.

I found SprightlySoft's S3Sync command line utility, which is free for personal use.  It facilitates backup of local files to an Amazon S3 bucket.  And yes, their docs show that you can specify regular-expression-based filename exclusion patterns in the tool invocation.

Crashplan also allows you to cherry-pick folders targeted for backup.

But what I really want to discuss is a way to simplify management of the folders you want to back up, cherry-picking and command-line options be damned.

I would nominally be inclined to simply target my user root folder, a.k.a. %USERPROFILE%, which is C:\Users\bshensky in my case.  But really, I do not want/need to back up all the cache data stored in AppData, Searches, Roaming, Contacts, Favorites, or any of the other folders that Microsoft puts here.  I really want to target Desktop, Documents, Downloads, Music, Pictures and Videos - the folders in which genuine user-generated content resides.

I wished to "Bundle" these folders so only they would be represented, and represented easily by the ID of the container they're in.  And then it hit me - Windows now supports folder links, just like its Unix/Linux counterparts.

I read:
http://en.wikipedia.org/wiki/NTFS_symbolic_link
http://en.wikipedia.org/wiki/NTFS_junction_point

Then I tried it out - I created a standalone folder called "C:\Backup", and executed the following:

C:\Backup>mklink /j Desktop c:\Users\bshensky\Desktop
Junction created for Desktop <<===>> c:\Users\bshensky\Desktop

C:\Backup>mklink /j Documents c:\Users\bshensky\Documents
Junction created for Documents <<===>> c:\Users\bshensky\Documents


Before long, I had a SINGLE folder root that represented ONLY the folders I was interested in backing up.

I tried S3Sync, targeting the SINGLE folder root of C:\Backup - it worked PERFECTLY! (Note: the ^ carets are the "command continuation" marker in CMD.EXE):


"C:\Program Files (x86)\SprightlySoft\S3 Sync\S3Sync.exe" ^
 -AWSAccessKeyId xxxxx ^
 -AWSSecretAccessKey xxxxx ^
 -BucketName MYBUCKETNAME ^
 -SyncDirection Upload ^
 -DeleteS3ItemsWhereNotInLocalList true ^
 -LogOnlyMode false ^
 -OutputLevel 2 ^
 -CompareFilesBy ETag ^
 -LogFilePath "C:\S3SyncLog.txt" ^
 -UploadHeaders "x-amz-storage-class:REDUCED_REDUNDANCY" ^
 -ExcludeLocalFilesRegularExpression ".*\.(au|vmdk|iso)$" ^
 -LocalFolderPath "C:\Backup"



I then tried the same in Crashplan - I got rid of the hodgepodge of cherry-picked folders and instead just targeted C:\Backup.  Success!

So, the takeaway here is that the new "mklink" utility that Windows 7 sports can aid in eased management of backup and recovery.  There's nothing really new here, just a technique that might make your life a little less complicated.

Link | | Share

Stoopid Drupal Trick: Change a single target Profile variable while retaining all other values

Oct. 23rd, 2012 | 04:14 pm

Anyone who has had to make data changes to a Drupal user's Profile data quickly learns that the full set of profile variables must be reconstituted prior to a profile_save_profile() even if only one or two of those variables are being touched.

The following recipe "future-proofs" the User Profile data by keeping from enumerating the profile variable names in the code.

It uses some PHP 5.3 stuff like get_object_vars() and an anonymous function, but it does work.

global $user;
profile_load_profile($user);
// tricky way to preserve the existing profile vars
foreach(array_filter(array_keys(get_object_vars($user)), function($var){return preg_match('/^profile_/', $var);}) as $s) {
  $edit[$s] = $user->{$s};
}
$edit['profile_tos_agree_flag'] = TRUE; // The single var I want to set
profile_save_profile($edit, $user, 'Personal Information');


Let me know if it's of value to you!

Link | | Share

Stoopid Drupal Trick: Node Delete SAFETY! Let your Drupal delete nodes with impunity and recourse.

Oct. 5th, 2012 | 02:28 pm

Record deletion scares me.

The Drupal world is replete with module solutions that replace node deletions with node "inactivations" - modules that do something to the record in lieu of actual physical deletion, like setting the "Published" flag off, or a Recycle-Bin-style management solution.

But my client wanted nodes to be DELETED.  I wanted recourse, in the event the deletions were accidental.

My solution is simple: Immediately prior to node deletion, serialize and save the node itself in the watchdog facility.

<?php
function node_safety_delete_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
  switch ($op) {
    case 'delete':
      if ($node->nid) {
        watchdog('safety delete', 'Node %nid snapshot recorded prior to permanent delete', array('%nid' => $node->nid, 'node' => serialize($node)), WATCHDOG_NOTICE, NULL);
      }
      break;
  }
}

The above code is placed into a homegrown module.  The module is enabled in Drupal.

Aside: Per the API, in the call to watchdog(), parameter 3 must be a hash, and the elements must be strings.  For this, we are forced to serialize the $node object here, even though the hash itself is serialized later downstream.  You would find this out if you tried to review the watchdog logs in the Drupal UI later, with bootstrap.inc throwing exceptions everywhere.

Hereafter, the record is removed from the node table and its ilk, but I could restore it by locating the record in the watchdog logs and unserializing the data back into an object (with, perhaps, a drush script).

A simple solution for a nagging problem...

Link | | Share

An open letter to No Thai!

Aug. 2nd, 2012 | 10:45 am

An open letter to No Thai!

We folks here in Dexter, just west of Ann Arbor, love our own community, as much as we love Ann Arbor.  It's not much more than a 8 minute drive to get to the Old West Side from the Village.  You might say that Dexter has become a bit of a bedroom community for A2.  My subdivision is replete with University doctors and teachers and researchers, and members of successful businesses of every size HQ'd here in Washtenaw county.  All these folks have a couple things in common - they're often too busy to spend hours in the kitchen preparing a family meal, AND they really appreciate GREAT Thai food.

I do my best to, a couple times a month, hit up the Plymouth Road No Thai! location on my way home from Domino Farms back to Dexter.  The drive is just long enough that I might have to nuke the dishes for a few seconds once I'm home, but it's a small price to pay for great Thai at a great price.  The closest Thai restaurant to Dexter is in fact on the Old West Side, but if I'm going to drive that far from Dexter, I'll go a few extra miles for No Thai!.  Go big or go home, right?

The Dexter Patch this week reported that not one but TWO new pizza places have chosen to locate in the Village of Dexter.  That sounds good, until you realize that Dexter is already REPLETE with pizza places: eight, yes EIGHT restaurants that currently either serve pizza exclusively or feature pizza at last count, and perhaps 2 more if you include the subs restaurants that happen to serve a "pizza" on the menu.  That makes TEN pizza joints in the Village of Dexter, population 4100.  Our new person-to-pizza ratio is around 400 to 1!  Apply that math to Ann Arbor, and there would have to be 250 pizza joints in the city to maintain the ratio!  250!!!

We've been discussing it on Dexter.Patch.com and on Facebook (remember, we tornado survivors still chat with each other a LOT online).  The consensus is clear.  WE. ARE. SICK. OF. PIZZA.

Mostly, we've been pining for Thai, Mexican, Korean, Chinese, etc.  And No Thai! has been mentioned SEVERAL times in these sick-of-pizza threads.  Many have noted that you've made plans to open a new location on South University near campus, hinting at the chain's long-term success and suggesting your expansion is well deserved.

Oh, what we would not do to see No Thai! here in Dexter.

There is plenty of room for you here, with unoccupied new construction at reasonable prices.  Plus a central Village location would extend your market toward Scio Township, Chelsea and Pinckney, which also suffer from a lack of great Thai (or other Pan Asian) fare.  We have a reliable local work force, and we're not far from I-94.  Finally, I believe there are enough campus students in love with No Thai! that they'd take up the slack lost from folks like me stopping for carry out on the way home from work.

So, what do you say, Mr. No?  What we wouldn't give to answer the question, "is No Thai! coming to Dexter?" with "No kidding!"

Link | | Share

A recipe for Linux file processing on lots of files, using "find" and "-print0"

Jun. 18th, 2012 | 02:40 pm

I've used "find" with wild abandon, but recently, I've had to process more than one "-exec" command on each file "find" finds.

This recipe does this trick for doing more than one thing with each file.

pushd TARGETFOLDER
find . -type f -print0 | while read -d $'\0' FILE
do
  # ... do stuff with ${FILE} here...
done
popd


Hope this helps!

Link | | Share

POSTED. For Smoking Drivers.

Jun. 4th, 2012 | 12:38 pm

Allow me a moment to ruin your day, the way some of you have ruined mine.

During this morning's commute, I tolerated not one, not two, but THREE instances whereby the driver of the car in front of me took a final puff of their lit cigarette, and then, with more consideration for the smell and decor of their own car than for anything else, TOSSED THE LIT CIGARETTE BUTT OUT THE WINDOW, where it BOUNCED OFF THE GROUND and LANDED on the HOOD and/or WINDSHIELD of MY CAR before falling permanently to the ground where the butt joined the tens of thousands of other non-biodegradable cigarette butts swirling among the curbside pea gravel.

Not ONLY have you acted against your own self-interest and health by catching a quick puff before arriving at the office (and to be clear, I certainly acknowledge the pain and inconvenience that comes with a nicotine addiction as well as the civil liberties that permit you to do whatever you damned well please to your own body, the threat of cancer be damned, you will live forever so why not), but rather than dare to smell up the interior of your vehicle and soil the carpet with burn marks and an assortment of ash stains, you cavalierly toss the lit butt out of the crack of your driver's side window, it, a metaphor for the place in your mind where things go to be permanently forgotten, like where the contents of your trash can go after the Waste Management guys toss the contents of your own can into the back of their truck every Friday morning.  Who gives a rat's ass what happens to that butt once it's outside the confines of your own car?  That moment was SO 250 milliseconds ago!  You exist in a vacuum of rich Corinthian leather that knows only its own time and space, even as - no, precisely BECAUSE - that time and space moves with you on the way to work.

Let me be clear.  YOUR LIT BUTTS LAND ON MY CAR.  They've done so countless times over so many prior commutes.  They did so THRICE this morning.

Then, AFTER landing on my car, they took their place among all the other forgotten roadside garbage - except that felons come by periodically to pick up ACTUAL garbage, you know, wrappers and shattered hubcaps and such - but the cigarette butts are too numerous to count, and too small to "bother with", to warrant our continued attention.  Something something something tax dollars and efficiency and going after the "biggest fish" corporate bureaucratic nonsense.  Your Government At Work - leaving the cigarette butts there is indeed the RIGHT THING TO DO.  Uh huh.

Here's the part where I identify the alleged smartest of you who chime in with the "well, they don't put ash trays in cars anymore" defense.  Y'all can take a moment of silence to think about WHY they were taken out of the Big Three Standard Equipment Package, and while you're at it, please take a moment to kick your own ass for not having already stopped at the auto parts department of your local Walmart or Dollar General to pick up a sand-weighted ash tray for your car.  They DO have colors that match your interior, you know, with imaginative corporate hue names like "tobacco" and "ash".  And they're a friggin' dollar.  You do have a dollar, don't you?  Just scale back from Marlboroughs to Salems - just once - and I'm sure you can save the requisite Washington to afford a tray.

The last time I threw anything - ANYTHING - out the confines of my moving car, an eaten apple core was ejected deep into the woods of a pastoral county farm road.  A decade later, a small apple tree had grown in that spot, attracting birds and squirrels and groundhogs and such, and the air was sweet with scents of honeysuckle and Granny Smith.  I didn't get cancer.

You had better hope and pray that I'm not behind you on the way home, or tomorrow morning, or during any other commute hereafter.  If I see you toss your butt out the window, I'll be a good citizen, stop the car, and run over and pick up what you clearly must have "accidentally dropped" on the ground.  What good steward of this good earth wouldn't?  And, dear Lord, it's still lit!  God forbid we start a fire in the median!  Let me put that puppy out for you, at NO CHARGE!  Mother Nature AND the fire marshall would both be proud.

Are you fortunate enough to be driving a truck or van with a company logo on it?  Excellent!  I'll gladly rely on the bevy of new mobile technology on my smartphone to Google your company name, ask the front desk operator to route me to the head of Human Resources, and get your sorry ass fired.

The oldest of you smokers surely remember the 1970's TV commercial of the head-dressed Native American surveying the littered, road-ribboned landscape with a tear in his eye.  Yeah, I'm sure he has wiped ALL his tears away now that only the cigarette butts are left at curbside.  Much better, much better, I must say, as I buff the burn mark off the hood of my poor car.  Much better.

Link | | Share

lsyncd + csync2: Server cluster filesystem synchronization goodness!

Apr. 20th, 2012 | 05:25 pm

Floren Munteanu's post about using lsyncd to trigger csync2 cross-server filesystem replication had me at lsync, which I've successfully been using to manage a poor man's "filesystem RAID" of my twin server hard drives.  But lsyncd is up to 2.0.7, and the LUA-backed support code he published back in the 2.0.4-ish days no longer works.  Usage of "inlet" has changed in 2.0.7, as has agent.etype, as detailed here.

So, without further ado, here's my lsyncd-2.0.7-ready version of the same:

-----
-- User configuration file for lsyncd.
--
-- This example synchronizes one specific directory through multiple nodes,
-- by combining csync2 and lsyncd as monitoring tools.
-- It avoids any race conditions generated by lsyncd, while detecting and
-- processing multiple inotify events in batch, on each node monitored by
-- csync2 daemon.
--
-- @author      Floren Munteanu, Brian Shensky (2.0.7 mods)
-- @link        http://www.axivo.com/
-----
settings = {
        logident        = "lsyncd",
        logfacility     = "user",
        logfile         = "/var/log/lsyncd/lsyncd.log",
        statusFile      = "/var/log/lsyncd/status.log",
        statusInterval  = 1
}

initSync = {
        delay = 1,
        maxProcesses = 1,
        action = function(inlet)
                local config = inlet.getConfig()
                local elist = inlet.getEvents(function(event)
                        return event.etype ~= "Init"
                end)
                local directory = string.sub(config.source, 1, -2)
                local paths = elist.getPaths(function(etype, path)
                        return "\t" .. config.syncid .. ":" .. directory .. path
                end)
                log("Normal", "Processing syncing list:\n", table.concat(paths, "\n"))
                spawn(elist, "/usr/sbin/csync2", "-C", config.syncid, "-x")
        end,
        collect = function(agent, exitcode)
                local config = agent.config
                if not agent.isList and agent.etype == "Init" then
                        if exitcode == 0 then
                                log("Normal", "Startup of '", config.syncid, "' instance finished.")
                        elseif config.exitcodes and config.exitcodes[exitcode] == "again" then
                                log("Normal", "Retrying startup of '", config.syncid, "' instance.")
                                return "again"
                        else
                                log("Error", "Failure on startup of '", config.syncid, "' instance.")
                                terminate(-1)
                        end
                        return
                end
                local rc = config.exitcodes and config.exitcodes[exitcode]
                if rc == "die" then
                        return rc
                end
                if agent.isList then
                        if rc == "again" then
                                log("Normal", "Retrying events list on exitcode = ", exitcode)
                        else
                                log("Normal", "Finished events list = ", exitcode)
                        end
                else
                        if rc == "again" then
                                log("Normal", "Retrying ", agent.etype, " on ", agent.sourcePath, " = ", exitcode)
                        else
                                log("Normal", "Finished ", agent.etype, " on ", agent.sourcePath, " = ", exitcode)
                        end
                end
                return rc
        end,
        init = function(event)
                local inlet = event.inlet;
                local config = inlet.getConfig()
                log("Normal", "Recursive startup sync: ", config.syncid, ":", config.source)
                spawn(event, "/usr/sbin/csync2", "-C", config.syncid, "-x")
        end,
        prepare = function(config)
                if not config.syncid then
                        error("Missing 'syncid' parameter.", 4)
                end
                local c = "csync2_" .. config.syncid .. ".cfg"
                local f, err = io.open("/etc/" .. c, "r")
                if not f then
                        error("Invalid 'syncid' parameter: " .. err, 4)
                end
                f:close()
        end
}

local sources = {
        ["/home/mytargetfolder"] = "test1"
}
for key, value in pairs(sources) do
        sync {initSync, source=key, syncid=value}
end


That is all!

Link | | Share

The Dexter Tornado is just coming through our subdivision now - the one you didn't see coming...

Mar. 27th, 2012 | 01:16 pm
mood: sadsad

We are almost 2 weeks in, and it's funny but sad how the story is being written on the Dexter tornado tragedy.  I see it playing out in the community, and in the individuals who live here, much like the death of a close family member in so many ways.  Moms are telling tales on our neighborhood Facebook support group of how they'll just spontaneously break down and cry; pictures posted of schoolkids' artwork narrated with their "wish lists", #1 being "my old house back"; and most recently, all sorts of blowback from greedy insurance companies (glances at State Farm) and utilities (spits upon Comcast), shifty contractors telling fragile homeowners to "just sign here, we'll fill in the details later"; bewilderment about which improvements need permits and other approvals from the subdivision association, Village, insurance companies and the Township; concern for our kids playing on lawns that still might have shards of broken glass on them; pets refusing to emerge from places of safety; a lack of housing rentals for displaced families; and, oddly, many victims refusing to take help from the organizations offering it because they're "just too proud" to do so.  Never mind just going to work to make a mortgage payment on a house that may no longer exist, or is too damaged to occupy.

I tried to cheer up our friend and neighbor Saturday by taking him and his son with us to the LEGO convention at Washtenaw Community College.  He admitted some depression attributable in part to "all the damage done by the tornado".

Last night, our 9 year old son climbed in bed with us after, in his own words, "having 5 consecutive nightmares" that included "our house burning down", and "another tornado coming through our neighborhood".  Yesterday, he exhibited obvious trauma at all the banging being done by all the roofing crews that surround us.

The TV trucks are gone, the helicopters no longer hover, the cops have finally throttled back their patrols, but the hard part - the personal part - of dealing with the tragedy has just begun.

Link | | Share

Stoopid Drupal Trick: Present "sensitive" data in the Drupal UI as a prepopulated password field

Feb. 15th, 2012 | 04:38 pm

In my last post, I detailed the manner in which a data in CCK field can be encrypted when saved to the database, and decrypted when retrieved from the database.

The project that needed the encryption-at-rest also demanded that the same data never be shown in the UI - on the screen - in an unobscured or native state.  This is to prevent "over-the-shoulder" viewing of the data.

The concept I had in mind was to present the field as a password field, rather than a standard-issue CCK text field.  It would be easy enough to change the field type in the Drupal Form API from "textfield" to "password", but this had an unintended consequence: Drupal password fields never pass the field's current value from the Form API to the theming engine for HTML rendering.  This is a security feature of Drupal's Form API, as it prevents a sensitive value (like a password) from being intercepted "over the wire" or in the HTML Source despite the fact that the field would otherwise show dots or asterisks in the "password" field.

But, in this case, the presentment of that sensitive data to the "password" field was acceptable!  We just needed a way to stuff the value="sensitivedata" attribute into the <input type="password"> tag the Drupal themer generates.

This can be accomplished with some Form API trickery that leverages the "after build" hook.  I'll cut to the chase, in my custom module called "encdectest":

/**
 *
 */
function encdectest_form_alter(&$form, &$form_state, $form_id) {
  if (($form_id == 'encdectest_node_form') && ($form['type']['#value'] == 'encdectest')) {
    $form['#after_build'][] = '_encdectest_form_encdectest_alter_after_build';
  }
}

/**
 *
 */
function _encdectest_form_encdectest_alter_after_build($form, $form_state) {
  $form['field_test_field'][0]['value']['#type'] = 'password';
  $form['field_test_field'][0]['value']['#attributes']['value'] = $form['field_test_field'][0]['value']['#value'];
  return $form;
}

We require only two small functions:
  • One of them officially a hook - hook_form_alter() - which sets up some #after_build post-form-build scaffolding
  • The other, a custom function with the Actual Magic:
    • Set the field's #type from textfield to password - to fool the theming engine into rendering the field as an HTML <input type="password> field
    • Since a password field will not officially recognize a #value form attribute, we fake one by way of the #attributes array.  In it, set the quasi-attribute "value" to the actual value of the field - you know, what it would be if this were a normal text field.

The theming engine is not smart enough to block an HTML form element attribute of "value=something" in it.  The net result is that the field's value does come through into the password field on the browser screen. 

It's important to recognize that this is NOT SECURE.  A simple "Show HTML Source" or other network traffic interceptor will quickly reveal the actual value being obscured.  In our case, this is acceptable risk, because the server link is encrypted (via HTTPS), and we assert that the end user retains control over the workstation and browser.  The desired result - that the field remains editable while its value being obscured from over-the-shoulder eyeballs - is achieved.

Another option might be to check out the "CCK Password Field" module:
http://drupal.org/project/cckpassword
It provides double-blind data-entry verification on password-style fields.  No D7 port as of yet.

If you find this approach acceptable if only the sensitive data were NOT presented in the clear in the HTML source code, you could potentially leverage JQuery and the jquery.crypt.js library (http://www.itsyndicate.ca/jquery/) to obfuscate the sensitive value while in transit to the browser.  I'll leave that exercise to you, the reader.

Link | | Share

Stoopid Drupal Trick: "Encrypt at Rest" a CCK data field

Feb. 15th, 2012 | 03:55 pm

A directive came down to my team to ensure that a field containing sensitive data is "encrypted at rest", meaning the data must sit in the database in an encrypted state.  If someone were to hack into the database, they'd see junk in the field.

Turns out, getting this done in Drupal 6 is easier than one might hope.  This post held the answer, and is accurate as of 15 February 2012:

http://drupal.org/node/1143106

Sure enough, all I needed to do was the following:

  1. Create the specific Content Type
  2. Create a new empty custom module (I called mine encdectest)
  3. Create a nodeapi hook in encdectest.module - with the "presave" and "load" ops therein
  4. Carefully select what kind of encryption to use.  There's DES, AES, custom, and other 2-way encryptions to choose from.  There's even some built into the MySQL database server we use - the ENCODE() and DECODE() functions - so you could theoretically issue the db_query calls to get the transformations there, with the added benefit of being able to issue direct-database queries on encrypted ("encoded") data by way of SQL statements that call DECODE() therein.  This is ultimately what we chose to do.
Here's the barebones module code:
/**
 * portal to the encryption/decryption mechanisms
 */
function encdectest_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
  $key = 'test';
  if (($node->type == "encdectest") && ($op == "presave"))  {
    $node->field_test_field[0]["value"]  = _encdectest_encrypt($node->field_test_field[0]["value"], $key);
  }
  if (($node->type == "encdectest") && ($op == "load")) {
    $node->field_test_field[0]["value"]  = _encdectest_decrypt($node->field_test_field[0]["value"], $key);
  }
}


Replace the _encdectest_encrypt() and _encdectest_decrypt() functions with your own.  That is IT!

UPDATE: Here is the crypt code we chose to use.  Note that database storage of the encrypted field requires twice the actual original length on account of the use of HEX() and UNHEX().  AFAIAC this is a small price to pay for 7-bitness of the encrypted data.

/**
 * encryption function with key
 */
function _encdectest_encrypt($string, $key) {
  $res = db_query("SELECT HEX(AES_ENCRYPT('%s','%s')) AS transformed", $string, $key);
  $row = db_fetch_object($res);
  return $row->transformed;
}

/**
 * decryption function with key
 */
function _encdectest_decrypt($string, $key) {
  $res = db_query("SELECT AES_DECRYPT(UNHEX('%s'),'%s') AS transformed", $string, $key);
  $row = db_fetch_object($res);
  return $row->transformed;
}

Be sure to check out my companion post about presenting "sensitive" data to the UI in view and edit mode, for an end-to-end solution for handling sensitive data from the users' eyes to the database record and field.

Link | | Share