How to Use Excel Pivot Tables to Summarize Continuous Data

Suppose you have a set of data points that represent some quantity in a volume.  It could be temperature in a solid, for example.  How can you use a Pivot Table find the average temperature in a region on the solid’s surface?

Well, if you’re lucky, the region is rectangular, bounded by some Xmin and Xmax and Ymin and Ymax.  Then you can use the built-in Label Filters.

I created a set of data for this article with -20<=X<=20, -20<=Y<=20, -20<=Z<=20.  The temperature is a function of X,Y,Z using the (arbitrary) formula =2.5*X+0.25*Y^2+2.5*SIN(Z).

First, set up the Pivot Table and chose the option to Allow multiple filters per field.

In the screenshots below, we define a Region with Label Filters in the PivotTable Rows within X=[-2,8] and Y=[5,7].  The function is not dependent on the Z coordinate – we’ll use a Page Filter to constrain Z to 0.  As can be seen in the status line of the fourth screenshot below, the average temperature in that region is 16.67 degrees C.

But suppose the region is more complicated than a rectangle, or you want to summarize multiple regions?

You can define a function in VBA to “discretize” the data, as shown in the code sample, below.

Option Explicit

Function Region(rngX As Range, rngY As Range, rngZ As Range) As String

Const regA_Xmin = -2#: Const regA_Xmax = 8#

Const regA_Ymin = 5#: Const regA_Ymax = 7#

Const regB_Xmin = -6#: Const regB_Xmax = -3#

Const regB_Ymin = 5#: Const regB_Ymax = 7#

Const regC_Xmin = -2#: Const regC_Xmax = 8#

Const regC_Ymin = -5#: Const regC_Ymax = -3#

Const regD_Xmin = 9#: Const regD_Xmax = 12#

Const regD_Ymin = 5#: Const regD_Ymax = 7#

Const regE_Xmin = 16#: Const regE_Xmax = 18#

Const regE_Ymin = -15#: Const regE_Ymax = -7#

Dim x As Double, y As Double, z As Double

x = rngX.Value: y = rngY.Value: z = rngZ.Value

If x >= regA_Xmin And x <= regA_Xmax And _

   y >= regA_Ymin And y <= regA_Ymax Then

        Region = "Region A"

ElseIf x >= regB_Xmin And x <= regB_Xmax And _

       y >= regB_Ymin And y <= regB_Ymax Then

        Region = "Region B"

ElseIf x >= regC_Xmin And x <= regC_Xmax And _

       y >= regC_Ymin And y <= regC_Ymax Then

        Region = "Region C"

ElseIf x >= regD_Xmin And x <= regD_Xmax And _

       y >= regD_Ymin And y <= regD_Ymax Then

        Region = "Region D"

Else

    Region = "NA"

End If

End Function

The function accepts three Range type parameters, each of which are intended to be the address of a single cell.  It returns one of five strings: Region A, Region B, Region C, Region D, NA.

The first four constants that are defined in the first two lines are minimum and maximum X and Y coordinates for the region that were defined in the previous example with Label Filters.  So the string “Region A” is returned if X and Y are within those X-Y intervals.

When we type in cell E2 =Region(A2, B2, C2) and then fill down, Excel calls the function for each of the cells that the formula occupies and then puts the result in that cell.  It can take several seconds for all the cells to be populated, even with an i5 processor.  And the calculation can increase the drain on the battery if you’re doing this unplugged / on the road.

But when it’s complete, you can set up your Pivot Table with the regions in the row.  The average of the temperature values for Region A is shown in cell B5 below as 16.67.

Introduction to CLCL Tools

CLCL is the clipboard manager that I prefer.  But it’s lacking in extensive documentation, at least for those who don’t know Japanese.  I figured out how to configure the Tools that it comes with, so I thought I’d share that with you.

This procedure assumes that you already know how to download, install and run the main program.

Download the Tools from http://www.nakka.com/soft/clcl/index_eng.html .  There are two ZIP files (tltxt002_eng.zip and tlutl002_eng.zip, as of this writing) that provide DLLs with the functions that serve as tools.  Copy the DLLs from those ZIP files into the directory in which the CLCL executable resides.  While you’re at the Nakka website, you might as well download the other ZIP files that provide ability to manage as wider range of file formats.  Copy their DLLs, too.  As of this writing, there are two: fmtrtf002_eng.zip; fmtmeta002_eng.zip .

Invoke Tools from the Viewer menu, then choose Tool Setup…

Click the Add… button.

In the DLL textbox, enter the path and filename of one of the tool DLLs, for example, C:\Program Files (x86)\CLCL\tool_text.dll .  If you click the Browse button, you can use a File Dialog box to navigate to and select the DLL.

Now choose one of the Tools to add.  I find &Edit to be useful, so choose that.  OK your way back out.

Now when you press your hot key to call up the clipboard list, you can right-click on a text entry, then click on Edit to open a simple editor to modify the text.  When you’re done, you can click OK to put the edited content onto the clipboard, replacing the original entry.

Use the same procedure to try out the other tools.

Where Should You Stick Your Function?

Let's say you need to create a VBA function to use in a spreadsheet. Where do you put it? Quick answer, put it in a module.

In Excel, press Alt-F11, Ctrl-R (to set focus to the Project Explorer). Right click on VBAProject (xls_name), where xls_name in the file name of your workbook. Select Insert Module.

A blank code window should appear and the cursor should be positioned in it.  This is "Module1" where you can enter your code, such as...

Option Explicit
Function FortyTwo(r As Range)
If r.Value = 42 Then
    FortyTwo = "Forty-Two"
Else
    FortyTwo = r.Value
End If
End Function

If you enter 1, apple, and =6*7 in cells A1, A2 A3, here's what you get as output if you enter =FortyTwo(A1) in cell B1 and drag down to B3:

Of course, if A3 contained =6*9 you should also get Forty-Two, but that's a programming challenge for another day.

The Dreaded Non-Disclosure / Non-Compete Agreement

If you work in the USA tech industry, you've probably been asked to sign a non-disclosure / non-compete agreement with a new employer.  It may be that even your current employer asked you to sign a new one.

If you're not familiar with it, let me describe it for you in a nutshell: Anything you witness or produce on company time becomes the property of your employer.  Even if you think of something, if you thought of it on company time, that idea is your employer's property.  This explains a lot of what I've observed during my career -- many coworkers are so poignantly aware of this that the second they clock in, they empty their minds completely of thoughts and go about their duties thoughtlessly.

If asked to sign a non-disclosure / non-compete agreement, what should you do?  This ebook from IEEE should help:
https://www.ieeeusa.org/members/IPandtheengineer.pdf

Good luck!

Hard Drive Thrashing? Maybe It's Windows Update

In a previous post, I wondered if my computer problems were due to AntiVirus interfering with Windows Update.

Perhaps.  But there's definitely a problem with Windows Update. The following seemed to get rid of the excessive and endless thrashing of the hard drive:

http://www.cagedrat.com/microsoft-windows/microsoft-windows-updates-cab-files-filling-up-hard-drive/

And yes, I plug in my computer and set it to Stay Awake before downloading and installing updates.

Some Things Never Change

For over two thousand years, engineers had to contend with manufacturing issues.  We're still doing that today...

"... its design conception exceeded the engineering precision of its manufacture by a wide margin—with considerable cumulative inaccuracies in the gear trains, which would have cancelled out many of the subtle anomalies built into its design."

From the Wikipedia page "Antikythera mechanism," Google's feature for today, 2017-05-17.
https://en.wikipedia.org/wiki/Antikythera_mechanism

Dummies Data Visualization Links

The following links are listed in Data Visualization for Dummies:

https://www.edwardtufte.com/tufte/
http://visual.ly/
http://chartporn.org/
https://excelcharts.com/posts/
http://flowingdata.com/

Seems to be more for artists than for engineers, though.

A Better Way to Link to a Slide in PowerPoint

I've written about how to force Excel to use absolute paths in its links by specifying the Hyperlink Base  That works in PowerPoint, too.  Unfortunately, it interferes with PowerPoint's ability to follow internal links (links to other slides in the document).

Here's a workaround for that.  It allows you to link to another slide in the same PPT even if you have Hyperlink Base set to some location.  This was tested on PowerPoint 2003.

There are two parts to this.  The first part is to create a "bookmark" at the slide you want to link to.  Do this by creating a “Custom Slide Show”.  It's analogous to adding a bookmark in Word (but, regrettably much more complicated and less intuitive).
1.  In version 2003, choose Slide Show from the menu and then Custom Shows...
2.  Click the New... button.
3.  Enter a name for the Show in the uppermost textbox for the slide that you want to link to.  For example, you might enter "Detailed Analysis".
4.  In the left box, click on the slide that you want to "link to".  Then click the Add >> button.  (You can add multiple slides.)
5.  Click OK and then Close.

The second part creates the link to it.  When you click the link, it will run a mini slide show that consists of the target slide(s).
1.  Select some text that you'd like to make into a hyperlink, for example, "Please see the Detailed Analysis".
2.  Press Ctrl-K (or right-click on the selection and choose Hyperlink).
3.  Click "Place in this Document" on the left.
4.  In the scroll box that opens, scroll down to find "Detailed Analysis" (or whatever) under Custom Shows and click it.  (If you see only Custom Shows, click the plus sign to the left of it).
5.  Click the "Show and return" checkbox.
6. OK your way out.

The really neat thing is that the "link" persists even if slides are added before the target slide.  This is an awesome improvement over the conventional bookmark mechanism in which you're limited to linking to a slide number.  On the other hand, the hyperlink works only during a Slide Show.  There is no option to right click and open hyperlink.  So if you like to present in “edit mode” it won't work.

Also, navigation is a bit wonky.  When you click on the link, PPT will show the target slide, as expected.  But Alt-Left (the universal keystroke for "Back") doesn't return you to the previous slide.  In order to go back, you have to tell PPT to advance to the next slide (press PgDn, for example) to get back to the previous slide.  Pressing PgUp will cause PPT to ring the error bell at you.  Don't worry.  After a few dings, you'll get the hang of it.

Even if you don't need to set Hyperlink Base, you might prefer this method over the simpler method of linking to a slide.  Because if that slide's position changes, the simple link won't work the way you intended.

Let me know if you find this useful.  And if anyone knows this “GadiN” who reported the problem, please thank him or her for me and share this method.

Is AntiVirus as Bad as Malware?

I've been using and recommending AVG AntiVirus Free for over a decade.  Whenever a new computer's trial version of Symantec AntiVirus would expire, I'd uninstall it and install AVG.

But when I accepted the upgrade to version 16, I lost the ability to control its update schedule.  It's no longer possible to disable automatic updates.  Now the only option is to accept automatic updates whenever they become available.  Users even lost the ability to force an update.

That's a big problem.  I once hosed a system when AVG updated during a Windows Update.  That was when I decided to disable automatic updates and run them manually with a simple right click of the taskbar icon.

I still have the install file for AVG 15.  Unfortunately, it downloads the full installer from the Internet and then installs the latest version.

I considered doing away with AntiVirus entirely.  But I'm not the only user of the computers -- the other users will need to have the safety net of an AntiVirus program.

So it's time to shop for a new AntiVirus program!

Adventures in Email Debugging

On Wednesday I was unable to download my personal email on my work computer.  Since downloading personal email at work is something I probably shouldn't be doing anyway, I didn't look into the matter right away.

And besides, my set up is a bit complicated, actually.  The email client is Outlook, which is used in IMAP mode to work with the Exchange email server.  But I also use Outlook to connect to Yahoo's POP and SMTP servers for my personal mail.  But not directly.

Outlook is configured to connect to Firetrust's Benign (0), which is set up to listen on a local IP address.  And Benign is configured to connect to POPFile (1).  So any one of those three programs could be at fault, or the Yahoo POP server could be down, or the IT department might've blocked the necessary ports.

I figured I'd restart my computer eventually.  But I was running a long script, and I didn't want to do a restart at that time.  So I first I stopped and restarted POPFile and then Benign.  And then Outlook.

When that didn't help, I fired up Firetrust's Mailwasher Pro (2), just to see if I could connect to the Yahoo POP server and download message headers.  I could.  So at least the POP port 995 (for SSL) was still open on the router.

Then I set up a new Outlook email account profile that would connect directly with the Yahoo mail server.  That worked, too.  So Outlook was still okay.  That meant Benign or POPFile was at fault.

I decided to check out the POPFile website first.  And there I read in the news listing, “The Windows version of POPFile 1.1.3 is no longer compatible with some SSL servers...”  True it was dated back to September 2015, but if Yahoo just upgraded its server, then it would make sense.

I downloaded and installed the SSL updater that POPFile provided (3) and got it all working again in a jiffy.

So if anyone else out there is using POPFile as an intermediary between Yahoo and his or her email client, be prepared to use this technique to get it working again.

(0) http://www.firetrust.com/products/benign
(1) http://getpopfile.org/
(2) http://www.firetrust.com/products/mailwasher-pro
(3) http://getpopfile.org/downloads/updateSSL-mk2-for-POPFile-1.1.3.zip

On the Benefits of Social Media

Today's Soup to Nutz comic strip by Rick Stromoski is definitely blog worthy...
http://www.gocomics.com/soup-to-nutz/2017/01/14

If you can't follow the link, it's simply that Babs remarks to her brother Roy-boy,

I'm so glad I live in a time when there's social media...  In the old days it took weeks, even months...  before finding out that someone was an idiot."

Naturally I posted it to Facebook.