Visible Certainty - The Verifiable.com Blog

Video: CNN Leaves It There | The Daily Show | Comedy Central

Tue, 13 Oct 2009 13:54:00 -0400

Video: CNN Leaves It There | The Daily Show | Comedy Central:

The Daily Show (yeah, yeah, a comedy show) does a great job explaining why we built Verifiable.com This is 10+ minutes of brilliant humor that makes me want to tear my (non-existent) hair out. ”CNN has 20 to 35 to 70% more facts than the other networks.”

— Jason

Modifying context menus with Flash Player 10 while avoiding crashes under Windows XP

Mon, 28 Sep 2009 15:26:52 -0400

(This is a post on coding. It has little to do with charting, but we ran into this problem and thought someone else might too)

In our app, we like to modify the context menu (what you get when you right-click) to have context-useful items. Not exactly groundbreaking. This has worked great for months. But recently, we received a bug report from a Windows XP user. It seems that the most recent versions of Flash Player 10 (we haven’t yet isolated when this bug started appearing) contains a bug that can get tickled when modifying the menu. The symptom is simple: a user right clicks, selects an item from the menu, and their browser crashes. That’s right: the browser crashes. Woot!

We discovered the underlying cause, which as far as we can tell, shouldn’t cause problems (according to the Flex documentation), but it clearly does. The cause is assigning contextMenu more than once, or removing items from contextMenu.customItems.

Therefore, the fix is straightforward, if a little ugly(?): Assign contextMenu and populate customItems with all your potential context menu items only once. Disable or enable individual ContextMenuItems in contextMenu.customItems based on the context.

The rest is left as an exercise for the reader.

— Jason

Verifiable Pro: Private Content, Chart/Data Sharing, and More...

Tue, 15 Sep 2009 16:59:37 -0400

We’re proud to announce the release of our Verifiable Pro product today, allowing you to upload, visualize, and share private data with only those you wish.

We tried hard to get the details right. Namely:

You can specify whether all new content should be private by default, so you’ll never accidentally publish something you didn’t mean to.
You can share private charts two ways: either by adding any individual email address to the full, interactive chart’s user access list, or by giving out a unique, hidden URL. The former requires viewers to create a Verifiable.com account if they don’t already have one, but it ensures that only they can see the content, and that they have full access to the tool. The hidden URL makes it easy to share a private chart image with anyone, but you can’t stop them from passing along the link to others.
You have fine-grained control over whether others can simply view, copy, and/or edit your content, and you can authorize different operations for different people. You can even delegate to someone else the power to invite new people to see a given chart or data set.
Priority Support: we work very hard to deliver first-class support to all our users, but we’ll prioritize requests from our Pro customers, and guarantee our response time.
A no-fuss, no-muss, no-asterisk, no-exception, no-gimmicks satisfaction guarantee. If you’re not happy, and want your money back, you can ask for a full refund at any time.

The Pro product also includes a powerful new feature allowing you to use an existing chart as a template for a new chart with different data. No more need to repeat the same careful layout operations over and over again. Just “Copy -> Using Different Data” in the File menu of any visualization.

We want to take this opportunity to reiterate, however, that the basic Verifiable product isn’t going anywhere, and will continue to be improved, not crippled. We will always have a free service that encourages people to upload data and make public charts. That’s why we got into this business, and why we continue.

In that spirit, new this week for everyone is a host of updates, including:

auto-detection of column names when importing data
more powerful filtering (now located in the sidebar for easier access)
better binning: specify your own custom bin size for numeric variables
dozens of small bug fixes and tweaks to make everything smoother

A different form of verifiability? — Jason jeannr: I made...

Wed, 19 Aug 2009 07:54:00 -0400

A different form of verifiability?

— Jason

jeannr:

I made a flow chart, that we might better understand.

Once upon a time I was falling in love, and now I’m only falling apart.

In response to Foreign Policy: don't look at Africa in a vacuum.

Tue, 04 Aug 2009 13:13:00 -0400

Foreign Policy published an article, “Think Again: African’s Crisis”, in which Charles Kenny writes, “Africa has seen child mortality fall from 26.5 to 15 percent since 1960 and life expectancy increase by 10 years.”

Although this is true, it’s deeply misleading to look at Africa in isolation. In fact, compared to the rest of the world, Africa lags considerably on child mortality:

(If you click on the picture to open an interactive version you can play with the data yourself).

If the big story is that child mortality is declining due to new technologies and medicine then why is Africa lagging?

Given the degree of improvement in the rest of the world, these small gains in Africa are a crime, not a victory.

- Jason Y., Verifiable.com

Visualizations We're Watching

Mon, 03 Aug 2009 16:22:00 -0400

Robert Kosara just released a new version of his Parallel Sets visualization app, a focused proof of concept of one visualization type (most popularly seen here with Titanic mortality data). Anything he does is worth paying attention to, and this is no exception.

After talking with him at last month’s OECD conference, I know he’s eager to move beyond “the same old chart types we’ve seen for the past sixty years”, and to inspire new and better visualizations in end-user tools like ours. Take a look, and let us know if this is something you’d find useful on Verifiable.

- Peter C., Verifiable.com

auto-updated data feeds + data joining + calculated variables == power

Thu, 30 Jul 2009 17:41:00 -0400

Jason Y. whipped up a fascinating chart this afternoon that shows off the power of our new data feeds when combined with joined data and calculated variables.

He started with a data set of the monthly closing price of the S&P 500 Index. Then he joined that with another monthly data set of the 10-year Treasury yield. Then he created two new calculated variable series representing the monthly change in each (which is not explicit in the original data).

Finally, he plotted them together on the X and Y axes of a point chart, in the typical “four-quadrant” style, annotated some interesting outliers, and colored all the data by date.

Total time spent: 30 minutes.

It’s interesting data, well-visualized.

- Peter C., Verifiable.com

PS: He also made this small-multiple view of the same data.

New auto-update capability for data on Verifiable!

Thu, 30 Jul 2009 13:17:00 -0400

Starting today, a number of our public data sets will be automatically updated from the source, as new data is published. Examples include the rich unemployment data sets from the U.S. Bureau of Labor Statistics and the Consumer Price Index data. We’ll be adding more all the time, each one tagged with “auto-updated” for easy search. If you want time series data that we aren’t yet supporting, please let us know.

Best yet, preexisting visualizations incorporate this new data automatically. And in some cases, new old data—for example, we extended some of the unemployment data back to 1954.

In general we avoid modifying the appearance of existing visualizations whenever we add or change features, but in this case we made an exception since the additional data should be a major improvement. You also should’ve received an email from us warning you of any changes to your visualizations (please let us know if we missed you).

We encourage you all to log in and experiment with the latest data. And if you’d like to your visualization to show only a fixed period, you can simply add a filter restricting the dates you see, as before.

We hope you enjoy the new functionality. Please let us know if you have any questions or feedback!

-Peter C., Verifiable.com

New Feature: Custom Colors

Thu, 23 Jul 2009 10:59:00 -0400

One of our most requested features has been the ability to assign custom colors to chart variables.

Although our default color palate has been carefully chosen to allow the maximum number of distinct variables (a future blog post in and of itself), our assignments won’t always correspond to the specific colors naturally associated with certain data, or may not match your personal or organizational aesthetic.

To choose your own color for a data series, just right-click on any data point in the series and select “Choose Series Color”.

For example, our popular chart of Party Affiliation By Religious Tradition can now be improved by using traditional party colors (red and blue), with lighter shades for “leaning”. Below are before and after images (click through for the interactive charts).

Before:

After:

- Peter C., Verifiable.com

"IT MAKES SENSE TO BUY GOLD !"

Fri, 15 May 2009 15:27:00 -0400

Something as simple as visualizing the value of gold over the past two centuries highlights two important and frequently overlooked problems when charting price data.

Here’s a typical historical price chart, from an informational website advocating gold as an investment:

The first problem with this chart is inflation. Changes in a commodity’s price do not necessarily correspond to changes in its value over time, because of price inflation — so if you don’t convert into a constant unit of value, it makes no sense to chart prices over time (unless your goal is actually just to show inflation, using the commodity as an example).

If we take the same source data and adjust for inflation (how?), the graph looks like this:

The second problem, even with this second chart, is that we’re visualizing absolute changes in price, rather than relative changes in value. It doesn’t really matter to anyone whether gold jumps from $5 to $10/ounce, or from $500 to $1000/ounce (even if you’re using the same 2007 dollars in all cases): either way, gold doubled in value. But if you chart gold prices on a linear scale, the former doubling is virtually invisible and the latter looks gargantuan. You’re not visualizing what you’re trying to visualize, which is the relative change in the value of a single ounce of gold.

But if we take the inflation-adjusted chart, and switch to a logarithmic scale, the graph looks like this:

As you can see, it’s shows a totally different story than the original chart we found. (To see the original and final versions together, see here.)

Why doesn’t everyone do this when they chart this kind of thing? Assuming no malice, one issue is obviously ignorance. They don’t teach this stuff in schools (usually), and it requires a comfort level with simple math. But another reason is the sheer difficulty of making the necessary adjustments, in practice. You’ve got to find the inflation data set, hope it covers the entire time period you need (and uses the same units of time), manually combine it with your data (often by hand, by reformatting or retyping it), and then you’ve got to do the math to actually convert everything into inflation-adjusted figures.

Excel doesn’t just have a “adjust this line for inflation” feature, and it certainly doesn’t provide the inflation data you need to do it.

We do.

And we’re planning to make it even easier in the future. Let us know your thoughts.

- Peter C., Verifiable.com

Elsewhere on tumblr.... 20 years of Haitian Inequality

Thu, 30 Apr 2009 12:18:53 -0400

Elsewhere on tumblr.... 20 years of Haitian Inequality

Tax Freedom Day

Tue, 21 Apr 2009 11:05:00 -0400

We’ve posted a new featured chart this morning, showing the power of data set joins on Verifiable.com. After one of our users uploaded a great data set listing Tax Freedom Day for each state, I wondered what the implications of low taxes were for a state’s population. I searched for other data sets to join it with (by state), and in just a few minutes came up with this.

The Tax Foundation website appears to be down, which is too bad as I’d love to get the 2006 data to see if an apples-to-apples chart looks much different.

I’d love to see what else people can do with this data.

- Peter C., Verifiable.com

What's the best way to tell this story?

Mon, 20 Apr 2009 15:31:19 -0400

Over at Statistical Modeling, Causal Inference, and Social Science, there is a discussion of this chart:

There has been some criticisms of it, and I posted the following reply. I encourage you to check out the whole discussion. My reply:

As the author of the chart in question, it’s neat to read this discussion and criticism. (Someone cares!)

I chose a stacked bar chart after experimenting with a variety of visualizations and judging that this resulted in the densest, clearest picture. There may be better visualizations, and I’d love to see someone make one. (And if our tool can’t do it, that’ll certainly help us prioritize our development plans.)

Regarding the choice of colors, you’re of course completely right. It’s a limitation right now that you can’t select them — I felt lucky just to get Republican to map to red. I’d prefer to use the traditional party colors, with lighter shades for leaning. We hope to address this soon.

As for where to put “other”, I thought carefully about this. I think it would be misleading to place “other” in the middle of the right/left continuum, and likewise would be misleading to omit, so I chose to include it at the end. To the extent that some unknown subset of “other” belongs between Democrats and Republicans, I’m failing to portray that, but it’s a tradeoff I thought made sense.

You can copy & edit my chart to put “other” in the middle (just add & remove the Y variables until they’re in the right order). I don’t think it’s quite as clear, but YMMV. You could also argue that there should be _no_ sorting, on principle, since the right/left continuum is somewhat bogus to begin with — but while agreeing with the sentiment, I think in practice it makes the picture harder to see.

The remaining issues mostly stem from the source data (which comes from the New York Times, who in turn credit the Pew folks, but don’t link anywhere — which is a bummer). There’s obviously text that should have accompanied the data, given the unexplained asterisk. But you’ve got to make do with what you’ve got.

Other people have commented on the LDS/Mormon issue in the NYT chart comments, but there’s been no response from the Times. Also, I’m no expert, but aren’t there people outside LDS who identify as Mormon, even though they may be few in number? I could add more annotation to the chart to clarify the issue, but instead I’ve made a new version with LDS removed, since the issue is not central to the chart.

If anyone can point me to a better data set, however, or a public link to the primary survey data the NYT republished, I’d love to use it. In fact, comparing such data would be interesting in its own right.

Please keep the feedback coming — I’d really like to make the best possible visualizations, and a better visualization tool, so this is extraordinarily valuable.

Peter C.

Public Launch Of Verifiable.com!

Mon, 13 Apr 2009 14:24:00 -0400

Hi Everyone,

First, thanks for all your support through our public beta. We really appreciate everyone’s great suggestions. If we haven’t acted on them yet, then feel secure in knowing that they are on The List. On that note, check out our great new mechanism for voting on which changes you want next. Today’s release is the first step in what we hope will be an amazing tool.

So….here we go: Public Launch!

We just put the finishing touches on verifiable.com, v1. It’s a major leap from our prior site, featuring:

a clean, low-chartjunk philosophy — no shadows, no pie charts, no 3-D bar graphs, just the ink you need
access to the data behind a chart, even from within the chart
the ability to easily join multiple data sets (e.g., to compare your time series data set with another).
a growing repository of economic, social, and sports data sets, suitable for joining with your own data
easy, one-click small multiples (something that’s either unavailable or a PITA in every other web tool we know of)
access to extra chart dimensions using bubbles and shading to increase the density of your data
handy shortcuts like the ability to automatically identify and label outliers
fine-grained control of layout and appearance — our goal is to enable the creation of publishable graphics and interactive web embeds
a nifty new look (with help from Involution Studios)

Our goal, however, isn’t just to develop a great vizualization app — it’s to develop a tool to make it easier for you to communicate data more clearly, more efficiently, and more honestly with the viewer. We want to make it easy for you to make a gorgeous, verifiable vizualization, by allowing viewers to explore the data, manipulate the presentation, and even build on it to bring out richer, clearer, or just plain different conclusions.

One reason there are so few great presentations is because it’s always been so much work to create them. Our goal is to remove that excuse, so that the only reason a chart isn’t verifiable is because it wasn’t meant to be.

Anyway, that’s it for now,

Stuart, Jason, Kris, Jason, and Peter

ps: In the coming weeks we’ll be adding tons of features, including a “Pro” product. Stay tuned.

What a long strange trip it's been (part deux)

Mon, 13 Apr 2009 14:24:00 -0400

Hi Everyone,

The last time I blogged about what a long strange trip it’s been was July 23, 2008, when we launched the public beta. We have been working like crazy, not just on bugs and features, but on the whole production environment. You may have read our blog entry: Public launch of Verifiable.com! And wondered what is this thing you call Verifiable? :-)

Since everyone is pushing for transparency from our startup culture to our President, I guess I should do my bit as well…

So using my handy dandy time machine I will take you all back to the end of July last year:
1. we were visiblecertainty.com
2. running out of the amazon cloud.
3. We didn’t have joining, new variables, or small multiples.
4. We had an old 2001ish type layout and interface
5. We opened our public beta to see how all you folks liked it.

We discovered that:
1. No one could spell visible certainty with any consistency (“visual certainty?”, ”virtual certainty?”, “visable certainty?”). It would have been funny if it hadn’t driven us so crazy.
2. The amazon cloud is expensive and for our application somewhat underpowered (hey, don’t hate. we aren’t the only ones who say so)
3. People like joining, new variables, and small multiples
4. Engineers are not the best designers so we hired some experts
5. With the explosion of data availability and the many demands for transparency, our mission is becoming more and more vital
6. Our rich media UI is really really hard to test

So we decided to:
1. create a new product name:
Boy was this a long sweaty ordeal. Our spec was that we had to have an available domain (a dot com not a dot net or a dot whatever), needed to evoke the spirit of our corporate mission, and be somewhat whimsical. For your enjoyment or our humiliation here were the runner ups: dataink.com, cogister.com, datadunk.com, datasofa.com, datatribune.com, presentfully.com, charti.st, datasoup.com. I still have a deep fondness for datasofa and datasoup both of which were suggested by my 3 year old daughter who to my momentary irritation “interrupted” an adult conversation to tell me something crucial and timely. In the end, we decided to give up whimsy for a name that once we chose it, seemed to shout out to us to work harder and get this product done. On the other hand, I think my daughter is still slightly pissed that we didn’t go with her suggestions. Anybody out there want a slightly used domain name?

2. Move out of the cloud:
Boy was this a long tough grind (are you seeing a trend here?). Our spec was extensibility and expandability and not too much money. I’ve never had the deep felt need to live near the colocation facility. In my first business, the computer (a half-refrigerator sized SGI box- remember those) ran out of a closet near Mae-East (does that even exist anymore) in Vienna Virginia and I didn’t go visit it for years at a time. So we chose Wisconsin as the unlikely place to look for colocation. If you must know, we chose big company over small (feel a little bad about that) and HP over Sun (don’t feel bad at all about that). Then we embarked on the task of inventing a whole new infrastructure for creating and fielding instances (only Peter and JY could have done it with such aplomb).

3. Lot’s of flex coding:
I’ve started writing GUI code in the last 4 months, something I have avoided my whole professional life. And for the record, I have no idea how JD and Kris managed to write all this code. It is a wonder to me. And I am very thankful that I fought the rich media user interface for all those years, oh my god, is it a lot of lines of code and a lot of work and sooooooo hard to test. But, here we are and I think that these guys have really done a remarkable job.

4. Hire a design firm:
We hired (actually JY hired) a design firm to figure out how to make our site easy to use. I have always been a put a couple of buttons on the page and put clear labels on them kind of guy, but JY was so adamant that I figured he either had to be right or would bring a gun to work if I didn’t agree (admittedly when everyone works from home bringing a gun to work is a lot less of an issue for co-workers). But I love his wife and kids, so I agreed to the plan. Frankly, after seeing how great it looks, I must admit that I was an incredible jerk for dragging my feet. A shout out to Juhan Sonin (of MIT <—- yes aren’t we coool) for heading up the design work. The strangest part of the process was when the project manager of the design team introduced us to Juhan and it turned out that JY and he had gone to Lexington High School together. Of course, the design is just the first step and would have gathered dust without Kris and JY’s brilliant implementation.

One last thing: this company is filed with adults; adults with lots of pulls and pushes on their time. I want to remark at the impossibility of raising a young family, taking care of your relationship with a spouse, and doing this kind of work. If you get to work early you miss the kids eating breakfast, you miss walking them to school. If you stay at work late you miss having dinner with the family. This is all a little different if you work from home, but there is enough heartache there that it might even be harder. For instance, you are working on a particularly thorny coding issue and your small child walks in wanting to get in your lap and type a little. What do you say to that? Or, imagine that you have a spouse with a real job that comes home in the late afternoon expecting you to be available and present. Yes, it is impossible. Yet somehow all these guys have managed to do it and do it really well.

That’s it for now,
I have some press releases to write,
Some flex code to break,
Stuart

Playing With Numbers, Health Care Edition.

Fri, 20 Feb 2009 22:28:00 -0500

In the United States, a shocking number of people have no health insurance and poor access to health care. Many go bankrupt because of medical bills. We desperately need reform. What we don’t need, however, is bad charts.

For example:

There are some major problems with this chart.

It’s got two different units superimposed. On the left hand side, there is the number of “Uninsured Americans” and on the right, “Unemployment Rate.” This problem is exacerbated because the scales are chosen so that the two lines track with the same slope. If you notice, the range of values on the right is from 44 to 50 million. That’s a difference of 12%. The range on the right is 4% to 8%, or a difference of 100%. Additionally, the lines are smoothed (this is monthly data, but the lines smoothly move between each sample, implying data that doesn’t exist). If instead of charting these absolute numbers with smoothing, you charted how they changed over time, you’d get something like this:

Chart provided by Visible Certainty

That’s a lot less dramatic. This leads us to next problem: the data is a little fishy.

Notice, there are three sources, the Urban Institute, the Bureau of Labor Statistics (BLS), and the Census Bureau. The BLS is the source of the unemployment rate. There are well known flaws in BLS methodology (e.g., Wikipedia’s explanation), but that’s not my main beef. My main problem is that the uninsured figures are guesses. In fact, no one collects monthly figures for the uninsured (that’s a problem in it’s own right!). According to the report, they rely on an earlier study that found for every 0.1% increase in unemployment, 110,000 people lost health insurance. That may be true, but it’s only part of the picture.

Their figures assume a 1:1 correlation - for every 0.1% the unemployment rate changes, the number of insured changed by 110,000. The flaw with this is that there are other factors that determine insurance levels. A simple thought experiment: if unemployment went to 0%, that would mean, based on March 2007 figures, that 4.8 million people would gain insurance (44 * 110,000). But that would still leave 40 million people uninsured. Clearly something more complicated is going on here than just employment rates. After all, many employers don’t offer insurance. In fact, I suspect uninsured rates are even higher with the economic crisis because many employers are probably eliminating health benefits.

Now, clearly the massive increase in unemployment, which shows no sign of abating, is going to make the situation worse. But there’s a lot more to the story. For example, what if we looked at the change in aggregate national output (GDP) and the change in the uninsured? I could only get figures from the Census for 1999-2007. The results are a little shocking:

Chart provided by Visible Certainty

Now that is definitely an interesting story. The larger point is: make your point honestly and accurately. Don’t fudge the charts (with dual axes) or the numbers. And when it comes to the scandalous state of health care in this country, there’s no need to tell anything but the verifiable truth.

— Jason Y., Verifiable.com

Disk Permissions Database Read Failure in OS X 10.5.6 caused by corruption in /Library/Receipts folder

Fri, 20 Feb 2009 10:52:00 -0500

This blog entry is not for the feint of heart. You need to be the type of person who feels comfortable in Terminal and knows what rsync is. I DON’T REP and WARRANT that this will work on your machine. For all I know it might make things worse. I am ONLY reporting what worked on MY MACHINE. Having said that, I wish this blog entry had been around for me, it would have saved me a couple of days of thrashing…

My problems were caused by my boot disk quietly committing suicide by inches. I replaced the disk, but still got the following errors because that original disk clearly hated me…
By default, SuperDuper wants to fix disk permissions before every backup. So if you have had your SuperDuper backup fail with the following error, then this blog entry is for you.

| 09:30:26 PM | Info | …ACTION: Repairing permissions on RAID

| 09:30:26 PM | Info | ……COMMAND => Repairing permissions on RAID

| 09:30:28 PM | Info | Started verify/repair permissions on disk disk4 RAID

| 09:30:28 PM | Info | Error -9972: The underlying task reported failure on exit

| 09:30:28 PM | Info |

Finished verify/repair permissions on disk disk4 RAID

| 09:30:28 PM | Info | Error detected while verifying/repairing permissions on disk4 RAID: The underlying task reported failure on exit (-9972)

| 09:30:28 PM | Error | ****FAILED****: result=256 errno=22 (Unknown error: 0)

Or if you have run Disk Utility->Repair Disk Permissions and gotten the following error then this blog is for you.

Error | ****FAILED****: result=256 errno=22 (Unknown error: 0)

Or if you have run DiskWarrior->Repair Disk Permissions and gotten the following error then this blog is for you.

DiskWarrior error 2903, -8902

1. cd into your Time Machine Volume holding the snapshots of your machine.

On my machine:

cd /Volumes/Backup\ of\ Stuart/Backups.backupdb/Stuart

2. The following will compare (without doing anything) all the latest snapshots with your broken diskRemember to change 2009-02 to whatever YYYY-MM is current for you and /Volumes/SuperDuperBackup to your disk volume that is getting the permissions error.

On my machine:

for i in `ls | grep 2009-02`; do echo $i;rsync -avz —dry-run $i/Macintosh\ HD/Library/Receipts/ /Volumes/SuperDuperBackup/Library/Receipts/;done

You will get output like this:

2009-02-01-000135
building file list … done
./
boms/
db/
db/a.receiptdb
sent 237644 bytes received 572 bytes 52936.89 bytes/sec
total size is 451835534 speedup is 1896.75
2009-02-02-005839
building file list … done
./
boms/
db/
db/a.receiptdb
<…I had a bunch of snapshots here…>
2009-02-14-000056
building file list … done
./
boms/
db/
db/a.receiptdb
sent 237644 bytes received 572 bytes 95286.40 bytes/sec
total size is 451835534 speedup is 1896.75
2009-02-15-001812
building file list … done
sent 266223 bytes received 548 bytes 76220.29 bytes/sec
total size is 213404079 speedup is 799.95
2009-02-16-001758
building file list … done
sent 266223 bytes received 548 bytes 177847.33 bytes/sec
total size is 213404079 speedup is 799.95
2009-02-17-000302
building file list … done
sent 266223 bytes received 548 bytes 106708.40 bytes/sec
total size is 213404079 speedup is 799.95

3. Which informs us that the disk permissions database stuff got corrupted between snapshots:2009-02-14-000056 AND 2009-02-15-001812. Notice, that 2009-02-15-001812 doesn’t have any changes from the current (BROKEN) system. Ergo the snap shot right before it 2009-02-14-000056 is the MOST RECENT good permissions data base.

4. so to fix my system, I restored that directory from 2009-02-14-000056 with the following commands:

> cd /Volumes/Backup of Stuart/Backups.backupdb/Stuart/2009-02-14-000056/Macintosh HD/Library/
> sudo rsync -avz Receipts/ /Volumes/SuperDuperBackup/Library/Receipts/

Password:
building file list … done
./
boms/
db/
db/a.receiptdb
sent 27675899 bytes received 588 bytes 1085352.43 bytes/sec
total size is 451835534 speedup is 16.33

5. You run Disk Utility choose the offending Volume, use First Aid and click on “repair disk permissions” and life is happy again!!!! In my case, I got a bunch of repairs and warnings, but I no longer get that annoying failure “Reading the permissions database”!!!

Good Luck!

Stuart

Don't forget your squares.

Tue, 17 Feb 2009 17:22:00 -0500

recovery.gov launched today.

Its mission is to provide transparency into how the US government is spending the money from the stimulus act that President Obama signed on February 17, 2009.

They have a chart showing the big buckets of where the money is going:

(as of now, you can see it here)

The “Other” dot represents $8 billion. The area of it should therefore be 1/36 (288/8) of the “Tax Relief” dot. If you grab a pixel ruler, you’ll see that they did it right, ignoring rounding errors.

This is in marked contrast to JP Morgan, as chronicled by Robert Cringley. So this could be a good sign.

The site promises lots more data in the coming months, so we’ll be looking to put together interesting charts. In the meantime, here’s a more interactive version of the chart. If you go to the full size chart, you can play around (switch between a bar chart and a points chart, etc):

Chart provided by Visible Certainty

You're supposed to update these blog things "regularly" ?

Fri, 22 Aug 2008 14:59:00 -0400

Although our site is now public, there are a number of features we want to add in the coming weeks and months. Our aim is to produce a website that will make it easier for you to analyze, display, present and share your information. Over time, we plan to add a number of tools to guide you in exploring your data. Just to give you a sense of where we are headed, these are our rough development goals.

Near term upcoming features include:

speed (we continue to profile and hack the code to improve responsiveness)
summarizing and aggregation of information (e.g., if you add a time series with daily observations, group and summarize them by month)
improved data importing (so we do the heavy lifting of pivoting your tables, concatenating column headers that are spread across rows, etc. into a format we can more easily chart)
joining data sets together to produce new charts
data units (you upload dollars, we keep track of that and let you specify what kind of dollars, etc.)
updatable data sets (you can re-upload a data set or if you have imported a google spreadsheet, we will automatically pull in changes, etc.)

Medium term features include:

revamped user interface
small multiples (many charts, laid out together in space)
print-quality table layout (sometimes a table of numbers is the best way to make your point!)
more annotation tools (e.g., trend lines on charts)
handling geographic data (plot it on maps)
box plots

Longer term features include:

full presentation creation
social networking (more tools to share, discuss, rate and review data, charts and presentations)
magical fairy things of which we can not speak

Obviously, all of this is subject to change. And if you have a suggestion, please let us know.

Jason

Going Public Beta Today

Wed, 23 Jul 2008 11:59:00 -0400

Hi Everyone,

Visible Certainty is going public beta today. This has been a long trip and might be worth visiting the history of how we got here.

First Jason and I spent from 1998 to 2006 trying to get to a Tufte conference. We were fascinated by the premise of a better way to communicate data, but we were also pretty busy. The first time we talked about it we were busy working on the software for a company called Gamesville. Jason’s friend Peter had heard Tufte speak and was bowled over by it. He even produced a “Tufte Presentation” at one of his own conferences and got rave reviews about it. So we were intrigued. But, two companies later (GameLogic, QuantumFoam) and we were still procrastinating. Finally the planets aligned and Tufte was in Boston and we were between companies trying to decide what the new new new thing would be.

It was riveting. It was a moment of clarity. When Tufte told that story of Galileo discovering actual scientific evidence that the earth revolved around the sun and termed it “visible certainty” we were sold. Even better that the Catholic Church told him never use those words again. Better yet, Tufte was complaining that no one had ever put together the right tool to produce elegant, simple, data-dense presentations in a single package. A quick dotster search showed that visiblecertainty.com was available so for $9.95 we started a new company. Okay maybe a little more for computers, lawyers, office space, employees, software, and some other stuff but that initial $9.95 was the important thing.

Ahhh… but how to start.

You really have to do a lot of reading about stuff that is way above my pay grade.

First there were the four Tufte books. Then the Grammar of Graphics by Wilkinson for ideas for how to boil a software design out of a visualization. Then there are Cleveland’s Visualizing Data, and The Elements of Graphing Data, which is really about the guts of the statistical ideas behind the visualizations. And finally Stephen Few’s Show Me The Numbers. You may notice that when you choose chart types in our tool they are broken down into his groupings. Few really seems to have struck a wonderful balance between elegance and daily effectiveness.

Now what technology to use? Jason and I had become huge fans of Ruby, Rails, EC2, and S3 over the last year or so (but don’t get me started on some of the scaling issues we are running into or EC2 down time). So that was a place to start. We did our time with gruff, ImageMagick, rmagick, rgplot, R, and ChartDirector.

Thank god we hired Kris in October of that year because in early December after banging our heads on the code for another month or so, he mentioned that Flex 3 was supposed to have fixed the configuration, scaling, development environment, and language issues that made us all hate/fear it before. So Jason and I discussed it, decided it was best to pretend it was our idea, pushed off our December release and moved the chart and data set generation to Flash. JD came on in January just in time to be tortured by this decision. He managed to solve all the things that overwhelmed the rest of us (including changing a O(n^2) “feature” to O(1) in the off-the-shelf data transport code we were using). Then about two weeks ago, Peter (remember him from the beginning of this story) agreed to join up - after all he was REALLY to blame for the whole thing.

And sixteen months later, with employees in three time zones, here we are! At least we didn’t have to name it Visible Certainty 2008 to release ;-)

Next stop: data joins and small multiples!

Hope you all enjoy the tool,

Stuart