Finally, a Python workflow that actually works

I do all my exploratory data work in Jupyter Notebook. It’s an amazing mix of nicely-formatted text, syntax-highlighted code and clear outputs. I love it:

But it’s not the complete package. I often want to peek at the results of a calculation I’ve done, just to verify to myself that I know what I’m doing. Here’s a typical example:

I don’t really want to print out the results of this calculation. I just want to peek at it to verify that it makes sense.

I’ve finally found a dream way of giving myself both things at once. A nice-looking record of what I’ve done, with the ability to easily run chunks of code again and again, and a way to peek at data and outputs in a forgettable, scroll-off-the-top-of-the-screen way.

It involves starting a Jupyter Notebook, then connecting an iPython console to the same kernel. This means that the iPython console and the Jupyter Notebook are seeing the same Python instance: the same variables, the same packages, the same function definitions. Here’s that example again, but this time, the Notebook doesn’t report the result of the calculation, it just does it. I’m then peeking at the result in a neighbouring console window:

To get this to work, you just copy/paste the long string of numbers and letters after “Kernel started” in the Jupyter Notebook window (here it’s 551204fc-c49f….):

Then, in your favourite terminal, type:

jupyter console --existing 561204fc-c49f-44a7-abf2-902535573282

You’ll now have an iPython console running off the same kernel. Wicked.

GDP may partly be “based on randomly generated numbers”

GDP is easy to describe. It’s one of the reasons it is so popular as a measure of what a country “does” economically. You just add up everything that’s produced in a given year, and subtract all the stuff that went into making it. What’s left is the genuinely “new” stuff the economy produced that year. Easy.

Behind that simple description lies a tangle of thorny issues involving philosophy (what counts as a “thing”? what gives a thing value?), sociology (what value does education have? or a healthy society?), gender politics (why does childcare not count as having value?) and a host of other hard-to-answer qualitative issues.

But these things seem like minutiae in comparison to the big data question this poses: How do we know anything about the economy at all? Who is actually counting and recording all this?

This might seem like a silly question. Surely the government just “knows” what is happening in the economy, right? Don’t they have to know what’s being bought and sold in order to calculate tax bills?

Well, they do know a lot of that stuff for VAT purposes but, for now at least, that information is not made available to the people who calculate GDP. (This is set to partly change in 2018.) Companies are obliged to report their turnover and their profits but, for the calculation of GDP, these are not timely enough, don’t cover enough of the economy (especially small firms) and are not cross-checked with other sources. There is also no requirement for companies to submit this data in a format which could easily be aggregated into a single, huge database. Imagine going through thousands of company reports copy/pasting values out of PDF tables.

So the government instead sends out around 50,000 monthly surveys asking questions about what was produced, how much was bought, and where any profits made went (i.e. to wages, investment or shareholders.) These are called the Monthly Business Survey (MBS) and are the little-known secret of how the government knows what’s happening in the economy and, hence, how it calculates GDP.

The accuracy of GDP depends directly on the accuracy of thousands of self-reported surveys. Statistics 101 tells us that random mis-measurements should tend to cancel one another out once results are aggregated across surveys, especially when there are lots of people being surveyed. But systematic under/over-reporting must play a role in some way or another.

Predicting the direction of this systemic mis-reporting of financial data is difficult to do. Would a firm want to over-report its successes for boastful purposes, or to under-report, fearful of a knock on the door from the taxman?

There are also at least some business owners who appear not to take the process of completing the survey quite as seriously as they might.

One business owner was mightily disgruntled at the amount of data being asked for:

The form asks for details of total turnover, retail turnover, commodity breakdown of retail cover, expenditure, employment costs, energy costs, goods materials & services costs, taxes, duties & levies paid, value of stock held, capital expenditure …

To which another gleefully responded:

I had to do one, waste of time, didn’t try too hard with the answers. Scribbled some rubbish and throw it back at ’em.

There’s even some suggestion that the process of checking that the responses for sanity is not as water-tight as it might be:

I had one of these about 10 years ago, followed up by threats of fines.
I eventually filled it in with totally vague responses and notes along the lines of “this is an estimate based on randomly generated numbers” it appears that they can’t force you to give meaningful data.
Apparently this blatantly stupid information was completely acceptable and they went away 🙂

Ouch. Let’s hope not too many business owners are following this one’s lead…

BBC gender pay gap

The release of the BBC’s list of top-paid talent has quite unexpectedly focused on the difference between men’s pay and women’s pay.

But it’s hard to look at a list and see whether women are doing worse than men, or whether there are fewer women on the list. Clearly either of these is a bad show, but a bit of data viz should be able to make the picture clearer.

In the pictures below, men and women are ranked from most well-paid at the top, to least well-paid at the bottom. On the right, all men and women are ranked together

In a perfectly fair distribution the lines would all be flat. Any downward-sloping lines suggest that someone is doing worse overall than their position within their gender would suggest.

Claudia Winkleman is the best-paid woman at the BBC but only the 8th best-paid person overall. And as for the hardest-working person on TV, Laura Kuenssberg, she’s the 13th best-paid woman, but is outbid by Jools Holland, the 36th best-paid man.

Talk about a pay Squeeze.

SpareRoom is over. What’s next for Britain’s flatsharers?

There’s an air of arms race about flatsharing adverts of the kind hosted by sites like SpareRoom.

All individuals want the same thing: to find other individuals, either to move into the spare room in their shared house, or who have a spare room to move into.

And all agents and landlords want the same thing too: to attempt to act as middleman between those two sets of individuals, taking an barely-earned cut in the process. Agents in particular have nothing at all to gain from the kind of peer-to-peer flatsharing which SpareRoom et al. are supposed to excel at.

So what do these middlemen do? They jump onto the site themselves, flooding it with adverts of their own, drowning the individuals out in a sea of noise. This heavy spamming of supoosedly peer-to-peer sites is characterised by attention-seeking tactics of ever-increasing volume.

Look closely at the first four hits in a search for rental rooms in one particular North London postcode district:

These adverts are laden with capital letters, random punctuation marks, and shouted claims about the virtues of the room available: “//SUPER PRICE TODAY/” yells one. Not one of these top four results is posted by an individual looking for a flatmate.

When one site becomes flooded with this kind of spammy advert (Gumtree was first to fall prey to this), attention shifts to another site where, for a time, flatsharers meet and interact unimpeded. This continues until, inevitably, the middlemen get wind of this new platform, and the spamming begins.

But this time seems to be different. SpareRoom has been pretty much defunct now for at least 18 months and, as far as I can tell, no platform has arisen to take its place.

This absence of a usable service has driven individuals looking for flats or flatmates into the walled garden of Facebook. One potential flatsharer told me:

…all about word of mouth I reckon…SpareRoom was shit!

Flatsharing via Facebook is great if you’re a social media bigwig, and your many friends are all sharing flats too, but it’s unhelpful for people who’ve just arrived in town, or whose friends are mostly homeowners or shacked up in couples.

The move to Facebook has shown that there is a real appetite for a reliable, trustworthy way to fill a room going in a flat share. What everyone really wants is to fill their room with a friend, a friend-of-friend, or someone like-minded enough to ensure they’ll get on.

Overheads for running a site which hosts genuine adverts from genuine flatsharers come pretty close to zero, so crazy fees like those of SpareRoom are not going to be the answer to keeping a new site clean of spammers. There needs to be some kind of verification process where your Facebook friends can vouch for the genuine nature of your flatshare advert: is it what it claims to be? is the advert genuinely from a current housemate?

These are problems which have been solved over and over again in other domains. It’s indicative of the lack of renters’ collective bargaining power that the process of finding rooms and flatmates is so poorly served by the options available.

Are bad exams harder to mark than good ones?

Bored out of my brains marking a large pile (84) of exams, I decided to spice things up a little bit by timing how long it took me to mark each question. I happen to favour the style of marking where you mark every question 1, followed by every question 2 etc.

It seemed clear to me that the speed of marking should depend on the order the paper was marked in: as I become more and more familiar with the kind of weird stuff students write in response to my questions, and as I solidify in my mind how many marks I feel particular answers are worth, it makes sense that I should speed up.

I also wondered whether bad answers are more difficult to mark than good answers.

Looking at this, I can see I’m right about my first assumption:

The duration clearly decreases on average as the marking exercise goes on.

But what about the crucial question: is marking a bad exam answer harder than marking a good one?

The answer is a fairly resounding “no” (Specification 1). The low R² also makes you think there’s something more important going on here: namely handwriting: I’d guess that’s the big factor.

But like a good social scientist, I wasn’t happy with leaving it there. Maybe there was some kind of nonlinear effect: very, very bad answers are easy to mark (since there’s usually little or nothing written), so are very good ones.

Specification (2) shows that there’s not much evidence that this is the case. (in fact p is less than 0.1 for both marks and squared marks in (2), so in some social science contexts, I’d award myself a little star!)

You might think: this is the most boring result imaginable. Why is this worth a blog post?

Well, you’re right of course…. but the struggle against publication bias just claimed one small victory!

#Procrastinalysis

Does your econ department care about new ideas?

The Reteaching Economics network is a group of early-career economics teachers interested in moving the teaching of economics on from nonsense like this:

NAIRU – Not in fact a Pacific island nation. Image by Asacarny, distributed under a CC by 2.5 licence.

to something a little bit more resembling the actual world which students expected to be finding out about when they signed up for an Economics undergraduate^[1].

They are inspired by the incredible Rethinking Economics, an international student group whose founders have just written an excellent book on the “perils of leaving economics to the experts.” The energy and vision of this group of students makes the average econ department look like exactly the kind of left-behind-with-dust-gathering legacy institution it usually is.

The Reteaching Economics group publishes a list of its members online, so we can see which institutions are most likely to care about teaching new ideas. Here’s the breakdown, including only institutions with more than one member.

There is a fairly long list of lonely singleton Reteachers (full disclosure: I’m one of them) who are the distant outposts of these ideas in otherwise skeptical economics departments. They should be a source of sympathy, greetings cards and reassuring poems/songs/flashmobs.

If you study Economics at an institution other than these, why not make your teacher aware of the new world being created just beyond the walls of their department?

@ReteachEcon

@RethinkEcon

[1] This is excluding the not-inconsiderable minority of econ undergrads who signed up because they want a job at Goldman/KPMG/etc. as soon as possible, who would probably be best served by finding out as little about the real world as possible. go back ^

Does your department care about tax havens?

Oxfam published a press release yesterday containing an open letter to world leaders calling for them to “make significant moves towards ending the era of tax havens” which are “distorting the working of the global economy”.

This seems to me like a pretty important intervention, and it’s a rare opportunity for economists to use their variably-justified reputation as “people who know about the economy” to do something positive to fix the way the system works at a global level.

These kinds of coordinated interventions into the actual workings of the global economic system seem woefully infrequent to me. So when it does happen, I’m anxious to be a part of it in whatever tiny way I can. And the least a bottom-rung economist like me can do is choose to work for the institution which is pulling hardest in the right direction.

Oxfam published the full list of signatories to the letter, so I thought I’d do a teeny bit of analysis, to see which countries and which UK departments care most about bringing an end to the global tax haven system. Perhaps this will help any early-careerists choose an institution whose interests align with theirs: after all, there is more to judging a faculty than counting its number of peer-reviewed publications.

Spoiler alert: the number of signatories from Bristol, the university I’ve recently moved to be a part of? Zero. Bad times…

---

 

First, here’s the breakdown of signatories by country:

Italy leads the field by a long, long way. From a first glance, it looks like they are from a decent variety of institutions but I haven’t checked this properly. Future work someone? Italy is a known advocate for change in economics, being a big adopter of the CORE project which aims to transform how undergrads are taught economics. The UK comes a pleasing second, I look at this result in a bit more detail below, and it’s pleasing to see the USA being well represented too.

It’s worth pointing out here that not all signatories are equal: although France has only a paltry ten signatories, one is Olivier Blanchard and another is Thomas Piketty. These are both absolute giants of the economics world, and their contribution here, particularly Blanchard who is not exactly known as an iconoclast, is very significant indeed. (One might do an analysis by Google Scholar citation count instead, which would show up these differences.)

Now let’s look at which institutions those 50 UK signatories come from:

The top two, SOAS and Greenwich, are both already on my radar as being slightly more radical than the standard econ department. But this is not a list dominated by such agitators: the LSE is hardly known as an anti-establishment hotbed, and nor are Warwick or Oxford.

Big-name mover for change, Ha-Joon Chang, cuts a lonely figure as the sole signatory from Cambridge. I hope he at least has some researcher staff of his own to go punting with.

And, as I mentioned, zero signatories from Bristol.

#EndTaxHavens.

 

Economists should start doing themselves out of a job

The reason why teaching undergrads is the best job I’ve ever done is because interacting with intelligent, energetic people is not the once-in-a-while happy coincidence it is in most jobs, but it’s the central purpose of what you’re supposed to be doing.

Sure, there are the hours of marking, the jocks, the whingers (colleagues that is, not students), the disastrous classroom IT. But the main point and purpose of my being here is to come into regular contact with open-minded people, and try and draw them into interesting discussion. And to do this with no agenda other than to keep everyone involved entertained and intellectually stimulated. It’s like being at a daytime teetotallers’ high-brow cocktail party.

But some conversations are more interesting than others. And a conversation I was drawn into on my first day of teaching after a long, isolated Easter break left me in need of a strong drink and a sit down.

It involved a student of the most memorable Myers-Briggs category: TICL (Talkative; Intelligent; Cheeky; Lazy). TICLs are either a joy or a damnation depending on whether you like the student in question, whether they like you, and how you deal with the inevitable uproar they cause. My managing of this particular TICL has been an up-and-down affair, depending on our respective moods, but on this occasion they were in a particularly restive mode.

After some fairly meandering discussion of an arcane trade theory, the TICL interjected with “when are we going to get to the good stuff?” Foolishly drawn in, I said “This is the good stuff. We’re talking about the actual world here! Trade! You might wish you were back in this class when you’re doing statistical asymptotics in 2nd year econometrics!” To which they smartly replied “Yeah well, I might wish I’d studied something different.”

The basic complaint is that the student had imagined, when applying to do economics, they would learn about what makes the world tick. How do banks make their billions? Why are some countries so seemingly doomed to mishap? Just what is it about China that makes everyone want to talk about it? And come to think of it what ever happened to the US? Or Japan? Or Europe?  In other words, to come to understand the world. To see it as a set of explicable causes and effects, not just a random mishmash of historical happenstance. To know the proposed remedies for the ailments of the world, and why the remedies of the past were no good. To have a sense that somebody, somewhere knows what the heck we should DO!

And I couldn’t completely disagree with this student’s analysis of their undergraduate course. Where were the big ideas? Where were the theories that made all the seemingly insane stuff we see happening actually fall into place? Where was the sense that here, in the Economics department, was a team of geniuses plotting the salvation of global capitalism from its cannibalistic excesses?

But worse than my half-agreeing that we were teaching these undergraduates the wrong stuff was a deeper more gruesome realisation by far. What if there is no explanatory grand theory? What if there is no team of geniuses? Why are we still debating the reasons why people trade with each other, or why they don’t all gamble their savings away on the stock market? Or what governments should do when a recession hits?

It seems to me that 90% of economists are still stuck trying to understand the developed old world, forever peering into the black boxes of the Federal Reserve or poring over UK tax returns, and trying to guess how the various levers work, while studies of development, or the power of Google, or the end of manufacturing, or the dark web of global capital of which Panama is only the tip of the iceberg, are relegated to tiny departmental outposts or, horror of horrors, the social sciences.

Hopefully I have successfully convinced you by the above tirade that I was in a bit of a neg. And so I retreated to the only place I knew I was guaranteed to be undisturbed by man nor beast: the Economics staff common room, a place so lonely and ill-loved that even the old copies of the FT cling and heap together for company.

And so it was that I starting reading this Lucy Kellaway article about how young people are encouraged to think that their shiny graduate jobs at KPMG and Goldman Sachs are going to be fun, important and exciting, only belatedly to discover that they are working for a tax auditor and an investment bank respectively. Needlessly to say they’re pissed off, and feel betrayed:

When the penny drops like this, there are only two possible outcomes. Either you quit… or you silence your doubts and get sucked into the machine.

This seemed to sum up the situation facing my angry, disappointed student. But, of course, these are not the only options. There is a third choice.

You can seek out the parts of your job or course that you do like, and seek out the people you respect and who you would actively like to work and spend time with, then stick to them as tightly as you can and insist without shyness on being in their teams, on their projects or in their classes. And while you’re there learn the ways of the organisation you’re a part of.

And by doing this you can come to know the enemy you seek to undermine. You need to be fluent in the arguments you’re hoping to counter, and to understand the culture you’re so frustrated by before you can become an agent for change within that culture.

So here’s my manifesto: I want to become an economist so I can change what becoming an economist entails, and what being an economist is like. I want to work in a supportive, collaborative department where the common goal of understanding the economy, and hence the world, is palpable all around me. And I want economics to at least start to do itself out of a job, rather than creating the vacancies for a thousand more economists as we did in the years before 2008.

Or, to put it better, in the words of good old JM Keynes:

The day is not far off when the economic problem will take the back seat where it belongs, and the arena of the heart and the head will be occupied or reoccupied, by our real problems – the problems of life and of human relations, of creation and behaviour and religion.

How to make an economist

I’ve often asked myself, in self doubting moments and imposter-syndrome-rich night sweat events, what the difference is really between a person who says they are an economist and, well, just a person. Can I really lay any claim to be something other than the averagely well-informed news media-consuming citizen?

Certainly a lot of what I covered in my MSc was hard to learn but, in terms of sheer insight and understanding of how the world works, I usually think that you can explain all of economics to an attentive listener during one good session in the pub.

But if there is a difference between an economist and our hypothetical well-informed media consumer, it’s perhaps this: it’s an economist’s actual business to read massive, threatening-looking books about the economy. I won’t understand the content of such books differently (or, crucially, retain anything any better) than the average reader. It’s just that I’ll actually take the time to do the reading in the first place rather than tackling, say, an account of the British protectorates in the Middle East or a tome on Cubism. You’ve only got a certain number of reading hours in your life (particularly if you watch as much John Oliver on YouTube as I do) and what you choose to dedicate those hours to ends up shaping the adjectives you feel comfortable using to describe your actual self: I am an economist.

It is in this spirit that I begin what looks like your archetype of the no-fucking-around serious and weighty tome on economics currently threatening to destabilise the centre of gravity of my pretty low-slung coffee table:

Holy shit it’s big. And it’s serious looking. The unbelievably old-fashioned twisted flax motifs bracketing the title are the sleeve-designer’s equivalent of a bow tie on a 35-year-old debate show host: they’re designed to fill you with awe at the seriousness of what’s being presented.

But so far, and it’s very VERY early days by the way, I’m impressed and excited by the tone that’s been set and the direction it seems to be heading. What I hadn’t realised is that the whole thing’s a summary of a load of work Picketty and assorted colleagues did to gather as much historical data as they possibly could about wealth and income (which not the same thing) in as many countries as they could, going back as far in history as they could. The introduction gives some details about the monster database they’ve built, and then the rest of the book is going to be all about the things they’ve actually managed to find out, by looking at this data and asking questions of it.

This is exactly what my whole philosophy of data-driven economics was aiming to be about, although I never actually managed to articulate it in any coherent way. But it’s basically about trying to think of economics as the social science it is, wanting to imitate anthropology or political science, instead of physics as the discipline’s been doing for 120 years. And that involves careful observation of the thing in question, in our case the economy, and conclusions drawn on the basis of those observations.

It also has given me a big grin in the form a wicked (in both senses) quote which sums up perfectly exactly how I feel about the economics establishment I’ve recently become a card-carrying part of:

To put it bluntly, the discipline of economics has yet to get over its childish passion for mathematics and for purely theoretical and often highly ideological speculation, at the expense of historical research and collaboration with the other social sciences. Economists are all too often preoccupied with petty mathematical problems of interest only to themselves.

Boom! It’s reading statements like this that not only make me feel more justified in calling myself an economist, but that actually make me proud to assign myself the title.

The new “improved” UN job application website

Anyone who has every applied for a job at the UN will feel a sweat break out on the back of their neck at the mention of the 90’s-style faux-Latin-named job application site “Inspira”. (Remember when things were all called “Exceptimus” and “Ignitor” and stuff? Ugh.)

Anyway, the site’s been down for a few days for modernisation, and the new improved site came back up on Friday. (They had a FOUR DAY outage! This alone is enough to make me suspect that the people who designed the website didn’t know what they were doing. Outages should measured in minutes or hours, not days.)

Given my experience of large organisations and their ability to procure IT which isn’t grossly overpriced and absolutely terrible (for which see Britain’s NHS, Barclays’ several iterations of terrible online banking websites, any government department in the world ever) it’s no surprise to see that the fundamentally broken Inspira has been given little more than a lick of paint.

Still present are the absence of a “Back” functionality, because the site doesn’t use URLs to control location, visible meaningless Javascript all over the place (hovering over a link to my job opening reveals “javascript:submitAction_win4(document.win4, ‘HRS_CE_WRK_HRS_JOB_LINK$21$$0′” which gives just a hint of the horrors people meeting the website’s code for the first time must experience), Windows 98-style icons and the overall user-experience and look and feel of being dragged by chain up a muddy slope by a slow moving tractor.

But my personal favourite is the built-in spell-check function. (Note to developers: people don’t develop their own spell-check functions any more. Browsers do this now.) My entire cover letter passed the spell-check with flying colours bar one exception: It suggests I replace the word “in” in the opening sentence with the admittedly more emphatic, but perhaps overzealous “IN”.

I’m not even joking.