Serious quantum computers are finally here. What are we going to do with them?

www.technologyreview.com - Comments

Inside a small laboratory in lush countryside about 50 miles north of New York City, an elaborate tangle of tubes and electronics dangles from the ceiling. This mess of equipment is a computer. Not just any computer, but one on the verge of passing what may, perhaps, go down as one of the most important milestones in the history of the field.

Quantum computers promise to run calculations far beyond the reach of any conventional supercomputer. They might revolutionize the discovery of new materials by making it possible to simulate the behavior of matter down to the atomic level. Or they could upend cryptography and security by cracking otherwise invincible codes. There is even hope they will supercharge artificial intelligence by crunching through data more efficiently. 

Yet only now, after decades of gradual progress, are researchers finally close to building quantum computers powerful enough to do things that conventional computers cannot. It’s a landmark somewhat theatrically dubbed “quantum supremacy.” Google has been leading the charge toward this milestone, while Intel and Microsoft also have significant quantum efforts. And then there are well-funded startups including Rigetti Computing, IonQ, and Quantum Circuits.

ma18cover_0.png?sw=180

This story is part of our March/April 2018 Issue

See the rest of the issue
Subscribe

“Nature is quantum, goddamn it! So if we want to simulate it, we need a quantum computer.”

No other contender can match IBM’s pedigree in this area, though. Starting 50 years ago, the company produced advances in materials science that laid the foundations for the computer revolution. Which is why, last October, I found myself at IBM’s Thomas J. Watson Research Center to try to answer these questions: What, if anything, will a quantum computer be good for? And can a practical, reliable one even be built?

Why we think we need a quantum computer

The research center, located in Yorktown Heights, looks a bit like a flying saucer as imagined in 1961. It was designed by the neo-futurist architect Eero Saarinen and built during IBM’s heyday as a maker of large mainframe business machines. IBM was the world’s largest computer company, and within a decade of the research center’s construction it had become the world’s fifth-largest company of any kind, just behind Ford and General Electric.

While the hallways of the building look out onto the countryside, the design is such that none of the offices inside have any windows. It was in one of these cloistered rooms that I met Charles Bennett. Now in his 70s, he has large white sideburns, wears black socks with sandals, and even sports a pocket protector with pens in it. Surrounded by old computer monitors, chemistry models, and, curiously, a small disco ball, he recalled the birth of quantum computing as if it were yesterday.

ma18-quantum2.png?sw=600&cx=2&cy=0&cw=11 Charles Bennett of IBM Research is one of the founding fathers of quantum information theory. His work at IBM helped create a theoretical foundation for quantum computing.

bartek sadowski

When Bennett joined IBM in 1972, quantum physics was already half a century old, but computing still relied on classical physics and the mathematical theory of information that Claude Shannon had developed at MIT in the 1950s. It was Shannon who defined the quantity of information in terms of the number of “bits” (a term he popularized but did not coin) required to store it. Those bits, the 0s and 1s of binary code, are the basis of all conventional computing.

A year after arriving at Yorktown Heights, Bennett helped lay the foundation for a quantum information theory that would challenge all that. It relies on exploiting the peculiar behavior of objects at the atomic scale. At that size, a particle can exist “superposed” in many states (e.g., many different positions) at once. Two particles can also exhibit “entanglement,” so that changing the state of one may instantaneously affect the other.

Bennett and others realized that some kinds of computations that are exponentially time consuming, or even impossible, could be efficiently performed with the help of quantum phenomena. A quantum computer would store information in quantum bits, or qubits. Qubits can exist in superpositions of 1 and 0, and entanglement and a trick called interference can be used to find the solution to a computation over an exponentially large number of states. It’s annoyingly hard to compare quantum and classical computers, but roughly speaking, a quantum computer with just a few hundred qubits would be able to perform more calculations simultaneously than there are atoms in the known universe.

In the summer of 1981, IBM and MIT organized a landmark event called the First Conference on the Physics of Computation. It took place at Endicott House, a French-style mansion not far from the MIT campus.

In a photo that Bennett took during the conference, several of the most influential figures from the history of computing and quantum physics can be seen on the lawn, including Konrad Zuse, who developed the first programmable computer, and Richard Feynman, an important contributor to quantum theory. Feynman gave the conference’s keynote speech, in which he raised the idea of computing using quantum effects. “The biggest boost quantum information theory got was from Feynman,” Bennett told me. “He said, ‘Nature is quantum, goddamn it! So if we want to simulate it, we need a quantum computer.’”

IBM’s quantum computer—one of the most promising in existence—is located just down the hall from Bennett’s office. The machine is designed to create and manipulate the essential element in a quantum computer: the qubits that store information.

ma18-quantum4.png?sw=1180&cx=0&cy=0&cw=1 This lab at IBM houses quantum machines connected to the cloud.

jeremy liebman

The gap between the dream and the reality


Th
e IBM machine exploits quantum phenomena that occur in superconducting materials. For instance, sometimes current will flow clockwise and counterclockwise at the same time. IBM’s computer uses  superconducting circuits in which two distinct electromagnetic energy states make up a qubit.

The superconducting approach has key advantages. The hardware can be made using well--established manufacturing methods, and a conventional computer can be used to control the system. The qubits in a superconducting circuit are also easier to manipulate and less delicate than individual photons or ions.

Inside IBM’s quantum lab, engineers are working on a version of the computer with 50 qubits. You can run a simulation of a simple quantum computer on a normal computer, but at around 50 qubits it becomes nearly impossible. That means IBM is theoretically approaching the point where a quantum computer can solve problems a classical computer cannot: in other words, quantum supremacy.

But as IBM’s researchers will tell you, quantum supremacy is an elusive concept. You would need all 50 qubits to work perfectly, when in reality quantum computers are beset by errors that need to be corrected for. It is also devilishly difficult to maintain qubits for any length of time; they tend to “decohere,” or lose their delicate quantum nature, much as a smoke ring breaks up at the slightest air current. And the more qubits, the harder both challenges become.

“If you had 50 or 100 qubits and they really worked well enough, and were fully error-corrected—you could do unfathomable calculations that can’t be replicated on any classical machine, now or ever,” says Robert Schoelkopf, a Yale professor and founder of a company called Quantum Circuits. “The flip side to quantum computing is that there are exponential ways for it to go wrong.”

ma18-quantum3.png?sw=600&cx=1&cy=0&cw=11 The chips inside IBM's quantum computer (at bottom) are cooled to 15 millikelvin.

jeremy liebman

Another reason for caution is that it isn’t obvious how useful even a perfectly functioning quantum computer would be. It doesn’t simply speed up any task you throw at it; in fact, for many calculations, it would actually be slower than classical machines. Only a handful of algorithms have so far been devised where a quantum computer would clearly have an edge. And even for those, that edge might be short-lived. The most famous quantum algorithm, developed by Peter Shor at MIT, is for finding the prime factors of an integer. Many common cryptographic schemes rely on the fact that this is hard for a conventional computer to do. But cryptography could adapt, creating new kinds of codes that don’t rely on factorization.

“The thing driving the hype is the realization that quantum computing is actually real. It is no longer a physicist’s dream—it is an engineer’s nightmare.”

This is why, even as they near the 50-qubit milestone, IBM’s own researchers are keen to dispel the hype around it. At a table in the hallway that looks out onto the lush lawn outside, I encountered Jay Gambetta, a tall, easygoing Australian who researches quantum algorithms and potential applications for IBM’s hardware. “We’re at this unique stage,” he said, choosing his words with care. “We have this device that is more complicated than you can simulate on a classical computer, but it’s not yet controllable to the precision that you could do the algorithms you know how to do.”

What gives the IBMers hope is that even an imperfect quantum computer might still be a useful one.

Gambetta and other researchers have zeroed in on an application that Feynman envisioned back in 1981. Chemical reactions and the properties of materials are determined by the interactions between atoms and molecules. Those interactions are governed by quantum phenomena. A quantum computer can—at least in theory—model those in a way a conventional one cannot.

Last year, Gambetta and colleagues at IBM used a seven-qubit machine to simulate the precise structure of beryllium hydride. At just three atoms, it is the most complex molecule ever modeled with a quantum system. Ultimately, researchers might use quantum computers to design more efficient solar cells, more effective drugs, or catalysts that turn sunlight into clean fuels.

Those goals are a long way off. But, Gambetta says, it may be possible to get valuable results from an error-prone quantum machine paired with a classical computer.

From a physicist’s dream to an engineer’s nightmare


“The thing driving the hype is the realization that quantum computing is actually real,” says Isaac Chuang, a lean, soft-spoken MIT professor. “It is no longer a physicist’s dream—it is an engineer’s nightmare.”

Chuang led the development of some of the earliest quantum computers, working at IBM in Almaden, California, during the late 1990s and early 2000s. Though he is no longer working on them, he thinks we are at the beginning of something very big—that quantum computing will eventually even play a role in artificial intelligence.

But he also suspects that the revolution will not really begin until a new generation of students and hackers get to play with practical machines. Quantum computers require not just different programming languages but a fundamentally different way of thinking about what programming is. As Gambetta puts it: “We don’t really know what the equivalent of ‘Hello, world’ is on a quantum computer.”

We are beginning to find out. In 2016 IBM connected a small quantum computer to the cloud. Using a programming tool kit called QISKit, you can run simple programs on it; thousands of people, from academic researchers to schoolkids, have built QISKit programs that run basic quantum algorithms. Now Google and other companies are also putting their nascent quantum computers online. You can’t do much with them, but at least they give people outside the leading labs a taste of what may be coming.

The startup community is also getting excited. A short while after seeing IBM’s quantum computer, I went to the University of Toronto’s business school to sit in on a pitch competition for quantum startups. Teams of entrepreneurs nervously got up and presented their ideas to a group of professors and investors. One company hoped to use quantum computers to model the financial markets. Another planned to have them design new proteins. Yet another wanted to build more advanced AI systems. What went unacknowledged in the room was that each team was proposing a business built on a technology so revolutionary that it barely exists. Few seemed daunted by that fact.

This enthusiasm could sour if the first quantum computers are slow to find a practical use. The best guess from those who truly know the difficulties—people like Bennett and Chuang—is that the first useful machines are still several years away. And that’s assuming the problem of managing and manipulating a large collection of qubits won’t ultimately prove intractable.

Still, the experts hold out hope. When I asked him what the world might be like when my two-year-old son grows up, Chuang, who learned to use computers by playing with microchips, responded with a grin. “Maybe your kid will have a kit for building a quantum computer,” he said.

The Muse (YC W12) Is Hiring a QA Lead – Help People Find Their Purpose

www.themuse.com - Comments

The Muse strives to make work more human by being a trusted resource for millions of people as they seek career satisfaction—not just another job. Companies partner with The Muse as they look to attract and retain the best talent by living an authentic and compelling employer story. Our mission is to create meaningful connections between companies and candidates to make the world of work—from the job search to career development—more personal.

There’s a lot of cool technology behind our products, and our engineering team is looking for a QA Lead to help build our QA practice from the ground up, fostering a culture where quality is embedded from the very beginning of the development process.

This role reports to our VP of Engineering.

How you’ll make an impact

  • You’ll own and drive the overall quality of The Muse codebase and development process, aiming to constantly improve performance and user experience
  • You’ll work with the engineering and leadership teams to define our QA strategy and develop a roadmap to support that strategy
  • You’ll work with our Tech Leads and Product Managers to better understand our product, building a testing environment that matches our needs and ensures high-quality releases
  • You’ll develop and execute test plans for identifying issues before they ship, and work with our engineering team to instill a culture of quality in development
  • You’ll help scale our QA team as we grow

Why We’ll Love You

  • You have extensive knowledge of major QA tools, bug-tracking systems, and automation tools
  • You know what it takes to build a QA team from ground up—you can recommend the right tools to use, are familiar with QA best practices, and know what to focus on first; Ideally you’ve done this before
  • You’re excited to teach and will take the time to share best practices for writing quality code and creating unit tests along the way
  • You’ll take the time to gain in-depth knowledge of every step of our development process, not just the end
  • You’re creative and know how to get more done with fewer resources
  • You’re comfortable being the only QA engineer for a while, owning both strategy and execution
  • You’re used to agile environments and know how to respectfully push back on things you think should not be released and will take the time to explain why
  • Bonus points if you’ve been in a management or QA tech lead role
  • Even better if you have development experience and can look at the code to provide more precise feedback to developers

Why You’ll Love Us

  • We’re fanatical about finding the right tool for each job. We pick technologies because they make sense, and we’re not afraid to try something new when we feel that it could be useful, even if we’ve never used it before.
  • Our team works on problems, not tasks. We like to help other teams turn their needs into great technology, and we give each engineer the freedom to tackle the challenges that they find most interesting.
  • We move fast, with multiple releases every day and an iterative approach to developing new features and measuring their success.
  • You’ll work at a tech company founded by two badass women—Our founders believe transparency is important so they really try to share as much as they can about changes to The Muse strategy, board meetings, and when they are wrestling with big company-wide decisions.
  • The Muse actually has—and sticks to—a “no assholes” policy, so you can come to work everyday knowing you will always be surrounded by good people who genuinely care about you.
  • We invest in growing our people—personally and professionally
  • We offer unlimited vacation—and we mean it!

Our Tech

In an industry full of wordpress installs, our tech stack stands above the rest. We've created a custom web application that builds on:

  • Python 3 and Go on the backend
  • Typescript and react on the frontend
  • PostgreSQL, Elasticsearch and Redis for data management
  • Node.js and Ruby on Rails for various tools and services

At The Muse, we believe that great ideas come from anywhere. We support a collaborative environment and value open participation from individuals with different ideas, experiences, and perspectives. We believe having a diverse team makes The Muse a more interesting and innovative place to work, and we strive every day to make The Muse a welcoming and inclusive place for all.

If this could be your dream job, please submit a cover letter and resume, so we can get to know you a little better.

The Northpole is warmer than Europe this weekend

twitter.com - Comments

Bienvenue chez vous !

Ce fil contenant les dernières actualités qui vous intéressent est l'endroit où vous passerez la majeure partie de votre temps.

Les Tweets ne fonctionnent pas pour vous ?

Survolez l'image de profil et cliquez le bouton Suivre pour vous désabonner de n'importe quel compte.

Faites passer le mot

La façon la plus rapide de partager le Tweet d'une autre personne avec vos abonnés se fait avec un Retweet. Appuyez sur l'icone pour l'envoyer instantanément.

Rejoignez la conversation

Ajoutez vos réflexions à n'importe quel Tweet avec une Réponse. Trouvez un sujet qui vous passionne et entrez dans la conversation.

Obtenez davantage de ce que vous adorez

Suivez plus de comptes pour accéder aux dernières actualités qui vous intéressent.

Ne manquez jamais un Moment

Rattrapez l'actualité avec les meilleures histoires qui ont lieu en même temps qu'elles se dévoilent.

Gobee.bike pulls out of France due to mass destruction of its bike fleet

www.theguardian.com - Comments

People wanting to get on their bikes in France have one option less to do so after the Gobee.bike hire service closed for business after a string of thefts and vandalism.

“It is with great sadness that we are officially announcing to our community the termination of Gobee.bike service in France on 24 February 2018,” the Hong Kong-based service said.

“Over the months of December and January, the mass destruction of our fleet has become the new entertainment of underaged individuals,” said Gobee.bike, which had rolled out 2,000 bikes in Paris alone and had 150,000 users across France.

The company said more than a thousand bikes had been stolen and almost 3,400 damaged nationwide, with almost 300 complaints filed to police and 6,500 repairs needed.

Gobee.bike had already thrown in the towel in the northern cities of Lille and Reims as well as Belgian capital Brussels for the same reason.

The bright green bicycles were located via an app and hired for 50 cents an hour by swiping a barcode to open the safety lock.

The user, who paid a €15 (£13) deposit, could then leave the bike anywhere, unlocked.

Gobee.bike said: “It was sad and disappointing to realise that a few individuals could ruin such a beautiful and promising project. We had to come to the conclusion that it could not be viable and there was no other choice for us than shutting down, nationwide.” 

Gobee.bike and other rivals such as Ofo or Obike had hoped to bag market share after Paris authorities in January threatened sanctions against the new operator of the city’s better known Velib system, over a chaotic rollout under a new contractor that left 300,000 users seething.

The sturdy grey bikes are a familiar sight in the French capital, popular with tourists and commuters alike a decade after the scheme was launched.

But a redesign and partial switchover to electric versions, plus a change of management from French giant JCDecaux to Franco-Spanish outfit Smovengo proved messy.

After weeks of disruption, just 64 docking stations were operational by mid-January out of 1,460 supposed to be up and running by April. Many riders complained the mobile app regularly crashed while calls to customer service went unanswered.

Smovengo blamed the delays on electrical problems and a legal dispute with JCDecaux. 

Dropbox saved almost $75M over two years by moving out of AWS

www.geekwire.com - Comments

A typical data center setup. (Courtesy: Wikipedia)

After making the decision to roll its own infrastructure and reduce its dependence on Amazon Web Services, Dropbox reduced its operating costs by $74.6 million over the next two years, the company said in its S-1 statement Friday.

Starting in 2015, Dropbox began to move users of its file-storage service away from AWS’s S3 storage service and onto its own custom-designed infrastructure and software, and the cost benefits were immediate. From 2015 to 2016, Dropbox saved $39.5 million in the cost of revenue bucket thanks to the project, which reduced spending on “our third-party datacenter service provider” by $92.5 million offset by increased expenses of $53 million for its own data centers. The following year in 2017, it saved an additional $35.1 million in operating costs beyond the 2016 numbers.

Dropbox makes it official, files to go public later this year with $1B in 2017 revenue

Dropbox was once the quintessential cloud success story, a startup that built a massive user base and brand presence thanks to its use of Amazon Web Services. Obviously, lots of companies are still happy to pay AWS to manage their infrastructure, as evidenced by the steady gains the cloud market leader made in 2017.

But once certain startups turn into big companies with hundreds of millions of users, with computing needs that they’ve come to intimately understand, it can be far more efficient to set up computing infrastructure designed exactly with those needs in mind. After a multiyear process, Dropbox completed what it calls its “Infrastructure Optimization” project in the fourth quarter of 2016.

“Our Infrastructure Optimization reduced unit costs and helped limit capital expenditures and associated depreciation. Combined with the concurrent increase in our base of paying users, we experienced a reduction in our cost of revenue, an increase in our gross margins, and an improvement in our free cash flow in the periods presented,” the company said in its S-1 statement, referring to 2015 through 2017.

Dropbox still uses AWS for less than 10 percent of its storage needs, it said in the statement. The company operates three data centers in the U.S. but none in Europe, and uses AWS resources in Europe to serve some customers on that continent.

In an Era of ‘Smart’ Things, Sometimes Dumb Stuff Is Better

www.nytimes.com - Comments

While riding a bicycle, for example, you often have to let go of the handle bar and lift the watch toward your face to check the time. When you’re standing on a bus or subway train and holding onto a pole, it is difficult to tilt your wrist at the correct angle to look at the time. Or when you’re in a meeting and want to see if you’re staying on schedule, flicking your wrist isn’t very subtle.

Until the Apple Watch manages to constantly display the time without sapping the battery, a normal wristwatch is better for telling the time in all those scenarios. That’s why you’ll see me wearing a normal watch at work but an Apple Watch at the gym.

A car mount vs. a smart car console

Many cars are now equipped with a touch-screen on the console that essentially mirrors your smartphone screen. Android phone users get to use Android Auto, and iPhone users hook into CarPlay.

These smart car systems are designed to seamlessly work with your smartphone. Plugging in an iPhone, for example, loads a screen of apps like Apple Maps, Apple Music and Apple’s podcast app, which you can then control on the console or with Siri instead of fiddling with your smartphone screen.

The problem with this concept is there are a limited number of apps that work with these smart infotainment systems. For example, if on CarPlay you prefer to use Google Maps or Waze, you’re out of luck and are stuck with Apple Maps.

In addition, if your smart car system needs a major software update, some car brands are lagging in allowing you to download and install the updates yourself. Instead, they require you to bring the car to the dealer and pay for the updates to be installed there. General Motors, for example, has for years declined to offer so-called over-the-air updates and will only say it plans to support them before 2020.

Using a phone mount is a cheap and simple solution that is far less frustrating. You just attach the mount to the dash, a CD player slot or an air conditioning vent, mount your phone and plug it into a power charger via the accessories port.

Voilà, your phone has become your infotainment system, capable of running your favorite navigation and music apps and using voice controls to place calls over speakerphone. The screen is large enough to clearly read maps, and you can update the operating system on your own. What more do you need?

An alarm clock vs. Amazon Echo Spot

Amazon recently introduced the Echo Spot, a smart alarm clock with a touch-screen and the Alexa virtual assistant. A less desirable feature is a built-in camera for placing video calls.

A camera on your nightstand that is constantly pointed at your bed? It’s like asking for your privacy to be violated. You might as well shop for your groceries in your underwear or post all your smartphone photos publicly on the web.

Amazon promises the camera software on the Echo Spot can be turned off whenever you aren’t using it. But it’s an obvious feature for hackers to target with malware.

So if your primary goal is to have a device that wakes you up on time to go to work, just get an old-school alarm clock.

A kitchen timer vs. Amazon Echo

One of the most common uses of Amazon’s Echo is to set a kitchen timer. Just say “Alexa, set a timer for 80 minutes” while you’re busy chopping vegetables.

Photo
It’s time to appreciate a kitchen timer over an Amazon Echo. Credit From left: Tony Cenicola/The New York Times; Mark Lennihan, via Associated Press

But there are reasons a cheap kitchen timer can be superior.

Cooking timing can vary depending on your heating element, among other factors. So if you have to check your food for doneness and change the kitchen timer, an old-school timer — either the analog variety or the type with a digital time display and two or three physical buttons — can be easier. It simply dings or beeps when the time is up and it’s quicker to add or subtract a few minutes by turning a dial or pressing a button or two.

You can also constantly see how much time is left on the timer, whereas with the Echo, you have to open a smartphone app to see the remaining time or ask Alexa to tell you how much time is left. Over the long term, using a smart speaker as a timer gets tedious.

A piece of paper vs. a tablet

When people buy new iPads or Amazon Fire tablets, they often give their older tablet a second life by designating it for the kitchen. There, the ancient tablet gets mounted to the refrigerator with a magnet and becomes a glorified recipe reader.

Having tried this experiment, it’s a hassle. You often have to clean the tablet after smearing food on the screen. The battery eventually needs to be recharged. And if you want to double or halve a recipe, you have to do some mental math, which makes multitasking more challenging when you are busy in the kitchen.

Printing out or jotting down a recipe on a piece of paper is just simpler. You can easily scribble additional notes, like changes and improvements to the recipe. Assuming you have decent handwriting, it’s easy to read the steps and ingredients.

And if it gets covered in food, you can just throw it away.

Continue reading the main story

Understanding Intel's Ivy Bridge Random Number Generator (2012)

www.electronicdesign.com - Comments

Random number generation is the Achilles heel of cryptography. Cryptographic algorithms – key generation, encryption and signing – need secret values that must be unknown to attackers, and the best way to create such a value is to choose it at random. But there is no way for software to create random numbers. All it can do is stretch out randomness from another source. That source might be the mechanical behavior of spinning discs, or temperature measurements, or network timing, or mouse and keyboard movements. Not all computers have mice, keyboards, temperature sensors or spinning discs. Embedded systems, virtual machines and servers with solid-state discs might have serious trouble gathering entropy.

Even when software can gather enough entropy, the random number generator (RNG) itself is notoriously difficult to test, and may contain subtle weaknesses that go unnoticed for years. These flaws can have serious consequences. For example, from September 2006 to May 2008, all OpenSSL keys generated on Debian and Ubuntu Linux systems were weak: server certificates, SSH login keys, email encryption and signing keys. Even using an ECDSA signing key on one of these systems would compromise it. This bug was caused by a one-line code change to OpenSSL's random number generator. Software-based RNGs are difficult to build and test, often hard to use, and still don't work everywhere. That's why security-oriented processors usually contain a dedicated hardware RNG, even though most general-purpose cores do not. Now Intel has included a hardware RNG on their "Ivy Bridge" processors, which were released earlier in 2012.

Table of Contents

  1. The Ivy Bridge Entropy Source
  2. The Conditioning Logic
  3. Cryptography Research's Review
  4. Conclusion

 

The Ivy Bridge Entropy Source

Each Ivy Bridge die contains one hardware RNG, shared by all the cores. The RNG begins with an entropy source (ES) whose behavior is determined by unpredictable thermal noise (Fig. 1). The core of Ivy Bridge's ES is an RS-NOR latch with the set and reset inputs wired together (red). When the R/S input is de-asserted, the latch becomes metastable, and its output eventually settles to 0 or 1, depending on thermal noise. The tricky part is consistently reaching that metastable state. This is accomplished by a negative feedback circuit (blue). This circuit adjusts the charge on a set of large capacitors, which is used as an extra input to the latch. The feedback nudges the latch slightly more toward 0 whenever it produces a 1 and vice-versa. The buffering circuit (green) detects when the latch has settled, and stores its output. After a delay it asserts and then de-asserts the latch's R/S input to produce another bit.

74774_fig1sm

Figure 1. This circuit uses a trimmed 1-shot to provide an entropy source.

The ES produces its own clock signal, which ticks irregularly at around 3GHz. The rest of the RNG operates at 800 MHz, so the RNG's output is first sampled into that clock domain. Next, a series of health tests inspects the samples to make sure that the entropy source hasn't failed.

The Conditioning Logic

The output of the ES is fairly high-quality, but it isn't strong enough for cryptographic purposes. The output won't be entirely balanced. The feedback circuit introduces bias, and the entire circuit may be influenced be analog effects such as ringing. To solve this, the output is passed through a cryptographic conditioner (Fig. 2), which condenses many mediocre random bits into a few very good ones. Even "unhealthy" samples are sent to the conditioner, because they can't hurt the quality of its output.

74774_fig2sm
 

Figure 2. The feedback circuit introduces bias so the output is passed through a conditioning pipeline to improve overall quality.

The conditioner produces a 256-bit seed value every few microseconds. But if all the cores on the chip are asking for random data, this won't satisfy the demand. Therefore, the seed value is fed into a NIST SP800-90A-based pseudorandom generator. This generator uses 128-bit AES in counter mode, and increases the amount of data that the generator can produce to around 800 megabytes per second.

Finally, the random data is put into an output buffer, where it can be retrieved, 64 bits at a time, by the RDRAND instruction. This instruction sets its destination register to a random value. RDRAND sets the processor's carry flag to indicate success. If the RNG fails (temporarily due to contention or permanently from a failed self-test), RDRAND instead clears the carry flag and outputs 0. Because the RNG itself signals failure by returning 0, the RDRAND instruction can never succeed with a 64-bit output of 0. This bias is generally small enough to ignore.

Cryptography Research's Review

Intel asked Cryptography Research to review the design of the RNG (download Analysis of Intel's Ivy Bridge Digital Random Number Generator). The summary is that the Ivy Bridge RNG is very conservatively designed, and the few concerns we identified do not materially affect its security.

The entropy source itself is mathematically sound. The capacitors in the tuning circuit need to be large; a pulse from the 1-shot must change them by less than the amplitude of the thermal noise in the circuit in order to consistently make the latch metastable. When they are at least this large, the ES circuit is robust against variations due to process, voltage and temperature.

We found that real-world data from the ES roughly matches these mathematical predictions. This means that in some cases, either the latch or the feedback circuit is not operating exactly the way that the model supposes. Still, the quality of ES's output is appropriately high, and even our pessimistic measurements on the worst samples are above its design goals.

The health-testing logic is ad-hoc, but it is sufficient to catch any simple patterns that might occur if the ES fails. The output of the ES is sampled down from about 3GHz to 800 MHz before the health tests run, and the sampling might hide some patterns from the health tests. Even so, if the ES's quality slips below its conservative design goal of 0.5 bits of entropy per output bit, then the health tests will almost certainly notice this failure.

The conditioner normally condenses 512 bits of ES output into 256 bits of reseeding data. Since the ES is expected to produce at least half a bit of entropy per output bit, these 256 bits should be almost completely random, but with a very small safety margin. Fortunately, when the RNG first starts, the initial seed is condensed from 32 kilobits of input, giving it a much wider safety margin. This strong seed is carried forward as the RNG runs, so the system remains strong even if it receives no additional entropy.

Finally, the seed is passed to a pseudorandom generator, which generates the RNG's final output. Under normal load, the generator reseeds every time it produces an output, so its output should be almost completely statistically random. Under very heavy load, it can generate multiple blocks of output between reseeds. In this case, the generator has 256 bits of state, and finding patterns in it is as hard as breaking 128-bit AES.

Conclusion

Intel's new RNG simplifies and improves cryptographic random number generation on servers and embedded platforms, where it is otherwise a challenge. It is carefully and conservatively engineered, and Cryptography Research recommends its use without reservation. Still, conservative system design will incorporate other sources of entropy if they are available. In all cases, users should confirm that the RDRAND instruction returns a carry flag of 1 to ensure that the RNG is reporting proper operation.

References

  1. C. E. Shannon, "A Mathematical Theory of Communication," Bell System Technical Journal, vol. 27, pp. 379–423, 623-656, 1948.
  2. E. Barker and J. Kelsey, Recommendation for Random Number Generation Using Deterministic Random Bit Generators, NIST Special Publication 800-90A, January 2012.
  3. B. Schneier, Security Pitfalls in Cryptography, Counterpane Systems, 1998.
  4. DSA-1571-1- openssl -- predictable random number generator," Debian, 13 May 2008. [Accessed 1 February 2012].
  5. A. K. Lenstra, J. P. Hughes, M. Augier, J. W. Bos, T. Kleinjung and C. Wachter, "Ron was wrong, Whit is right," IACR eprint archive, vol. 064, 2012.
  6. D. Bleichenbacher, On the generation of one-time keys in DL signature schemes, IEEE P1363 Working Group Meeting, November 2000.
  7. D. J. Johnston, "Mircoarchitecture Specification (MAS) for PP-DRNG," Intel Corporation (unpublished), V1.4, 2009.
  8. C. E. Dike, "3 Gbps Binary RNG Entropy Source," Intel Corporation (unpublished), 2011.
  9. C. E. Dike and S. Gueron, "Digital Symmetric Random Number Generator Mathematics," Intel Corporation (unpublished), 2009.
  10. M. Dworkin, "Recommendation for Block Cipher Modes of Operation: The CCM Mode for Authentication and Confidentiality," NIST Special Publication 800-38C, May 2004.
  11. Z. Rached, F. Alajaji and L. Campbell, "Rényi's Entropy Rate For Discrete Markov Sources," 1999.
  12. NIST, "NIST Special Publication 800-22rev1a," 11 August 2010. [Accessed 2 February 2012].
  13. M. Hamburg, P. Kocher. M. Marson, Analysis of Intel's Ivy Bridge Digital Random Number Generator

Likelihood of discontinuous progress around the development of AGI

aiimpacts.org - Comments

We aren’t convinced by any of the arguments we’ve seen to expect large discontinuity in AI progress above the extremely low base rate for all technologies. However this topic is controversial, and many thinkers on the topic disagree with us, so we consider this an open question.

Details

Definitions

We say a technological discontinuity has occurred when a particular technological advance pushes some progress metric substantially above what would be expected based on extrapolating past progress. We measure the size of a discontinuity in terms of how many years of past progress would have been needed to produce the same improvement. We use judgment to decide how to extrapolate past progress.

For instance, in the following trend of progress in chess AI performance, we would say that there was a discontinuity in 2007, and it represented a bit over five years of progress at previous rates.

Figure 1: Machine chess progress, in particular SSDF records.

Relevance

Discontinuity by some measure, on the path to AGI, lends itself to:

  • A party gaining decisive strategic advantage
  • A single important ‘deployment’ event
  • Other very sudden and surprising events

Arguably, the first two require some large discontinuity. Thus the importance of planning for those outcomes rests on the likelihood of a discontinuity.

Outline

We investigate this topic in two parts. First, with no particular knowledge of AGI as a technology, how likely should we expect a particular discontinuity to be? We take the answer to be quite low. Second, we review arguments that AGI is different from other technologies, and lends itself to discontinuity. We currently find these arguments uncompelling, but not decisively so.

Default chance of large technological discontinuity

Discontinuities larger than around ten years of past progress in one advance seem to be rare in technological progress on natural and desirable metrics. We know of around four examples, though have not completed this investigation.

This does not include the discontinuities when metrics initially go from zero to a positive number. For instance, the metric ‘number of Xootr scooters in the world’ presumably went from zero to one on the first day of production, though this metric had seen no progress before. So on our measure of discontinuity size, this was infinitely many years of progress in one step. It is rarer for a broader metric (e.g. ‘scooters’) to go from zero to one, but it must still happen occasionally. We do not mean to ignore this type of phenomenon, but just to deal with it separately, since discontinuity in an ongoing progress curve seems to be quite rare, whereas the beginning of positive progress on a metric presumably occurs once for every metric that begins at zero.

Proposed reasons to expect a discontinuity near AGI

We have said that the base rate of large technological discontinuities appears to be low. We now review and examine arguments that AGI is unusually likely to either produce a discontinuity, or follow one.

Hominid variation

Argument

Humans are vastly more successful in certain ways than other hominids, yet in evolutionary time, the distance between them was small. This suggests that evolution saw discontinuous progress in returns, if not in intelligence itself, somewhere approaching human-level intelligence. If evolution experienced this, this suggests that artificial intelligence research may do also.

Counterarguments

Evolution does not appear to have been optimizing for the activities that humans are good at. In terms of what evolution was optimizing for—short term ability to physically and socially prosper—it is quite unclear that humans are disproportionately better than earlier hominids, compared to differences between earlier hominids and less optimized animals.

This counterargument implies that if one were to optimize for human-like success that it would be possible (and likely at some point) to make agents with brains the size of chimps, that were humanlike but somewhat worse in their capabilities.

It is also unclear to us that humans succeed so well on something like ‘intelligence’ from vastly superior individual intelligence, rather than vastly superior group intelligence, via innovations such as language.

Status

See status section of ‘brain scaling’ argument—they are sufficiently related to be combined.

Brain scaling

Argument

If we speculate that the main relevant difference between us and our hominid relatives is brain size, then we can argue that brain size in particular appears to produce very fast increase in impressiveness. So if we develop AI systems with certain performance, we should expect to make disproportionately better systems by using more hardware.

Variation in brain size among humans also appears to be correlated with intelligence, and differences in intelligence in the human range anecdotally correspond to large differences in some important capabilities, such as ability to develop new physics theories. So if the relatively minor differences in human brain size cause the differences in intelligence, this would suggest that larger differences in ‘brain size’ possible with computing hardware could lead to quite large differences in some important skills, such as ability to successfully theorize about the world.

Counterarguments

This is closely related to the ‘hominids saw discontinuous progress from intelligence’ argument (see above), so shares the counterargument that evolution does not appear to have been optimizing for those things that humans excel at.

Furthermore, if using more hardware generally gets outsized gains, one might expect that by the time we have any particular level of AGI performance, we are already using extremely large amounts of hardware. If this is a cheap axis on which to do better, it would be surprising if we left that value on the table until the end. So the only reason to expect a discontinuity here would seem to be if the opportunity to use more hardware and get outsized results started existing suddenly. We know of no particular reason to expect this, unless there was a discontinuity in something else, for instance if there was indeed one algorithm. That type of discontinuity is discussed elsewhere.

Status

This argument seems weak to us currently, but further research could resolve these questions in directions that would make it compelling:

  1. Are individual humans radically superior to apes on particular measures of cognitive ability? What are those measures, and how plausible is it that evolution was (perhaps indirectly) optimizing for them?
  2. How likely is improvement in individual cognitive ability to account for humans’ radical success over apes? (For instance, compared to new ability to share innovations across the population)
  3. How does human brain size relate to intelligence or particular important skills, quantitatively?

Intelligence explosion

Argument

Intelligence works to improve many things, but the main source of intelligence—human brains—is arguably not currently improved substantially itself. The development of artificial intelligence will change that, by allowing the most intelligent entities to directly contribute to their increased intelligence. This will create a feedback loop where each step of progress in artificial intelligence makes the next steps easier and so faster. Such a feedback loop could potentially go fast. This would not strictly be a discontinuity, in that each step would only be a little faster than the last, however if the growth rate became sufficiently fast sufficiently quickly, it would be much like a discontinuity from our perspective.

This feedback loop is usually imagined to kick in after AI was ‘human-level’ on at least some tasks (such as AI development), but that may be well before it is recognized as human-level in general.

Counterarguments

Positive feedback loops are common in the world, and very rarely move fast enough and far enough to become a dominant dynamic in the world. So we need strong reason to expect an intelligence-of-AGI feedback loop to be exceptional, beyond the observation of a potential feedback effect. We do not know of such arguments, however this is a widely discussed topic, so contenders probably exist.

Status

The counterargument currently seems to carry to us. However we think the could be argument strong if:

  1. Strong reason is found to expect an unusually fast and persistent feedback loop
  2. Quantitative models of current relationships between the variables hypothesized to form a feedback loop suggest very fast intelligence growth.

One algorithm

Argument

If a technology is simple, we might expect to discover it in a small number of steps, perhaps one. If it is also valuable, we might expect each of those those steps to represent huge progress in some interesting metric. The argument here is that intelligence—or something close enough to what we mean by intelligence—is like this.

Some reasons to suspect that intelligence is conceptually simple:

  1. Evolution discovered it relatively quickly (though see earlier sections on hominid evolution and brain size)
  2. It seems conceivable to some that the mental moves required to play Go are basically all the ones you need for a large class of intellectual activities, at least with the addition of some way to navigate the large number of different contexts that are about that hard to think about.
  3. It seems plausible that the deep learning techniques we have are already such a thing, and we just need more hardware and adjustment to make it work.

We expect that this list is missing some motivations for this view.

This argument can go in two ways. If we do not have the one algorithm yet, then we might expect that when we find it there will be a discontinuity. If we already have it, yet don’t have AGI, then what remains to be done is probably accruing sufficient hardware to run it. This might traditionally be expected to lead to continuous progress, but it is also sometimes argued to lead to discontinuous progress. Discontinuity in hardware is discussed in a forthcoming section, so here we focus on discontinuity from acquiring the insight.

Counterarguments

Society has invented many simple things before. So even if they are unusually likely to produce discontinuities, the likelihood would still seem to be low, since discontinuities are rare. To the extent that you believe that society has not invented such simple things before, you need a stronger reason to expect that intelligence is such a thing, and we do not know of strong reasons.

Status

We currently find this argument weak, but would find it compelling if:

  1. Strong reason emerged to expect intelligence to be a particularly simple innovation, among all innovations, and
  2. We learned that very simple innovations do tend to cause discontinuities, and we have merely failed to properly explore possible past discontinuities well.

Deployment scaling

Argument

Having put in the research effort to develop one advanced AI system, it is very little further effort to make a large number of them. Thus at the point that a single AGI system is developed, there could shortly thereafter be a huge number of AGI systems. This would seem to represent a discontinuity in something like global intelligence, and also in ‘global intelligence outside of human brains’.

Counterarguments

It is the nature of almost every product that large quantities of upfront effort are required to build a single unit, at which point many units can be made more cheaply. This is especially true of other software products, where a single unit can be copied without the need to build production facilities. So this is far from a unique feature of AGI, and if it produced discontinuities, we would see them everywhere.

The reason that the ability to quickly scale the number of instances of a product does not generally lead to a discontinuity seems to be that before the item’s development, there was usually another item that was slightly worse, which there were also many copies of. For instance, if we write a new audio player, and it works well, it can be scaled up across many machines. However on each machine it is only replacing a slightly inferior audio player. Furthermore, because everyone already has a slightly inferior audio player, adoption is gradual.

If we developed human-level AGI software right now, it would be much better than what computing hardware is already being used for, and because of that and other considerations, it may spread to a lot of hardware fast. So we would expect a discontinuity. However if we developed human-level AGI software now, this would already be a discontinuity in other metrics.  In general, it seems to us that this scaling ability may amplify the effects of existing discontinuities, but it is hard to see how it could introduce one.

Status

This argument seems to us to rest on a confusion. We do not think it holds unless there is already some other large discontinuity.

Train vs. test

Argument

There are at least three different processes you can think of computation being used for in the development of AGI:

  1. Searching the space of possible software for AGI designs
  2. Training specific AGI systems
  3. Running a specific AGI system

We tend to think of the first requiring much more computation than the second, which requires much more computation than the third. This suggests that when we succeed at the first, hardware will have to be so readily available that we can quickly do a huge amount of training, and having done the training, we can run vast numbers of AI systems. This seems to suggest a discontinuity in number of well-trained instances of AGI systems.

Counterarguments

This seems to be a more specific version of the deployment scaling argument above, and so seems to fall prey to the same counterargument: that whenever you find a good AGI design and scale up the number of systems running it, you are only replacing somewhat worse software, so this is not a discontinuity in any metric of high level capability. For there to be a discontinuity, it seems we would need a case where step 1 returns an AGI design that is much better than usual, or returns a first AGI design that is already very powerful compared to other ways of solving the high level problems that it is directed at. These sources of discontinuity are both discussed elsewhere.

Status

This seems to be a variant of the deployment scaling argument, and so similarly weak. However it is more plausible to us here that we have failed to understand some stronger form of the argument, that others find compelling.

Starting high

Argument

As discussed in Default chance of technological discontinuity above, the first item in a trend always represents a ‘large’ discontinuity in the number of such items—the number has been zero forever, and now it is one, which is more progress at once than in all of history. When we said that discontinuities were rare, we explicitly excluded discontinuities such as these, which appear to be common but uninteresting. So perhaps a natural place to find an AGI-related discontinuity is here.

Discontinuity from zero to one in the number of AGI systems would not be interesting on its own. However perhaps there are other metrics where AGI will represent the first step of progress. For instance, the first roller coaster caused ‘number of roller coasters’ to go from zero to one, but also ‘most joy caused by a roller coaster’ to go from zero to some positive number. And once we admit that inventing some new technology T could cause some initial discontinuity in more interesting trends than ‘number of units of T’, we do not know very much about where or how large such first steps may be (since we were ignoring such discontinuities). So the suggestion is that AGI might see something like this: important metrics going from zero to high numbers, where ‘high’ is relative to social impact (our previous measure of discontinuities being unhelpful in the case where progress was previously flat). Call this ‘starting high’.

Having escaped the presumption of a generally low base rate of interesting discontinuities then, there remain the questions of what the base rate should be for new technologies ‘starting high’ on interesting metrics, and whether we should expect AGI in particular to see this even above the new base rate.

One argument that the base rate should be high is that new technologies have a plethora of characteristics, and empirically some of them seem to start ‘high’, in an intuitive sense. For instance, even if the first plane did not go very far, or fast, or high, it did start out 40 foot wide—we didn’t go through flying machines that were only an atom’s breadth, or only slightly larger than albatrosses. ‘Flying machine breadth’ is not a very socially impactful metric, but if non-socially impactful metrics can start high, then why not think that socially impactful metrics can?

Either way, we might then argue that AGI is especially likely to produce high starting points on socially impactful metrics:

  1. AGI will be the first entry in an especially novel class, so it is unusually likely to begin progress on metrics we haven’t seen progress on before.
  2. Fully developed AGI would represent progress far above our current capabilities on a variety of metrics. If fully developed AGI is unusually impactful, this suggests that the minimal functional AGI is also unusually impactful.
  3. AGI is unusually non-amenable to coming in functional yet low quality varieties. (We do not understand the basis for this argument well enough to relay it. One possibility is that the insights involved in AGI are sufficiently few or simple that having a partial version naturally gets you a fully functional version quickly. This is discussed in another section. Another is that AGI development will see continuous progress in some sense, but go through non-useful precursors—much like ‘half of a word processor’ is not a useful precursor to a word processor. This is also discussed separately elsewhere.)
Counterarguments

While it is true that new technologies sometimes represent discontinuities in metrics other than ‘number of X’, these seem very unlikely to be important metrics. If they were important, there would generally be some broader metric that had previously been measured that would also be discontinuously improved. For instance, it is hard to imagine AGI suddenly possessing some new trait Z to a degree that would revolutionize the economy, without this producing discontinuous change in previously measured things like ‘ability to turn resources into financial profit’.

Which is to say, it is not a coincidence that the examples of high starting points we can think of tend to be unimportant. If they were important, it would be strange if we had not previously made any progress on achieving similar good outcomes in other ways, and if the new technology didn’t produce a discontinuity in any such broader goal metrics, then it would not be interesting. (However it could still be the case that we have so far systematically failed to look properly at very broad metrics as they are affected by fairly unusual new technologies, in our search for discontinuities.)

We know of no general trend where technologies that are very important when well developed cause larger discontinuities when they are first begun. We have also not searched for this specifically, but isn’t an immediately apparent trend in the discontinuities we know of.

Supposing that an ideal AGI is much smarter than a human, then humans clearly demonstrate the possibility of functional lesser AGIs. (Which is not to say that ‘via human-like deviations from perfection’ is a particularly likely trajectory.) Humans also seem to demonstrate the possibility of developing human-level intelligence, then not immediately tending to have far superior intelligence. This is usually explained by hardware limitations, but seems worth noting nonetheless.

Status

We currently think it is unlikely that any technology ‘starts high’ on any interesting metric, and either way AGI does not seem especially likely to do so. We would find this argument more compelling if:

  1. We were shown to be confused about the inference that ‘starting high’ on an interesting metric should necessarily produce discontinuity in broader, previously improved metrics. For instance, if a counterexample were found.
  2. Historical data was found to suggest that ultimately valuable technologies tended to ‘start high’ or produce discontinuities.
  3. Further arguments emerged to expect AGI to be especially likely to ‘start high’.

Awesome AlphaZero

Argument

We have heard the claim that AlphaZero represented a discontinuity, at least in the ability to play a variety of games using the same system, and maybe on other metrics. If so, this would suggest that similar technologies were unusually likely to produce discontinuities.

Counterarguments

Supposing that AlphaZero did represent discontinuity on playing multiple games using the same system, there remains a question of whether that is a metric of sufficient interest to anyone that effort has been put into it. We have not investigated this.

Whether or not this case represents a large discontinuity, if it is the only one among recent progress on a large number of fronts, it is not clear that this raises the expectation of discontinuities in AI very much, and in particular does not seem to suggest discontinuity should be expected in any other specific place.

Status

We have not investigated the claims this argument is premised on, or examined other AI progress especially closely for discontinuities. If it turned out that:—

  1. Recent AI progress represented at least one substantial discontinuity
  2. The relevant metric had been a focus of some prior effort

—then we would consider more discontinuities around AGI more likely. Beyond raising the observed base rate of such phenomena, this would also suggest to us that the popular expectation of AI discontinuity is well founded, even if we have failed to find good explicit arguments for it. So this may raise our expectation of discontinuous change more than the change in base rate would suggest.

Uneven skills

Argument

There are many axes of ability, and to replace a human at a particular task, a system needs to have a minimal ability on all of them. At which point, it should be substantially superhuman on some axes. For instance, by the time that self-driving cars are safe enough in all circumstances to be allowed to drive, they might be very much better than humans at noticing bicycles. So there might be a discontinuity in bicycle safety at the point that self-driving cars become common.

If this were right, it would also suggest a similar effect near overall human-level AGI: that the first time we can automate a variety of tasks as well as a human, we will be able to automate them much better than a human on some axes. So those axes may see discontinuity.

Figure 2: Illustration of the phenomenon in which the first entirely ‘human-level’ system is substantially superhuman on most axes. (Image from Superintelligence Reading Group)
Counterarguments

This phenomenon appears to be weakly related to AGI in particular, and should show up whenever one type of process replaces another one. Replacement of one type of system with another type does not seem very rare. So if it is true that large discontinuities are rare, then this type of dynamic does not produce large discontinuities often.

This argument benefits from the assumption that you have to ‘replace’ a human in one go, which seems usually false. New technology can usually be used to complement humans, admittedly with inefficiency relative to a more integrated system. For instance, one reason there will not be a discontinuity in navigation ability of cars on the road is that humans already use computers for their navigation. However this counterargument doesn’t apply to all axes. For instance, if computers operate quite fast but cannot drive safely, you can’t lend their speed to humans and somehow still use the human steering faculties.

Status

This argument seems successfully countered, unless:

  1. We have missed some trend of discontinuities of this type elsewhere. If there were such a trend, it seems possible that we would have systematically failed to notice them because they look somewhat like ‘initial entry goes from zero to one’, or are in hard to measure metrics.
  2. There are further reasons to expect this phenomenon to apply in the AGI case specifically, especially without being substantially mitigated by human-complementary technologies

Payoff thresholds

Argument

Even if we expect progress on technical axes to change continuously, there seems no reason this shouldn’t translate to discontinuous change in a related axis such as ‘economic value’ or ‘ability to take over the world’. For instance, continuous improvement in boat technology might still lead to a threshold in whether or not you can reach the next island.

Counterarguments

There actually seems to be a stronger theoretical reason to expect continuous progress in metrics that we care about directly, such as economic value, than on more technical metrics. This is that if making particular progress is expected to bring outsized gains, more effort should be directed toward the project until that effort has offset a large part of the gains. For instance, suppose that for some reason guns were a hundred times more useful if they could shoot bullets at just over 500m/s than just under, and otherwise increasing speed was slightly useful. Then if you can make a 500m/s gun, you can earn a hundred times more for it than a lesser gun. So if guns can currently shoot 100m/s, gun manufacturers will be willing to put effort into developing the 500m/s one up to the point that it is costing nearly a hundred times more to produce it. Then if they manage to produce it, they do not see a huge jump in value of making guns, and the buyer doesn’t see a huge jump in value of gun – their gun is much better but also much more expensive. However at the same time, there may have been a large jump in ‘speed of gun’.

This is a theoretical argument, and we do not know how well supported it is. (It appears to be at least supported by the case of nuclear weapons, which were discontinuous in the technical metric ‘explosive power per unit mass’, but not obviously so in terms of cost-effectiveness).

In practice, we don’t know of many large discontinuities in metrics of either technical performance or things that we care about more directly.

Even if discontinuities in value metrics were common, this argument would need to be paired with another for why AGI in particular should be expected to bring about a large discontinuity.

Status

This argument currently seems weak. In the surprising event that that discontinuities are actually common in metrics more closely related to value (rather than generally rare, and especially so that case), the argument would only be mildly stronger, since it doesn’t explain why AGI is especially susceptible to this.

Human-competition threshold

Argument

There need not be any threshold in nature that makes a particular level of intelligence much better. In a world dominated by humans, going from being not quite competitive with humans to slightly better could represent a step change in value generated.

This is a variant of the above argument from payoff thresholds that explains why AGI is especially likely to see discontinuity, even if it is otherwise rare: it is not often that something as ubiquitous and important as humans is overtaken by another technology.

Counterarguments

Humans have a variety of levels of skill, so even ‘competing with humans’ is not a threshold. You might think that humans are in some real sense very close to one another, but this seems unlikely to us.

Even competing with a particular human is unlikely to have threshold behavior—without for instance a requirement that someone hire one or the other—because there are a variety of tasks, and the human-vs.-machine differential will vary by task.

Status

This seems weak. We would reconsider if further evidence suggested that the human range is very narrow, in some ubiquitously important dimension of cognitive ability.

Buffett: How inflation swindles the equity investor (Fortune Classics, 1977)

fortune.com - Comments

Buffett: How inflation swindles the equity investor (Fortune Classics, 1977)
Search

Email exchange between MIT Media Lab and the IOTA Foundation [pdf]

www.tangleblog.com - Comments

Overconfident Students, Dubious Employers

www.insidehighered.com - Comments

College students may believe they’re ready for a job, but employers think otherwise.

At least, that’s according to data from the National Association of Colleges and Employers, which surveyed graduating college seniors and employers and found a significant difference in the groups' perceptions.

The association surveyed 4,213 graduating seniors and 201 employers on eight “competencies” that it considers necessary to be prepared to enter the workplace. This information comes from the association’s 2018 Job Outlook Survey.

For the most part, a high percentage of students indicated in almost every category they thought they were proficient. Employers disagreed.

“This can be problematic because it suggests that employers see skills gaps in key areas where college students don’t believe gaps exist,” a statement from the association reads.

Figure 1: Employer vs. student perception of proficiency in career readiness competencies, by percentage of respondents. On professionalism/work ethic, 42.5 percent of employers rated recent grads proficient, while 89.4 percent of students considered themselves proficient. On oral/written communications, 41.6 percent of employers rated recent grads proficient, while 79.4 percent of students considered themselves proficient. On critical thinking/problem solving, 55.8 percent of employers rated recent grads proficient, while 79.9 percent of students considered themselves proficient. On teamwork/collaboration, 77 percent of employers rated recent grads proficient, while 85.1 percent of students considered themselves proficient. On leadership, 33 percent of employers rated recent grads proficient, while 70.5 percent of students considered themselves proficient. On digital technology, 65.8 percent of employers rated recent grads proficient, while 59.9 percent of students considered themselves proficient. On career management, 17.3 percent of employers rated recent grads proficient, while 40.9 percent of students considered themselves proficient. On global/intercultural fluency, 20.7 percent of employers rated recent grads proficient, while 34.9 percent of students considered themselves proficient. Source: Job Outlook 2018 (N=201 employing organizations) and The Class of 2017 Student Survey Report (N=4,213 graduating seniors), National Association of Colleges and Employers. The percentages corresponding to “rated proficient” represent, among all responding employers, the percentage who, on a five-point scale, rated recent graduates “very” (4) or “extremely” (5) proficient in the respective competency. The percentages corresponding to “considered proficient” represent, among all graduating seniors from the Class of 2017, the percentage who, on a five-point scale, considered himself/herself either “very” (4) or “extremely” (5) proficient in the respective competency.The biggest divide was around students’ professionalism and work ethic. Almost 90 percent of seniors thought they were competent in that area, but only about 43 percent of the employers agreed.

Nearly 80 percent of students also believed they were competent in oral and written communication and critical thinking, while only roughly 42 percent and 56 percent of employers, respectively, indicated that students were successful in those areas.

Per the survey, only in digital technology skills were employers more likely to feel that students were prepared versus the seniors themselves.

Almost 66 percent of employers rated students proficient in technology compared to 60 percent of the seniors.

But Brandon Busteed, executive director of Gallup's higher education division, which also conducts research related to graduates and careers, said these sorts of definitions can vary.

For instance, Gallup has found that generally an employer believes that "critical thinking" is coming up with new, original thought. But in an academic sense, it can mean more picking apart ideas in depth, he said.

Written communication can differ, too, he said -- some students might excel at writing technical reports or papers with lots of citations, but these are far different than writing for marketing, Busteed said.

"I think in some ways these studies beg for further exploration," he said.

Busteed also pointed out that the lifestyle for the traditional undergraduate student likely does not match how they will need to operate when they enter the work force.

Undergraduates are typically scheduling classes later in the morning and staying up until the late hours of the night, which does not prepare them for an eight-hour workday, he said.

The easy solution: set students up in a more professional environment, Busteed said -- this could be internships or co-op programs. If students can't go to an actual office, then the environment should be brought to them so they have a better sense of how a workplace runs.

"It's good news because there's real quick fixes, but it's not a prevalent as it should be," he said.

The Association of American Colleges and Universities has conducted similar research. In 2015, it found that students thought they were far better equipped for jobs than employers did.

The AAC&U looked at some of the same measures as the association. Specifically around oral communication, students ranked themselves highly -- about 62 percent of students believed they did well in this area compared to 28 percent of employers. That and written communication showed the biggest gaps in the AAC&U report (27 percent of employers versus 65 percent of students).

“When it comes to the types of skills and knowledge that employers feel are most important to workplace success, large majorities of employers do NOT feel that recent college graduates are well prepared,” the AAC&U report states. “This is particularly the case for applying knowledge and skills in real-world settings, critical thinking skills, and written and oral communication skills -- areas in which fewer than three in 10 employers think that recent college graduates are well prepared.”

Getting Started with MapD, Part 2: Electricity Dataset

randyzwitch.com - Comments

In my previous MapD post, I loaded electricity data into MapD Community Edition, intentionally ignoring the what of the data to keep that post from being too overwhelming. Now let’s take a step back and explain the dataset, show how to format the data using Python that was loaded into MapD, then use the MapD Immerse UI to build a simple dashboard.

PJM Metered Load Data

I started off my career at PJM doing long-term electricity demand forecasting, to help power engineers do transmission line studies for reliability and to support expansion of the electrical grid in the U.S. Because PJM is a quasi-government agency, they provide over 25 years of hourly electricity usage for the Eastern and Central U.S., both in aggregate and in by local power region (roughly, the local power company territories).

However, just because the data is available doesn’t mean it’s convenient, and unfortunately, the data are stored as Excel spreadsheets. This is easily remedied using pandas (v0.22.0, python3.6):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
import os
import pandas as pd

#change to directory with files for convenience
os.chdir("~/electricity_data/")

#first sheet in workbook contains all info for years 1993-1999
df1993_1999 = [pd.read_excel(str(x) + "-hourly-loads.xls") for x in range(1993,1999)]

#melt, append df1993-df1999 together
df_melted = pd.DataFrame()
for x in df1993_1999:
    x.columns = df1993_1999[1].columns.tolist()
    x_melt = pd.melt(x, id_vars=['ACTUAL_DATE', 'ZONE_NAME'], var_name = "HOUR_ENDING", value_name = "MW")
    df_melted = df_melted.append(x_melt)

#multiple sheets to concatenate
#too much variation for a one-liner
d2000 = pd.read_excel("2000-hourly-loads.xls", sheet_name = [x for x in range(2,17)])
d2001 = pd.read_excel("2001-hourly-loads.xls", sheet_name = None)
d2002 = pd.read_excel("2002-hourly-loads.xls", sheet_name = [x for x in range(1,18)])
d2003 = pd.read_excel("2003-hourly-loads.xls", sheet_name = [x for x in range(1,19)])
d2004 = pd.read_excel("2004-hourly-loads.xls", sheet_name = [x for x in range(2,24)])
d2005 = pd.read_excel("2005-hourly-loads.xls", sheet_name = [x for x in range(2,27)])
d2006 = pd.read_excel("2006-hourly-loads.xls", sheet_name = [x for x in range(3,29)])
d2007 = pd.read_excel("2007-hourly-loads.xls", sheet_name = [x for x in range(3,29)])
d2008 = pd.read_excel("2008-hourly-loads.xls", sheet_name = [x for x in range(3,29)])
d2009 = pd.read_excel("2009-hourly-loads.xls", sheet_name = [x for x in range(3,29)])
d2010 = pd.read_excel("2010-hourly-loads.xls", sheet_name = [x for x in range(3,29)])
d2011 = pd.read_excel("2011-hourly-loads.xls", sheet_name = [x for x in range(3,33)])
d2012 = pd.read_excel("2012-hourly-loads.xls", sheet_name = [x for x in range(3,33)])
d2013 = pd.read_excel("2013-hourly-loads.xls", sheet_name = [x for x in range(3,34)])
d2014 = pd.read_excel("2014-hourly-loads.xls", sheet_name = [x for x in range(3,34)])
d2015 = pd.read_excel("2015-hourly-loads.xls", sheet_name = [x for x in range(3,40)])
d2016 = pd.read_excel("2016-hourly-loads.xls", sheet_name = [x for x in range(3,40)])
d2017 = pd.read_excel("2017-hourly-loads.xls", sheet_name = [x for x in range(3,42)])
d2018 = pd.read_excel("2018-hourly-loads.xls", sheet_name = [x for x in range(3,40)])

#loop over dataframes, read in matrix-formatted data, melt to normalized form
for l in [d2000, d2001, d2002, d2003, d2004, d2005, d2006, d2007, d2008, d2009, d2010,
          d2011, d2012, d2013, d2014, d2015,d2016,d2017,d2018]:

    for k in l:
        temp = l[k].iloc[:, 0:26]
        temp.columns = df1993_1999[1].columns.tolist()
        df_melted = df_melted.append(pd.melt(temp, id_vars=['ACTUAL_DATE', 'ZONE_NAME'], var_name = "HOUR_ENDING", value_name = "MW"))

#(4898472, 4)
#152MB as CSV
df_melted.to_csv("hourly_loads.csv", index=False)

The code is a bit verbose, if only because I didn’t want to spend time to figure out how to programmatically determine how many tabs each workbook has. But the concept is the same each time: read an Excel file, get the data into a dataframe, then convert the data to long form. So instead of having 26 columns (Date, Zone, Hr1-Hr24), we have 4 columns, which is quite frequently a more convenient way to access the data (especially when using SQL).

The final statement writes out a CSV of approximately 4MM rows, the same dataset that was loaded using mapdql in the first post.

Top 10 Usage Days By Season

One of the metrics I used to monitor as part of my job was the top 5/top 10 peak electricity use days per Summer (high A/C usage) and Winter (electric space heating) seasons. Back in those days, I used to use SAS against an enterprise database and the results would come back eventually

Obviously, it’s not a fair comparison to compare today’s GPUs vs. late ’90s enterprise databases in terms of performance, but back then it did take a non-trivial amount of effort to run this query to keep the report updated. With MapD, I can do the same report in ~100ms:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
--MapD doesn't currently support window functions, so need to precalculate maximum by day
with qry as (select
actual_date,
zone_name,
max(MW) as daily_max_usage
from hourly_loads
where zone_name = 'MIDATL' and actual_date between '2017-06-01' and '2017-09-30'
group by 1,2)
select
hl.actual_date,
hl.zone_name,
hl.hour_ending,
hl.MW
from hourly_loads as hl
inner join qry on qry.actual_date = hl.actual_date and qry.daily_max_usage = hl.mw
order by daily_max_usage desc
limit 10;

top 10 electric usage

The thing about returning an answer in 100ms or so is that its fast enough where calling these results from a webpage/dashboard would be very responsive; that’s where MapD Immerse comes in.

Building A Dashboard Using MapD Immerse

Rather than copy/pasting the query in and running it, it’s pretty easy to build an automated report using the Immerse dashboard builder. I’m limited to a single data source because I’m using MapD Community Edition, but in just a few minutes I was able to create the following dashboard:

mapd immerse dashboard

I took the query from above and built a view to encapsulate the query, so I didn’t have to worry about the with statement or joins, I could just use the view as if the results were pre-calculated. From there, adding in a results table and two bar charts was fairly quick (in the same drag-and-drop style of Tableau or other BI/reporting tools).

While this dashboard is pretty rudimentary in its design, were this data source set up as real-time using Apache Kafka or similar, this chart would always be up-to-date for use on a TV screen or as a browser bookmark without any additional data or web engineering.

Obviously, many dashboarding tools exist, but its important to note that no pre-aggregation or column indexing or other standard database performance tricks are being employed (outside of specialized hardware and fast GPU RAM caching). Even with 10 dashboard tiles updating serially 100ms at a time, you are still in the 1-2s page load time, on par with the fastest-loading dynamic webpages on the internet.

Programmatic analytics using pymapd

While dashboarding can be very effective for keeping senior management up-to-date, the real value of data is unlocked with more in-depth analytics and segmentation. In my next blog post, I’ll cover how to access MapD using pymapd in Python, doing more advanced visualizations and maybe even some machine learning…

FAA Tapes from That Oregon UFO Incident

www.thedrive.com - Comments

First we hear about the big question as to who "requested" the scramble, as according to the call, it has to come from FAA headquarters. The manager floats the idea, in retrospect, of having the airliners keep a visual on the craft instead of allowing them to descending into PDX, at least until the F-15s show up, but the FAA official swats that down as they didn't know what the aircraft was, "if they are equipped with anything" or its intentions. She reiterates that getting the military involved was a good idea but that it should have come from FAA headquarters over the DEN. The manager reminds her again that he doesn't know who requested military assistance and that Oakland Center told him to call WADS initially. 

Next we hear from Oakland Center again, at first discussing who ordered the scramble, but then the conversation goes into talking about what actually happened. Both agree that there was "definitely something out there" with the Oakland Center controller saying the aircraft first appeared going southbound at high speed before executing an abrupt maneuver and then "took off northbound." Even figuring out how to report the encounter seems totally foreign to both higher ranking controllers, with one stating "I have a feeling someone is going to go through this with a fine-tooth comb." 

Then we get into the pilot interviews over the phone, with the manager's intention being for each crew to write up a report detailing their individual perspective of the incident. During the call with United 612 there are some odd dropped moments, but the pilot describes the encounter, stating that he was too far away to make out the type. The next call, with Alaska Airlines 525, also doesn't reveal much as the crew says they never were able to see it, but the crew of Skywest 3478 did, although he didn't have much to add. 

The call with the pilot of Southwest 4712 was by far the most interesting. He immediately notes how strange the encounter was and how he has never seen an incident like it in nearly 30 years of flying jets. The pilot noted, "if it was like a Lear (private jet) type airframe I probably would not have seen it this clear. This was a white airplane and it was big. And it was moving at a clip too, because we were keeping pace with it, it was probably moving faster than we were... It was a larger aircraft yeah." He also said they watched the object from Northern California all the way to their descent into Portland. 

The manager's final call, was with the FAA's Quality Assurance Group, who is taken by surprise by the details surrounding the event, and especially with the fact that nobody still knew what the aircraft was or where it ended up. "Wow that's weird" is the operative quote by the FAA official, which is insightful to say the least as these people deal with unique incidents that occur in American airspace on a daily basis. The manager agreed with the sentiment and noted that it wasn't some small aircraft and it was moving fast, outpacing a 737 cruising nearby. The official also says that the incident should be classified as "potentially significant" on reporting documents. She even said that this was "a weird enough thing that there is not a set procedure... It's not often we hear about an unknown guy up at that altitude." 

Collectively these materials give us incredible insight not only into this incident, but also into how such an event is actually handled in real-time by those who are responsible for the safety of those in the air and those on ground below. What they don't offer is any sort of an explanation for what happened on that fall evening. But really, the fact that all those involved, from air traffic controllers, to Air Force radar operators, to airline pilots, and even special FAA officials tasked with responding to all types of out of the ordinary incidents that occur in the sky on a daily basis seem just as puzzled with this event as we are makes the story all that much more intriguing.  

Above all else, this new evidence underscores just how rare these events actually are, especially ones that include multiple sightings, the use of multiple sensors, the involvement of various agencies including the military, and some of the most capable air-to-air fighters in the world. And after recent reports unmasking how the Pentagon remains highly interested in encounters just like this one, it holds even more weight than it did three months ago.

We will continue to investigate this bizarre incident and we will keep you updated as to what else we discover.

Contact the author: [email protected]

Planet Shadertoy

www.shadertoy.com - Comments

  • type radians (type degrees)
  • type degrees (type radians)
  • type sin (type angle)
  • type cos (type angle)
  • type tan (type angle)
  • type asin (type x)
  • type acos (type x)
  • type atan (type y, type x)
  • type atan (type y_over_x)
  • type sinh (type x)
  • type cosh (type x)
  • type tanh (type x)
  • type asinh (type x)
  • type acosh (type x)
  • type atanh (type x)
  • type pow (type x, type y)
  • type exp (type x)
  • type log (type x)
  • type exp2 (type x)
  • type log2 (type x)
  • type sqrt (type x)
  • type inversesqrt (type x)
  • type abs (type x)
  • type sign (type x)
  • type floor (type x)
  • type ceil (type x)
  • type trunc (type x)
  • type fract (type x)
  • type mod (type x, float y)
  • type modf (type x, out type i)
  • type min (type x, type y)
  • type max (type x, type y)
  • type clamp (type x, type minV, type maxV)
  • type mix (type x, type y, type a)
  • type step (type edge, type x)
  • type smoothstep (type a, type b, type x)
  • float length (type x)
  • float distance (type p0, type p1)
  • float dot (type x, type y)
  • vec3 cross (vec3 x, vec3 y)
  • type normalize (type x)
  • type faceforward (type N, type I, type Nref)
  • type reflect (type I, type N)
  • type refract (type I, type N,float eta)
  • float determinant(mat? m)
  • mat?x? outerProduct(vec? c, vec? r)
  • type matrixCompMult (type x, type y)
  • type inverse (type inverse)
  • type transpose (type inverse)
  • vec4 texture( sampler? , vec? coord [, float bias])
  • vec4 textureLod( sampler, vec? coord, float lod)
  • vec4 textureLodOffset( sampler? sampler, vec? coord, float lod, ivec? offset)
  • vec4 textureGrad( sampler? , vec? coord, vec2 dPdx, vec2 dPdy)
  • vec4 textureGradOffset sampler? , vec? coord, vec? dPdx, vec? dPdy, vec? offset)
  • vec4 textureProj( sampler? , vec? coord [, float bias])
  • vec4 textureProjLod( sampler? , vec? coord, float lod)
  • vec4 textureProjLodOffset( sampler? , vec? coord, float lod, vec? offset)
  • vec4 textureProjGrad( sampler? , vec? coord, vec2 dPdx, vec2 dPdy)
  • vec4 texelFetch( sampler? , ivec? coord, int lod)
  • vec4 texelFetchOffset( sampler?, ivec? coord, int lod, ivec? offset )
  • vec? textureSize( sampler? , int lod)
  • type dFdx (type x)
  • type dFdy (type x)
  • type fwidth (type p)
  • type isnan (type x)
  • type isinf (type x)
  • float intBitsToFloat (int v)
  • uint uintBitsToFloat (uint v)
  • int floatBitsToInt (float v)
  • uint floatBitsToUint (float v)
  • uint packSnorm2x16 (vec2 v)
  • uint packUnorm2x16 (vec2 v)
  • vec2 unpackSnorm2x16 (uint p)
  • vec2 unpackUnorm2x16 (uint p)
  • bvec lessThan (type x, type y)
  • bvec lessThanEqual (type x, type y)
  • bvec greaterThan (type x, type y)
  • bvec greaterThanEqual (type x, type y)
  • bvec equal (type x, type y)
  • bvec notEqual (type x, type y)
  • bool any (bvec x)
  • bool all (bvec x)
  • bvec not (bvec x)

Childhood intelligence in relation to major causes of death in 68 year follow-up

www.bmj.com - Comments

Abstract

Objectives To examine the association between intelligence measured in childhood and leading causes of death in men and women over the life course.

Design Prospective cohort study based on a whole population of participants born in Scotland in 1936 and linked to mortality data across 68 years of follow-up.

Setting Scotland.

Participants 33 536 men and 32 229 women who were participants in the Scottish Mental Survey of 1947 (SMS1947) and who could be linked to cause of death data up to December 2015.

Main outcome measures Cause specific mortality, including from coronary heart disease, stroke, specific cancer types, respiratory disease, digestive disease, external causes, and dementia.

Results Childhood intelligence was inversely associated with all major causes of death. The age and sex adjusted hazard ratios (and 95% confidence intervals) per 1 SD (about 15 points) advantage in intelligence test score were strongest for respiratory disease (0.72, 0.70 to 0.74), coronary heart disease (0.75, 0.73 to 0.77), and stroke (0.76, 0.73 to 0.79). Other notable associations (all P<0.001) were observed for deaths from injury (0.81, 0.75 to 0.86), smoking related cancers (0.82, 0.80 to 0.84), digestive disease (0.82, 0.79 to 0.86), and dementia (0.84, 0.78 to 0.90). Weak associations were apparent for suicide (0.87, 0.74 to 1.02) and deaths from cancer not related to smoking (0.96, 0.93 to 1.00), and their confidence intervals included unity. There was a suggestion that childhood intelligence was somewhat more strongly related to coronary heart disease, smoking related cancers, respiratory disease, and dementia in women than men (P value for interactions <0.001, 0.02, <0.001, and 0.02, respectively).Childhood intelligence was related to selected cancer presentations, including lung (0.75, 0.72 to 0.77), stomach (0.77, 0.69 to 0.85), bladder (0.81, 0.71 to 0.91), oesophageal (0.85, 0.78 to 0.94), liver (0.85, 0.74 to 0.97), colorectal (0.89, 0.83 to 0.95), and haematopoietic (0.91, 0.83 to 0.98). Sensitivity analyses on a representative subsample of the cohort observed only small attenuation of the estimated effect of intelligence (by 10-26%) after adjustment for potential confounders, including three indicators of childhood socioeconomic status. In a replication sample from Scotland, in a similar birth year cohort and follow-up period, smoking and adult socioeconomic status partially attenuated (by 16-58%) the association of intelligence with outcome rates.

Conclusions In a whole national population year of birth cohort followed over the life course from age 11 to age 79, higher scores on a well validated childhood intelligence test were associated with lower risk of mortality ascribed to coronary heart disease and stroke, cancers related to smoking (particularly lung and stomach), respiratory diseases, digestive diseases, injury, and dementia.

Introduction

Findings from prospective cohort studies based on populations from Australia, Sweden, Denmark, the US, and the UK indicate that higher cognitive ability (intelligence) measured with standard tests in childhood or early adulthood is related to a lower risk of total mortality by mid to late adulthood.1 The association is evident in men and women1 2; is incremental across the full range of ability scores2 3; and does not seem to be confounded by socioeconomic status of origin or perinatal factors.1 4 5 Whereas similar gradients are also apparent for selected causes of death, such as cardiovascular disease,3 6 7 8 9 10 suicide,7 10 11 12 13 14 15 and injuries,7 16 17 the association with other leading causes remains inconclusive or little tested. Mortality surveillance for the entire population of one country born in 1936 who had an assessment of childhood intelligence provides the valuable opportunity to examine little tested associations between intelligence and mortality and consider specificity by exploring the strengths of these associations according to leading causes of death, in men and women, and over almost the entire life course.

Several hypotheses have been proposed to explain associations between intelligence and later risk of mortality.18 The suggested causal mechanisms put forward, in which cognitive ability is the exposure and disease or death the outcome, include mediation by adverse or protective health behaviours in adulthood (such as smoking, physical activity), disease management and health literacy, and adult socioeconomic status (which could, for example, indicate occupational hazards).18 Recent evidence of a genetic contribution to the association between general cognitive ability and longevity,19 however, might support a system integrity theory that posits a “latent trait of optimal bodily functioning” proximally indicated by both cognitive test performance and disease biomarkers.20 None of these possibilities are mutually exclusive. Whereas cognitive epidemiology18 makes a unique contribution to improved understanding of health inequalities in populations, by its successful application of a well validated behavioural trait that performs independently of social gradients in its association with health indices,21 22 there remains a fundamental question regarding how specific and multifaceted the link is between individual differences in intelligence and longevity. Evidence for an association with several leading causes of death has either not been replicated (dementia and respiratory disease),7 23 is conflicting (different cancer sites),5 7 8 24 25 26 27 or is hitherto untested (digestive system disease). At least six publications have compared associations between premorbid intelligence and a selection of cause specific mortalities,5 7 8 10 24 28 including some well characterised and extremely large cohorts with extensive follow-up; however, over-representation of men only samples7 10 and follow-up that was terminated in middle age5 10 has limited the generalisability of findings. Furthermore, low numbers of events for diseases common in older adult populations could have contributed to some apparently conflicting results.8 24 28

We investigated the magnitudes of the association between childhood intelligence and all major causes of death, using a whole year of birth population followed up to older age, therefore capturing sufficient numbers of cases for each outcome. Secondly, we investigated sex differences in the associations. Thirdly, we carried out sensitivity analyses to test for some possible mechanisms of association, including confounding and mediation by socioeconomic status.

Methods

Study population and data sources

In this prospective cohort study, all individuals born in Scotland in 1936 and registered at school in Scotland in 1947 were targeted for tracing and subsequent data linkage to death certificates. This was carried out by the National Records of Scotland, with permission from the registrar general of Scotland, using the National Health Service (NHS) central register for members traceable in Scotland, and the MRIS Integrated Database and Administration System for those in England and Wales. The confidentiality advisory group of the health research authority gave support under section 251 of the NHS Act 2006 for linkage without consent of mortality records including cause of death with intelligence scores age 11 from the Scottish Mental Survey 1947 (SMS1947).25 29 30

Childhood intelligence

On 4 June 1947, about 94% of the Scottish population born in 1936 who were registered as attending school in Scotland (75 252) completed a test of general intelligence in the SMS1947 (n=70 805).25 29 31 This involved administration of the Moray House test No 12, which has 71 items tapping verbal and non-verbal reasoning ability.29 The test was introduced by school teachers who read aloud instructions before the start of a 45 minute test period. Each participant was allocated a score out of the maximum of 76 (some items scored more than one point). A recent follow-up study showed that the test had good concurrent validity (correlation coefficient about 0.8) with a well standardised individually administered Stanford version of the Binet test of intelligence in 194725 29 and with performance on Raven’s progressive matrices—a widely used non-verbal ability test (correlation coefficient about 0.7).32 The test has high lifelong stability of individual differences,33 34 and several studies have provided good evidence for its external validity.28 35 36

Cause specific death outcomes

Causes of death were coded according to the ICD-6-10 (international classification of diseases, 6th to 10th revisions) codes (see table A in appendix for codes). NHS (England and Wales) recorded deaths linked to cause of death data were provided for dates up to and including 31 December 2015, and respective data for NHSCR (Scottish) recorded deaths were provided up to 30 June 2015. Subsequent cause of death updates provided by NHSCR, on a quarterly basis up to 31 December 2015, were provided without unique ID numbers for individuals, and we therefore applied a matching process to link these to SMS1947 data (appendix).

Patient involvement

No cohort participants contributed to the development of the present research question or the outcome measures, nor were they involved in the design, recruitment, or conduct of the study. There is no intention to disseminate the results of this electronic data linkage study to its participants.

Statistical analysis

The analytic sample includes 65 765 sample members who were traced for linkage to mortality records and who had a census date (that is, a last known GP registration date for those who emigrated or joined the armed forces) and an intelligence test score from 1947 (fig 1). They represent 92.3% of those who took part in the SMS1947 and 87.4% of Scottish schoolchildren born in 1936. Exclusions because of missing intelligence test scores affected 6.8% (5103) of the total 1936 Scotland birth cohort, which has previously been reported,25 31 and 6.6% (4744) of those who were successfully linked and had a census date.

Figure1

Fig 1 Derivation of Scottish Mental Survey 1947 (SMS1947) analytic sample. Most of those with missing census dates had migrated or joined armed forces for whom records contained no last known GP registration date. *Analytic sample represents 92% of survey. †46.0% of those with vital status had died, which is approximate to Scottish population statistics for those aged 75-80. ‡Includes 5381 who emigrated abroad (2857 men); 3391 cancelled from GP registration (1383 men); 454 armed forces recruits (411 men); 96 residents of Northern Ireland or Isle of Man (52 men)

We conducted analyses with SPSS Statistics 21, unless otherwise stated. Cox proportional hazards regression models produced hazard ratios with 95% confidence intervals to summarise the relation between childhood intelligence and each cause of death, adjusted for age (in days) at cognitive testing and sex (dummy variable), for which there were no missing data. We ran models in which the cause of death was cardiovascular disease, coronary heart disease, stroke, cancer by type, respiratory disease, digestive disease, externally caused, injury, suicide, or dementia. Given that suicide is the likely cause of death in most death certificates that state “open verdict” or “undetermined intent,”37 we reported not only on models to predict deaths formally recorded as suicide but added deaths of undetermined intent to models of suicide. A cause of death was included if it was listed among multiple causes of death (MCOD) on the death certificate, but we repeated each model in which the endpoint was the underlying cause of death38 (see appendix).

In survival analyses, the time scale was calendar time (days) from the date of the intelligence test (4 June 1947) to the census date, which was the earliest of date of death; last known date of GP registration (for migration or armed forces entrants); or 31 December 2015 (end of follow-up). Members who died from a different cause to that being modelled were included in the denominator (people at risk) and censored at date of death, as is standard practice in epidemiological analyses. The proportional hazards assumption was assessed by inspection of log−log plots and formally tested in Stata 14 with the Schoenfeld residuals test. The assumption held for associations between intelligence and all major causes of death according to both methods, with the exception of deaths related to cancer with log−log plots and deaths from injury under formal testing (P=0.03). To visually inspect associations for linearity, we graphically plotted hazard ratios and their 95% confidence intervals to show the risk of each event type in accordance with intelligence test score in 10ths (for >1000 numbers of deaths) or quarters (for <1000 numbers of cases). For our main results, we estimated hazard ratios and their 95% confidence intervals for cause specific mortality according to a 1 SD (about 15 points) advantage in intelligence test score. With the suggestion that some associations between intelligence and death are modified by sex, we formally tested for sex differences by including an interaction term (sex × intelligence score) in models predicting cause of death. We then produced effect estimates separately for men and women. Finally, to adjust for multiple testing, using R,39 we made false discovery rate correction to significant P values resulting from all models40 and report on significant models (P<0.05) that failed this correction.

Assessing selection bias caused by missing intelligence test data

We examined whether the 6.8% missing data on the intelligence test in the SMS1947 caused selection bias that affected the magnitude of the association between intelligence and mortality. We conducted sensitivity analyses using a representative subsample (1.7%; n=1208) of the SMS1947, the so called “6 day sample.”25 37 38 This subsample was affected by a similar degree of missing data as the whole SMS1947 on the Moray House test; however, they all took an additional, individually administered intelligence test on another occasion (see appendix for more details).

Confounder adjustment: adjustment for school

In the absence of individual level data on socioeconomic status in the full SMS1947, we used school attended as a proxy measure to adjust for potential confounding by background socioeconomic status. A Scottish cohort study has shown that primary school has moderate correlation with paternal social class.41 Schools were specified as strata in the models for major causes of death and cancer subtypes, and fixed effects models adjusted for all characteristics shared by pupils from the same school were. As children could be selected into some schools because of previous cognitive ability, however, there might be over-adjustment in this particular sensitivity analysis.

Confounder adjustment: subgroup analyses

We repeated models for leading causes of death in association with childhood intelligence scores in a representative subsample of the SMS1947, the so called “30 day sample” (7.2%; n=5083), on whom more background data were available to test for potential confounding (see appendix for more information). These data included socioeconomic status variables (paternal occupational status, home overcrowding, and school absenteeism) and indicators of physical status (height and physical disability).

Adjustment for adult socioeconomic status and smoking: replication sample

In the absence of data in the intermediary “lifetime” period of the study with which to test for potential mediation, we report on equivalent models run on a replication sample—the “West of Scotland Twenty-07 study”—with and without adjustment for adult smoking status and occupational status. This cohort shares similarities to the SMS1947, including Scotland derived 1930s birth cohort and linkage to mortality records to at least 2015 (see appendix).

Results

Characteristics of the sample

Among 75 252 members of the 1936 birth cohort, 3242 were untraceable, 630 had no census dates (that is, migrated without a last known GP registration date), 4744 had no data on childhood intelligence, and 871 were mismatches of the linkage (fig 1). Among 65 765 individuals (32 229 were women; 5182 attended different schools) comprising the analytic sample, 25 979 had died and 30 464 were confirmed to be living in the UK at follow-up. Mean age at death was 66.1 (SD 10.6); mean time to follow-up was 57.0 (SD 18.4) years, and total person years of follow-up was 3.54 million. There were no marked differences in the characteristics of the analytic sample and the remainder of the SMS1947, except that, among those with a childhood mental test score, the analytic sample scored on average higher on the Moray House test (mean 37.1, SD 15.7) than those excluded from the analyses (35.4, SD 16.4; P<0.001).

Childhood intelligence in relation to cause specific mortality

Figure 2 shows the associations between childhood intelligence (by 10ths or quarters) and risk of the major causes of death. These show that, for most endpoints, associations were inverse—that is, higher childhood intelligence was associated with a lower risk of cause specific death. Risk of death related to lifetime respiratory disease was two thirds lower in the top performing 10th for childhood intelligence versus the bottom 10th. Furthermore for deaths from coronary heart disease, stroke, smoking related cancers, digestive diseases, and external causes, risk of mortality was halved for those in the highest versus lowest 10th of intelligence. The risk of dementia related mortality and deaths by suicide were reduced by at least a third in the highest performing quarter of intelligence test score versus the lowest quarter. There was no evident association between childhood intelligence and mortality from cancers not related to smoking.

Figure2

Fig 2 Association between intelligence (Moray House test score) at age 11 and major causes of death (age and sex adjusted hazard ratios and 95% confidence intervals) to age 79 in Scottish Mental Survey 1947 (SMS1947)

Figure 3 shows the risk of major causes of death associated with a 1 SD higher score in childhood intelligence (see fig A in appendix for the equivalent plot where the outcome was underlying cause of death rather than MCOD). All hazard ratios showed inverse associations, and most showed narrow confidence intervals. The strongest effect sizes were seen for respiratory disease (hazard ratio 0.72, 95% confidence interval 0.70 to 0.74), coronary heart disease (0.75, 0.73 to 0.77), and stroke (0.76, 0.73 to 0.79). All others fell within the hazard ratio range 0.80-0.87, except for deaths from cancers not related to smoking cancer (0.96, 0.93 to 1.00)—the significance of this latter effect did not survive correction for false discovery rate. The relatively wide confidence interval for deaths from intentional self harm reflects the lower number of cases for this endpoint (0.87, 0.74 to 1.02). When deaths of undetermined intent were included in the model for suicide, the confidence interval fell below 1 (0.84, 0.74 to 0.96; 220 cases).

Figure3

Fig 3 Hazard ratios (95% confidence intervals) for association between 1 SD higher score in intelligence test score at age 11 and cause of death to age 79 in 65 765 people in Scottish Mental Survey 1947 (SMS1947). Effect sizes are adjusted for sex and age at cognitive testing

Sex interaction effects were computed with childhood intelligence in the total sample, and sex specific hazard ratios were also estimated for the associations between childhood intelligence and leading causes of death (table B in appendix). In models of the total sample, interaction terms for sex × intelligence were significant for risk of cardiovascular disease (P<0.001), coronary heart disease (P<0.001), smoking related cancers (P=0.02), respiratory diseases (P<0.001), and dementia (P=0.02). Though the inverse patterns of the association between intelligence and mortality observed for the total sample were seen in both men and women, the magnitude of the associations were slightly greater among women than among men for most cause specific deaths. The greatest difference in effect estimate was for dementia, whereby a 1 SD higher score in childhood intelligence was associated with a 10% reduced risk of death from dementia in men and a 24% reduced risk in women. Otherwise, between men and women there were equivalent hazard ratios for deaths from cancer not related to smoking and external causes of death, except for deaths by intentional self harm, which were associated with childhood intelligence in men (hazard ratio 0.80, 95% confidence interval 0.66 to 0.96) but not women (1.15, 0.82 to 1.60). This sex difference was also evident for suicide when we included open verdict deaths (0.76, 0.66 to 0.89, in men; 1.05, 0.83 to 1.34, in women; P=0.03 for sex × intelligence interaction effect in the total sample).

Childhood intelligence in association with death by specific cancer type

Figure 4 shows the associations between groupings of childhood intelligence score (10ths or quarters) and deaths related to 15 specific cancers. About half of these showed inverse patterns of association with a degree of linearity, including cancers of the oesophagus, colon or rectum, stomach, liver, lung, kidney, bladder, and blood. The strongest association was evident for death related to lung cancer: the risk in the highest performing 10th of childhood intelligence was reduced by two thirds compared with the lowest performing 10th. Cancers showing negligible or irregular associations with childhood mental ability included mouth, pancreas, skin, ovaries, breast (women only), prostate (men only), and brain or central nervous system.

Figure4

Fig 4 Association between intelligence (Moray House test score) at age 11 and deaths from specific cancers (age and sex adjusted hazard ratios and 95% confidence intervals) to age 79 in Scottish Mental Survey 1947 (SMS1947)

Figure 5 shows the strengths of association between a 1 SD higher score in childhood intelligence and risk of death related to 15 specific cancers, with total cancers as a comparator (see fig B in appendix for equivalent forest plot reflecting underlying cause of death cases only). Whereas the effect estimate for death by any cancer clearly shows an inverse association with narrow confidence intervals (hazard ratio 0.86, 95% confidence interval 0.84 to 0.88), there is obvious heterogeneity across the results for specific cancer sites. Among the eight specific cancers that showed linearity in their inverse association with childhood mental ability in figure 4, the plot shows robust associations for seven of these (oesophageal, colorectal, stomach, liver, lung, bladder, and lymphoid or haematopoietic), with hazard ratios in the range 0.75-0.91. The strongest associations with childhood mental ability were deaths associated with lung cancer (0.75, 0.72 to 0.77) and stomach cancer (0.77, 0.69 to 0.85).

Figure5

Fig 5 Hazard ratios (95% confidence intervals) for association between 1 SD higher score in intelligence at age 11 and cause of death by cancer type to age 79 in Scottish Mental Survey 1947 (SMS1947). Effect sizes are adjusted for age at intelligence testing and sex, with exception of ovarian and breast cancer (women only) and prostate cancer (men only). *Non-smoking-related cancers (all others are smoking related)

We included an interaction term that included sex and childhood intelligence in models for IQ and specific cancer related mortality in the total sample and calculated sex specific hazard ratios (table C in appendix). These analyses showed no clear sex differences in direction or magnitude of the association between childhood intelligence and cancer specific mortality. The significant sex interaction effect (P=0.013) with mental ability in predicting lung cancer in the total sample could be a chance finding, and the direction of association was the same in men (hazard ratio 0.77, 95% confidence interval 0.74 to 0.81) and women (0.70, 0.66 to 0.75).

Sensitivity analyses: assessment of selection bias caused by missing intelligence test data

In analyses of the so called “6 day sample” of the SMS1947 (that is, those born on the first day of the even numbered months of 1936), selection bias according to missing Moray House test scores in the SMS1947 did not affect the effect size of the association between intelligence test scores (on the Terman-Merrill, an individually administered test) and total mortality (see appendix).

Sensitivity analyses: assessment of potential confounding

Firstly, we attempted to control for potential confounding by adjusting for the fixed effects of school attended in the full SMS1947. The hazard ratios for cause specific mortality and for deaths related to cancer subtype mostly showed a reduction by a small amount (attenuation range 0-30%) (see figs C and D in appendix), and the patterns of specificity were similar to those in the main analyses.

Secondly, in subgroup analysis of the SMS1947, including 2039 men and 1992 women in the 30 day sample, hazard ratios and their 95% confidence intervals showed inverse associations between childhood intelligence and all cause mortality and mortality related to cardiovascular disease, cancer (smoking related), respiratory disease, and digestive disease (but not cancers not related to smoking, external causes, and dementia, unlike in the total sample) (see table D in appendix). Respiratory disease held the strongest association with childhood intelligence (hazard ratio 0.73, 95% confidence interval 0.66 to 0.80), as in the full sample. In confounder adjusted models that included three indicators of childhood socioeconomic status, the relation between intelligence and all cause mortality or mortality from cardiovascular disease, any cancer, smoking related cancer, respiratory disease, or digestive disease was attenuated by 7-26%. Addition of physical status to the model had modest impact (attenuation range 10-26%).

Sensitivity analyses: assessment of mediation

In a replication sample—the West of Scotland twenty-07 study—we observed similar hazard ratios to the SMS1947 for the associations between intelligence (tested in middle age) and specific underlying causes of death (fig C in appendix), though they were slightly stronger and with less precision. The one noticeable difference was for stroke as the underlying cause of death, which showed a weak association with midlife intelligence (hazard ratio 0.90, 95% confidence interval 0.70 to 1.15). After we adjusted for smoking, adult socioeconomic status, and self rated health, there were attenuation effects of 27-65% (table E in appendix), and associations remained the most robust for respiratory disease (0.77, 0.59 to 0.99) and coronary artery disease (0.79, 0.65 to 0.96).

Discussion

In this first whole population birth cohort study linking childhood intelligence test scores to cause of death, in a follow-up spanning age 11-79, we found inverse associations for all major causes of death, including coronary heart disease, stroke, cancer, respiratory disease, digestive disease, external causes of death, and dementia. For specific cancer types the results were heterogeneous, with only smoking related cancers showing an association with childhood ability. In general, the effect sizes were similar for women and men (albeit marginally greater for women), with the exception of death by suicide, which had an inverse association with childhood ability in men but not women. In a representative subsample with additional background data, there was evidence that childhood socioeconomic status and physical status indicators had no more than a modest confounding impact on the observed associations. A replication study with adult data on smoking and socioeconomic status showed less than a quarter to two thirds percentage attenuation of the equivalent effect estimates for intelligence test performance in association with cause specific mortality.

Strengths and weaknesses of this study

One weakness of the study is the absence of covariate data from childhood that could account for the effect estimates we observed in our main analysis. This lack of data, however, should be balanced with evidence from our sensitivity analyses, including a representative subsample of the cohort. Additionally, there are extensive reports from nearly all published studies that have attempted to control for potential confounding. In our own analysis of the whole SMS1947, adjustment for school led to little attenuation of the associations between intelligence and mortality. Because this is only a proxy for background socioeconomic status, and because it could lead to over-adjustment as some schools select on cognitive ability, however, we undertook further sensitivity analyses. In our own analysis of a representative subsample of the SMS1947, three indicators of background socioeconomic status, which are significant correlates of intelligence test scores, explained a quarter or less of the associations between intelligence and cause specific mortality. This is consistent with the main literature that has reported low attenuation effects in models of premorbid intelligence and all cause (such as in the meta-analysis by Calvin and colleagues1) or cause specific mortality, including cardiovascular disease,4 5 10 42 43 44 cancer,5 26 and external causes,5 10 45 including suicide10 15 46 and death from injury.10 16 17 Additional adjustments for perinatal factors and/or physical status indicators also show marginal (if any) attenuation effects,4 5 as we observed here in our adjustment of height and physical disability. Other psychological factors that have been shown to correlate with intelligence test performance, including personality traits and mental health indicators, could yet partially explain these associations. In a previous paper on the 6 day sample of the SMS1947, however, we found that the personality trait of “dependability” (rated in adolescence and similar to the widely studied trait called conscientiousness), which was also predictive of mortality, did not account for the association with intelligence.47 In a different sample, the personality trait of neuroticism was shown to moderate, but not attenuate, the association between intelligence and death.48 The strength of some of the associations we reported, however, such as with respiratory disease, are perhaps too strong to be explained by any potential confounders we are aware of from the literature.

Another factor affecting validity of findings is historical inaccuracy of death certificates. Particular causes of death that could be under-reported by coroners include dementia,49 stroke,50 and suicide.37 Therefore, we judge that the greater chance of false negatives relative to false positives in the present study would act, if at all, to slightly attenuate the observed effect sizes. Indeed, a further analysis, in which we included “open verdict” deaths in the suicide model, increased the hazard ratio and narrowed its confidence interval.

Strengths of the present study include its analysis of a near complete year of birth cohort from a nation’s population, with a high return of linkage to all historical death records. The follow-up period from childhood through to older age enabled us to consider all major causes of death in the UK across the life course. Not only can we reproduce more accurately, and with low bias, results from previous cohorts on cardiovascular disease and external causes of death, but we have been able to make a substantial contribution to the literature on premorbid intelligence and risk of cancer and report for the first time on associations between childhood intelligence and death from diseases of the digestive system. Furthermore, our whole population sample of men and women enables the reporting of sex specific effects, which are often lacking from the literature, which draws heavily on data from male conscripts. For example, we have been able to report for the first time on the association between premorbid intelligence and the full range of subtypes of cancer in women. And we report for the first time on somewhat greater effect sizes among women relative to men for several causes of death, including respiratory disease and smoking related cancers.

Comparison with other studies

An important aspect of the present study was to consider the role of premorbid intelligence in relation to specific cancer sites, given the considerable heterogeneity in cancer aetiology and conflicting findings from previous studies of an association between premorbid intelligence and risk of total cancer mortality.5 7 8 24 25 26 27 A cohort study of more than a million Swedish male conscripts—the largest to date—reported that, among 20 incident cancers, only three had significant associations with premorbid intelligence.27 The authors concluded that, with the likelihood of a few statistical false positives, cancer was unlikely to explain the association between premorbid intelligence and total mortality risk. Based on evidence from the present study, and other comparatively smaller cohorts, however, we would caution against over-generalising this summary. Our results are consistent with previously reported effect sizes for premorbid intelligence in association with lung cancer5 7 8 24 and stomach cancer.8 24 27 Furthermore, we have shown, perhaps for the first time, that an association between previous intelligence and risk of cancer might be specific to types of smoking related cancer. We found significant inverse associations for five other smoking related cancers: oesophageal, colorectal, liver, bladder, and haematopoietic. Whereas two previous studies reported null findings for risk of colorectal and pancreatic cancer in association with childhood intelligence, there were low case numbers.8 24 Indeed in a recent study of 728 165 Danish men, intelligence in early adulthood was significantly inversely associated with total gastrointestinal cancers.7 A pertinent question remains why more types of smoking related cancer were not related to premorbid intelligence in the study of a million Swedish men.27 This could be because of their much younger age at death ascertainment and/or national differences in prevalence of cigarette smoking. Sweden has been exemplary in its reduction in cigarette use (particularly among men) since the late 1960s,51 and maximum prevalence rates of smoking in Sweden at the time when the Swedish conscript study was conducted would have been much lower52 than equivalent rates in Scotland during our follow-up.53 If smoking is less prevalent in the Swedish population, and if smoking is an important mediator of the association between premorbid intelligence and risk of cancer, then this might explain the reduced and non-significant effect sizes reported by Batty and colleagues.27 One notable and significant finding from Sweden was a positive association between intelligence at conscription and incident skin cancer, indicating that higher ability was associated with an increased risk.27 Although the pattern of this relation in our own data corroborates this result, we did not observe a significant effect, perhaps because our older cohort grew up in a time before cheap flights to hot countries became ubiquitous and the proliferation in risk of skin cancer. The literature on premorbid intelligence and other cancer types is relatively small and/or underpowered. We report for the first time on the association with ovarian cancer in women, which we found to be null, and we replicate findings of previous studies of null associations with breast cancer in women5 8 24 and prostate cancer in men.27

Preliminary evidence of an inverse association between childhood intelligence and risk of dementia related death was reported from case-control studies of the 1932 Scottish Mental Survey that were linked to incident dementia records.54 55 A recent prospective cohort study of this sample, followed up to 92 years, validated the finding.23 Our findings are consistent with this, as is evidence of a greater association with risk of dementia related mortality among women than men, also observed in the older Scottish cohort.23 This sex differential is also supported by evidence from individual participant meta-analyses showing that early school leaving in childhood is associated with death from dementia in women and not men.56

Mortality related to respiratory disease had the strongest association with premorbid intelligence in the present study (hazard ratio 0.72, 95% confidence interval 0.70 to 0.74). In a recent study of male conscripts in Denmark, which was the only previous study to report on the association, mortality from respiratory disease also had one of the strongest associations with premorbid intelligence (0.62, 0.60 to 0.65)7—only homicide was stronger in its association. Previous evidence that premorbid intelligence is associated with self reported chronic lung disease by midlife57 58 and measured lung function in later life59 60 61 62 supports these results. In contrast, a Scottish study reporting a weak and non-significant inverse association with incident respiratory disease might have been underpowered.8

Our own effect sizes for risk of mortality in association with premorbid intelligence corroborate those reported in the literature for all cause mortality (hazard ratio 0.80 versus aggregate hazard ratio of 0.80 in studies of ≥40 years’ duration),1 total cardiovascular disease (0.76 versus hazard ratios of 0.74-0.93),7 8 9 10 63 and coronary heart disease (0.75 versus hazard ratios of 0.70-0.86).3 4 7 9 42 43 64 The effect sizes for risk of stroke in association with premorbid intelligence have been less consistent in the literature, with significant inverse associations for incident or fatal only stroke observed in cohorts from Denmark, Scotland, Sweden, and the US3 4 7 65 and negligible associations reported in separate cohorts from Denmark and Scotland.8 9 42 Weaker associations with stroke in previous studies are probably influenced by low numbers of stroke cases in cohort studies, driven in part by historical under-reporting of stroke in death certificates50 and/or relatively low numbers of ischaemic versus haemorrhagic strokes,7 particularly in cohort studies with follow-up in younger age groups. In our study cohort followed to older age, the observed effect size for risk of fatal stroke was equivalent in magnitude to that for coronary heart disease.

Our findings accord with previous reports of inverse associations between premorbid intelligence and risk of externally caused deaths, specifically from Danish and Scottish cohorts.5 7 45 In the Scottish study that followed up on deaths from ages 15 to 57, external causes of death held the strongest association with childhood intelligence relative to cancer, cardiovascular disease, and total mortality.5 In contrast, we found that external causes of death held the weaker association when compared with mortality from cardiovascular disease. It could be that differences in the composition of specific types of death that are externally caused in a younger versus older age cohort could influence the magnitude of this effect. Moreover, the hazard ratio we report for death from injury in association with childhood intelligence (0.81, 95% confidence interval 0.75 to 0.86) is similar in magnitude to that is a study of Danish men born in 1953, intelligence tested at age 12, and followed up to age 48 (0.82, 0.78 to 0.86).16 On the other hand, in studies where intelligence was tested in late childhood or early adulthood, the effect size in relation to death from injury has been greater (0.71 and 0.76, respectively).7 16 17 Several high powered prospective studies of cohorts of male conscripts have reported inverse associations between premorbid intelligence and later risk of suicide, including studies from Australia,11 Denmark,7 12 and Sweden.10 13 14 15 Far less is reported on the association in women and with follow-up to older ages. Our findings, however, are similar to the only previous study we are aware of to report on sex differentials for incident suicide risk in association with premorbid intelligence in a 40 year follow-up study, where an association was evident in men (age adjusted odds ratio of 0.90, 95% confidence interval 0.83 to 0.99) but negligible in women (1.04, 0.90 to 1.20).66

Explanations and implications for future research

The strongest associations we observed were for natural causes of death related to coronary heart disease, stroke, respiratory disease, lung cancer, and stomach cancer. Smoking is a modifiable risk factor in each of these diseases, and higher premorbid intelligence has been related to lower likelihood of current smoking67 or past regular smoking68 and the increased likelihood of quitting smoking.68 69 Indeed, in a recent comparison of effect sizes for intelligence and cause specific mortalities in Danish men, the authors made a similar summary of their own findings by implicating smoking as an important mediator in the context of natural causes of death.7 Nevertheless, when tested, smoking is no more than a partial mediator in associations between premorbid intelligence and total mortality,3 mortality related to coronary heart disease,63 and incident coronary heart disease or stroke.64 In our replication study, the addition of adult smoking showed small attenuating effects (12-27%) on the associations between midlife performance on intelligence tests and most cause specific mortalities, including coronary artery disease and respiratory disease, and had moderate attenuation effects on the associations with smoking related cancers and lung cancer (40% and 50%, respectively). Educational attainment or indicators of occupational status in adulthood have generally shown stronger attenuating effects.1 Though education is likely to fall on the causal pathway, after intelligence and before death, given evidence that differences in intelligence predict later outcomes of national school examinations,70 education might alternatively be a proxy indicator of intelligence and yet have no direct causal association with longer life. This is an ongoing discussion about possible over-adjustment by education in cognitive epidemiology.71 It is likely that the influence of a health behaviour (or pattern of behaviours) in explaining associations between premorbid function and cause specific mortality will be more precisely measured by future studies with repeat assessment across the life course, capturing cumulative risk. Indeed, follow-up of the Whitehall II study showed that four repeat measures of lifestyle behaviours accounted for far more of the socioeconomic-mortality gradient than baseline measurement alone,72 albeit this difference was evident for specific behaviours (especially physical activity) and not all (such as smoking). Alcohol consumption might have accounted for some of the association we observed between premorbid intelligence and digestive related mortality, and this could be similarly assessed. Alcohol related deaths were associated with premorbid intelligence in a 37 year follow up of the 1969-70 Swedish conscripts cohort.73 Childhood performance on intelligence tests was most strongly associated with deaths from respiratory disease. This might be attributable to a combination of prevalence of smoking and occupational related exposures. For many in this birth cohort, those in the working classes spent their early careers in mining and shipbuilding industries (men) or as cleaners and factory workers (women). Therefore, it is possible that, with the decline in rates of smoking and toxic exposures in the workplace, the magnitude of associations between IQ and death could be lower in more recent birth cohorts. Nevertheless, when we adjusted for background social class in our subgroup analyses, and for smoking and adult occupational status in our replication study, the risk of deaths from respiratory disease in relation to intelligence scores was only partially attenuated (by 12% and 33%, respectively).

Whereas the research literature has taken advantage of often rich lifestyle data to explore mediating effects on a pathway from premorbid intelligence to risk of mortality, recent evidence from longitudinal data from twins suggests the association might be largely caused by common genetic effects.19 With newly emerging data from genome-wide association studies (GWAS), there is likely to be a sizeable shift towards the testing of potential genetic factors, alongside environmental factors, in cognitive epidemiology. Recent GWAS data used in the context of the large UK Biobank sample has provided evidence for pleiotropy between midlife (premorbid) cognitive performance and genetic variants of specific diseases, including coronary heart disease, ischaemic stroke, and Alzheimers’ disease.74 Whether this is evidence for biological pleiotropy (thus supporting bodily system integrity theory) or a causal pathway from genetic variant to disease outcome, mediated by cognitive ability and subsequent health risk behaviours and/or occupational hazards, is the subject of ongoing work with Mendelian randomisation methods.75 There is the additional prospect of interaction between health risk behaviours and genetic markers for disease in explaining health differentials attributed to variation in intelligence, and none of these possibilities is mutually exclusive.

Because of the widely different causes of death in our study, there is no assumption of a “one size fits all” explanation of the observed effects. The association between premorbid intelligence and specific external causes of death warrants in depth and separate coverage. The specific association between premorbid intelligence and risk of suicide in men, for example, might need to be included in risk models alongside other psychosocial factors, such as the modifying effects of psychosis66 and high achieving parents,46 and the potentially confounding and mediating effects of depression and anxiety disorders76 and antisocial personality.13 15 Furthermore, risk of suicide might be associated with premorbid intelligence through reduced opportunities in the job market, putting a greater strain on men than women in our society. Other cause specific mortalities, however, could be usefully studied together in their association with premorbid intelligence. For example, the underlying mechanisms in the association between premorbid intelligence and risk of mortality from dementia could have commonality with those of coronary heart disease and stroke. In support of this, previous evidence indicates that the effect on total dementia is driven by associations with vascular dementia and not Alzheimer’s disease.54 Furthermore, that risk of dementia mortality relates to premorbid intelligence for late onset but not early onset cases55—which tend to have a higher genetic component—might implicate health behaviours and lifestyle factors.

Conclusions

We have been able to report on the strength of association between premorbid intelligence and a range of specific causes of death in a full population and with follow-up from childhood to near the end of the life course. The specialty of cognitive epidemiology remains in a growth period; therefore it is premature to make recommendations to practitioners and policymakers, though we have made a start on that elsewhere.18 Although we report that smoking and socioeconomic status are unlikely to fully mediate the observed associations, future studies would benefit from measures of the cumulative load of such risk factors over the life course. Finally, we highlight to researchers active in this specialty our robust findings of the association between premorbid intelligence and lifetime risk of mortality from coronary heart disease, stroke, smoking related cancers, respiratory disease, digestive related disease, external causes, and dementia, specifically the relatively strong lifetime associations with smoking related diseases. We also emphasise the marginally greater associated risks among women for coronary heart disease, smoking related cancers, respiratory disease, and dementia, which might indicate sex differential effects of intelligence on modifiable risk factors of disease.

What is already known on this topic

  • Prospective cohort studies has shown that higher intelligence tested in childhood or early adulthood is related to greater longevity

  • This relation is replicated for deaths related to cardiovascular disease related and external causes, though evidence for the association with cancer related mortality is conflicting, and associations with other leading causes of death (such as respiratory and digestive diseases) are largely unknown.

  • Previous studies have mostly been in male conscripts followed up to middle adulthood

What this study adds

  • This study reported on the association between childhood intelligence in all men and women born in one year in one country and specific causes of death in older adulthood

  • Intelligence tested in childhood was inversely associated with leading lifetime causes of death, including coronary heart disease, stroke, cancer, external causes, respiratory disease, digestive related diseases, and dementia. The results for specific cancer sites were heterogeneous, and significant only for smoking related cancers

  • Lower intelligence in childhood was consistently associated with most leading causes of death in women, as well as in men as has previously been reported.

Who Goes Nazi? (1941)

harpers.org - Comments

It is an interesting and somewhat macabre parlor game to play at a large gathering of one’s acquaintances: to speculate who in a showdown would go Nazi. By now, I think I know. I have gone through the experience many times—in Germany, in Austria, and in France. I have come to know the types: the born Nazis, the Nazis whom democracy itself has created, the certain-to-be fellow-travelers. And I also know those who never, under any conceivable circumstances, would become Nazis.

It is preposterous to think that they are divided by any racial characteristics. Germans may be more susceptible to Nazism than most people, but I doubt it. Jews are barred out, but it is an arbitrary ruling. I know lots of Jews who are born Nazis and many others who would heil Hitler tomorrow morning if given a chance. There are Jews who have repudiated their own ancestors in order to become “Honorary Aryans and Nazis”; there are full-blooded Jews who have enthusiastically entered Hitler’s secret service. Nazism has nothing to do with race and nationality. It appeals to a certain type of mind.

It is also, to an immense extent, the disease of a generation—the generation which was either young or unborn at the end of the last war. This is as true of Englishmen, Frenchmen, and Americans as of Germans. It is the disease of the so-called “lost generation.”

Sometimes I think there are direct biological factors at work—a type of education, feeding, and physical training which has produced a new kind of human being with an imbalance in his nature. He has been fed vitamins and filled with energies that are beyond the capacity of his intellect to discipline. He has been treated to forms of education which have released him from inhibitions. His body is vigorous. His mind is childish. His soul has been almost completely neglected.

At any rate, let us look round the room.

The gentleman standing beside the fireplace with an almost untouched glass of whiskey beside him on the mantelpiece is Mr. A, a descendant of one of the great American families. There has never been an American Blue Book without several persons of his surname in it. He is poor and earns his living as an editor. He has had a classical education, has a sound and cultivated taste in literature, painting, and music; has not a touch of snobbery in him; is full of humor, courtesy, and wit. He was a lieutenant in the World War, is a Republican in politics, but voted twice for Roosevelt, last time for Willkie. He is modest, not particularly brilliant, a staunch friend, and a man who greatly enjoys the company of pretty and witty women. His wife, whom he adored, is dead, and he will never remarry.

He has never attracted any attention because of outstanding bravery. But I will put my hand in the fire that nothing on earth could ever make him a Nazi. He would greatly dislike fighting them, but they could never convert him. . . . Why not?

Beside him stands Mr. B, a man of his own class, graduate of the same preparatory school and university, rich, a sportsman, owner of a famous racing stable, vice-president of a bank, married to a well-known society belle. He is a good fellow and extremely popular. But if America were going Nazi he would certainly join up, and early. Why? . . . Why the one and not the other?

Mr. A has a life that is established according to a certain form of personal behavior. Although he has no money, his unostentatious distinction and education have always assured him a position. He has never been engaged in sharp competition. He is a free man. I doubt whether ever in his life he has done anything he did not want to do or anything that was against his code. Nazism wouldn’t fit in with his standards and he has never become accustomed to making concessions.

Mr. B has risen beyond his real abilities by virtue of health, good looks, and being a good mixer. He married for money and he has done lots of other things for money. His code is not his own; it is that of his class—no worse, no better, He fits easily into whatever pattern is successful. That is his sole measure of value—success. Nazism as a minority movement would not attract him. As a movement likely to attain power, it would.

The saturnine man over there talking with a lovely French emigree is already a Nazi. Mr. C is a brilliant and embittered intellectual. He was a poor white-trash Southern boy, a scholarship student at two universities where he took all the scholastic honors but was never invited to join a fraternity. His brilliant gifts won for him successively government positions, partnership in a prominent law firm, and eventually a highly paid job as a Wall Street adviser. He has always moved among important people and always been socially on the periphery. His colleagues have admired his brains and exploited them, but they have seldom invited him—or his wife—to dinner.

He is a snob, loathing his own snobbery. He despises the men about him—he despises, for instance, Mr. B—because he knows that what he has had to achieve by relentless work men like B have won by knowing the right people. But his contempt is inextricably mingled with envy. Even more than he hates the class into which he has insecurely risen, does he hate the people from whom he came. He hates his mother and his father for being his parents. He loathes everything that reminds him of his origins and his humiliations. He is bitterly anti-Semitic because the social insecurity of the Jews reminds him of his own psychological insecurity.

Pity he has utterly erased from his nature, and joy he has never known. He has an ambition, bitter and burning. It is to rise to such an eminence that no one can ever again humiliate him. Not to rule but to be the secret ruler, pulling the strings of puppets created by his brains. Already some of them are talking his language—though they have never met him.

There he sits: he talks awkwardly rather than glibly; he is courteous. He commands a distant and cold respect. But he is a very dangerous man. Were he primitive and brutal he would be a criminal—a murderer. But he is subtle and cruel. He would rise high in a Nazi regime. It would need men just like him—intellectual and ruthless. But Mr. C is not a born Nazi. He is the product of a democracy hypocritically preaching social equality and practicing a carelessly brutal snobbery. He is a sensitive, gifted man who has been humiliated into nihilism. He would laugh to see heads roll.

I think young D over there is the only born Nazi in the room. Young D is the spoiled only son of a doting mother. He has never been crossed in his life. He spends his time at the game of seeing what he can get away with. He is constantly arrested for speeding and his mother pays the fines. He has been ruthless toward two wives and his mother pays the alimony. His life is spent in sensation-seeking and theatricality. He is utterly inconsiderate of everybody. He is very good-looking, in a vacuous, cavalier way, and inordinately vain. He would certainly fancy himself in a uniform that gave him a chance to swagger and lord it over others.

Mrs. E would go Nazi as sure as you are born. That statement surprises you? Mrs. E seems so sweet, so clinging, so cowed. She is. She is a masochist. She is married to a man who never ceases to humiliate her, to lord it over her, to treat her with less consideration than he does his dogs. He is a prominent scientist, and Mrs. E, who married him very young, has persuaded herself that he is a genius, and that there is something of superior womanliness in her utter lack of pride, in her doglike devotion. She speaks disapprovingly of other “masculine” or insufficiently devoted wives. Her husband, however, is bored to death with her. He neglects her completely and she is looking for someone else before whom to pour her ecstatic self-abasement. She will titillate with pleased excitement to the first popular hero who proclaims the basic subordination of women.

On the other hand, Mrs. F would never go Nazi. She is the most popular woman in the room, handsome, gay, witty, and full of the warmest emotion. She was a popular actress ten years ago; married very happily; promptly had four children in a row; has a charming house, is not rich but has no money cares, has never cut herself off from her own happy-go-lucky profession, and is full of sound health and sound common sense. All men try to make love to her; she laughs at them all, and her husband is amused. She has stood on her own feet since she was a child, she has enormously helped her husband’s career (he is a lawyer), she would ornament any drawing-room in any capital, and she is as American as ice cream and cake.

You are currently viewing this article as a guest. If you are a subscriber, please sign in. If you aren't, please subscribe below and get access to the entire Harper's archive for only $45.99/year. Or purchase this issue on your iOS or Android devices for $6.99.

= Subscribers only.
Sign in here.
Subscribe here.

Veil is private browsing for the ultra-paranoid

techcrunch.com - Comments

13334048894_6e8b421c4e_o-e1438887295188.

If you’re worried about someone finding out what you’re pointing your browser at, there are plenty of options for keeping it secret, with varying levels of difficulty and effectiveness. Veil takes things further than perhaps any other anonymous browsing method by masking the page you’re viewing not just from would-be attackers, but from your own operating system.

The problem, as the MIT researchers behind Veil explain in a new paper outlining the service, is that private browsing modes, even ones using Tor and other measures, can still leave a trace of your history on the device itself, in RAM or temporary storage.

When you visit a page, even anonymously, that page and its components still have to be loaded into memory, displayed and cached, libraries or plugins perhaps accessed or modified. The browser may try to delete these traces, but success can vary depending on how things are set up. A sort of ghost version of your activity may live on in your RAM, even if it’s just a hash of some data or timestamp.

“The fundamental problem is that [the browser] collects this information, and then the browser does its best effort to fix it. But at the end of the day, no matter what the browser’s best effort is, it still collects it,” explained MIT graduate student Frank Wang, the lead author of the paper, in a news release. “We might as well not collect that information in the first place.”

Veil takes things several steps further by handling delivery of the site via what they call a “blinding server.” You enter the URL into the site and the page is retrieved for you from the special servers, encrypted in transit and in your browser cache, and only decrypted for your viewing. Links and URLs are themselves encrypted so they can’t be linked to the content requested.

Furthermore, it injects invisible garbage code into the page while also “mutating” the content (again, invisibly) so that you could load it a thousand times on the same computer and although it would look the same to you, any resulting digital fingerprints like hash, payload size and so on would always be different.

This way your computer never actually registers the actual URL, never caches any of the data and any traces left won’t match up with any database or even each other.

In the most extreme privacy case, users don’t even interact with the actual webpage — just an image of it

Converting webpages to Veil-compatible ones can be done via a special compiler provided by the researchers. It shouldn’t break anything, though it will add to the bandwidth used and requests served, owing to the mutations and extra crypto operations.

But wait, there’s more! For those of you worried that even that level of obfuscation isn’t enough, there’s an option that I’ve always considered a possibility but never seen executed: browsing via visual data alone.

If you want, Veil will instead of showing you the target site’s actual code, just take a screenshot and show you that. You can click on the screenshot and it will record the location of that click, then transmit it to the actual page, returning a screenshot of the result.

The next step, I suppose, will be to employ a robot that looks at a completely separate computer somewhere and tells you what it sees in pig Latin. Onnection-cay ecure-say!

Of course, this isn’t a zero-trust situation: the blinding servers will, like Tor nodes, be run by volunteers and organizations that could attempt to compromise the system (a site could also run its own). But if the process is mathematically and procedurally verifiable, you should be okay. Of course, security researchers will want to audit the code.

Veil was presented this week at the Network and Distributed Systems Security Symposium in San Diego. You can read the full paper here.

Featured Image: Yuri Samoilov/Flickr UNDER A CCPL LICENSE (IMAGE HAS BEEN MODIFIED)

AMP: the missing controversy

ferdychristant.com - Comments

1*ceh7eagXiM6ahQTL_ZkXWw.png

How Google cheats with performance

Introduction

AMP, Accelerated Mobile Pages, is a technology first launched in 2015 by Google. The main goal of the initiative is to drastically speed up the loading of web pages on mobile. Sorely needed, as the typical web page on slow 3G (a very typical network condition the world over) takes a long time to load.

So the goal in itself seems worthy. Yet the initiative has been met with lots of controversy over the years. I’m going to go briefly over these main points of controversy.

The main goal of this article though is to add a new point of controversy, one hardly discussed. The reason why AMP has instant performance.

First, let’s look at the other points of controversy.

Controversy #1: a new web “standard”

AMP has been created completely outside of W3C and WHATWG, the main standard bodies for the web. Standard bodies in which Google has a large, if not dominant presence.

AMP makes up its own standards that break with what is considered valid HTML. Case in point, have a look at how the AMP project’s homepage, which itself is an AMP page, produces over a 100 validation errors:

In AMP’s defense, despite it breaking the standard, browsers will render that page just fine because browser know how to deal with invalid HTML quite well. Because, well, pretty much every web page ever created is invalid HTML.

So this is merely a theoretical controversy. We have a shared web, governed by standards bodies. AMP ignores this reality.

Google’s main defense is that AMP is open source. Which isn’t just a weak defense, it’s no defense at all. I can open source a plan for genocide. The term “open source” is meaningless if the thing that is open source is harmful.

Controversy #2: loss of sovereignty of your website

One of the main implications of publishing an AMP page is that the page will be served from the Google domain. Or whoever is serving the AMP cache, yet mostly that will be Google.

This means less direct traffic on your origin, and more time spent at Google. Less traffic on your origin could mean less monetization opportunities. In general, it means less control of anything. You’re subject to whatever the AMP standard allows or disallows.

You’re giving up the sovereignty of your web site. Similar to when you publish all your content directly inside Facebook, you’re effectively losing all control of it now and in the future.

A simple defense here could be to just not use AMP. Ironically, AMP can actually increase traffic and drive better conversions. For the simple reason that it performs so much better than a normal web page. Hence, in the desperate race for traffic, organizations are lured towards AMP. The only price they have to pay is an almost complete loss of control over their web experience.

Build faster web pages then, and don’t use AMP? Even with this strategy, you cannot win from AMP’s performance. More on this in a minute.

Controversy #3: loss of diversity

On a diverse web, you would visit different origins, each having a different experience. Now compare this to Facebook, where every piece of content looks the same, coming from all over the web, yet flattened into a dull timeline. You don’t get to experience the home of that content, just the content itself.

This same effect of everything looking the same applies to AMP, in a lesser extend. In AMP’s defense, it’s possible to style an AMP page to fit your brand, yet there’s severe limitations in what you can and cannot do. Not just in styling, even more so in interactivity.

Controversy #4: performance

The items discussed above are not new, they’ve been discussed for years. So let us move on to the point of this article: the performance controversy of AMP. Which hardly is a topic of discussion it seems.

What we know about AMP is that the technical standard in itself enforces good performance. For example, CSS is limited to 50KB and only inline, custom JS is not allowed (other than via an iframe method), and all images are guaranteed to be lazy loading.

Hence, this is why AMP pages load instantly compared to a “normal” web page. It’s enforced by the standard. Right?

Wrong.

Let us look at a typical AMP page:

It’s an ordinary AMP page. I didn’t pick an intentionally bad example. We’re going to check the performance of this AMP page by directly measuring it from the origin itself (so not from AMP cache):

1*zJ8s3Q2JFfRp5XajtbOlog.jpeg

On webpagetest.org/easy (which preselects average 3G), the first meaningful paint is at 3.5s. A number that is not terrible in itself, pretty good actually, yet strangely far away from AMP’s typical instant performance.

Testing directly in Chrome, using “low-end mobile” and “mid-tier mobile” as presets (these slow down both the network and the CPU), we get 8.4s and 2.3s as first meaningful paint. Both aren’t close to instant performance, and 8.4s is just plain awful. This is an AMP page!?

1*6z2higw-CIOex7f3Eo18lw.jpeg

Using a Lighthouse audit, the page fails to score on the good side of performance. Despite being fully valid AMP, a standard that enforces performance.

So whilst obviously the technical restrictions of the AMP standard help with performance, they are in no way responsible for having instant performance. Technically correct AMP pages will perform very similar to any non-horrible web page.

The difference between AMP performing instantly and getting numbers ranging from 2–8s as seen above have to be explained.

Part of that answer you can probably guess: the cache is simply very fast. It’s hard to compete with a Google-class CDN. I imagine thousands of servers strategically placed worldwide on the best connections available.

Yet this fast cache doesn’t explain a 2–8s difference. It hardly affects it. This is what does:

Here we are on Google Search on mobile. We searched for a term (“Elon Musk”). We scroll down in the results, in the bottom you can start to see the “Scientias” article that we profiled starting to appear.

At this moment, the network panel fills up with resources from that AMP page. Pretty much anything that page needs to render is preloaded, whether you actually open it not. If you do, it’s going to render instantly.

Not in 2–8s. Instantly. Technically, a clever trick. It’s hard to argue with that. Yet I consider it cheating and anti competitive behavior.

The AMP page, which we all believe to be super fast and optimized for slow mobiles because it is AMP, isn’t that fast. Its true speed comes from preloading.

What it means

The above findings have a few very serious implications:

  • Although AMP as a standard helps with mobile performance, the standard itself is not responsible for instant performance. AMP fails at those first crucial seconds just like any other web page. AMP pages on their own are in no way faster than a well designed mobile web page (in which you would have freedom to do anything).
  • You can never compete with AMP’s instant performance, even if you’d build the fastest website in the world.

Here’s why I consider the scenario described above cheating and anti-competitive:

You likely are not the owner of a search monopoly, hence you cannot control loading behavior. Only Google can. Taking that page from 2–8s to instant performance is something only Google is capable of, because it is the only entity in the world controlling the most important information portal: search.

Preloading is exclusive to AMP. Google does not preload non-AMP pages. If Google would have a genuine interest in speeding up the whole web on mobile, it could simply preload resources of non-AMP pages as well. Not doing this is a strong hint that another agenda is at work, to say the least.

What it means for you

The dilemma that comes with AMP, is that it works. Very well. End users will experience much faster pages and it’s very likely that your conversions will improve dramatically. A tempting, almost irresistible proposition for many organizations and businesses.

The point of my article is not to point you in any direction, even though it seems that way. I merely point out the main controversial points of AMP, and how and why it works the way it does. In particular, why it’s so fast. Which is not because of AMP. Not at all because of AMP.

Doctors, Revolt

www.nytimes.com - Comments

But Dr. Lown identifies first and foremost as a healer. In 1996, he published “The Lost Art of Healing,” an appeal to restore the “3,000-year tradition, which bonded doctor and patient in a special affinity of trust.” The biomedical sciences had begun to dominate our conception of health care, and he warned that “healing is replaced with treating, caring is supplanted by managing, and the art of listening is taken over by technological procedures.”

He called for a return to the fundamentals of doctoring — listening to know the patient behind the symptoms; carefully touching the patient during the physical exam to communicate caring; using words that affirm the patient’s vitality; and attending to the stresses and situations of his life circumstances.

This time he was the patient in need of healing. And I was the doctor, the product of a system that has, if anything, become even more impersonal and transactional since he first wrote those words.

Despite his reputation, Dr. Lown was treated like just another widget on the hospital’s conveyor belt. “Each day, one person on the medical team would say one thing in the morning, and by the afternoon the plan had changed,” he later told me. “I always was the last to know what exactly was going on, and my opinion hardly mattered.”

What he needed was “the feeling of being a major partner in this decision,” he said. “Even though I am a doctor, I am still a human with anxieties.”

The medical team was concerned that because Dr. Lown was having trouble swallowing, he was at risk for recurrent pneumonias. So we restricted his diet to purées. Soon the speech therapist recommended that we forbid him to ingest anything by mouth. Then the conversation spiraled into ideas for alternative feeding methods — a temporary tube through the nose followed, perhaps, by a feeding tube in the stomach.

“Doctors no longer minister to a distinctive person but concern themselves with fragmented, malfunctioning” body parts, Dr. Lown wrote in “The Lost Art of Healing.” Now, two decades later, he’d become a victim of exactly what he had warned against.

As the intern and the perpetrator of the orders, I felt impossibly torn and terribly guilty. So after Dr. Lown was discharged the next week, I kept in touch, hoping to continue this important conversation.

We have since spent time together at his home, where he is back to living peacefully and swallowing carefully (no alternative feeding methods necessary).

I had known Dr. Lown as a doctor and a patient; now I got to know him as an activist. We agreed that the health care system needed to change. To do that, Dr. Lown said, “doctors of conscience” have to “resist the industrialization of their profession.”

This begins with our own training. Certainly doctors must understand disease, but medical education is overly skewed toward the biomedical sciences and minutiae about esoteric and rare disease processes. Doctors also need time to engage with the humanities, because they are the gateway to the human experience.

To restore balance between the art and the science of medicine, we should curtail initial coursework in topics like genetics, developmental biology and biochemistry, making room for training in communication, interpersonal dynamics and leadership.

Such skills would not only help doctors care for our fellow human beings but would also strengthen our ability to advocate for health care as a human right and begin to rectify the broken economics and perverse incentives of the system.

Finally, hospitals should be a last resort, not the hallmark of the health care system. The bulk of health care resources should go instead into homes and communities. After all, a large majority of health problems are shaped by nonmedical factors like pollution and limited access to healthy food. Doctors must partner with public health and community development efforts to create a culture of health and well-being in patients’ daily lives.

As I navigate my professional journey, Dr. Lown’s example inspires me to go to work every day with the perspective of a patient, the spirit of an activist and the heart of a healer.

Continue reading the main story

Ken and Roberta (2011)

www.filfre.net - Comments

There are two prototypical kinds of “computer professionals” in the world. First there are the purist hackers, who dive into the depths of circuits, operating systems, and programming languages like explorers discovering new lands; it wasn’t by chance that away from the computer Will Crowther was a caver, nor that he now spends his time deep-sea scuba diving. For the purists the reward is in the thing itself, in learning to understand and navigate this binary wonderland and maybe, just maybe, someday making (or helping to make) something really, truly new and cool. The other group is made up of the careerists. These people end up in the field for some mixture of a variety of reasons: because they need to earn a good living to support their family (no shame in that); because they’ve heard computers are cool and the next big thing (hello, Internet bubble); because they have a vision of society which requires computers as its enabler (hello, Steve Jobs); because they just want to get really, really rich (why, there’s Steve again hiding out in the corner hoping not to be noticed — hi!). One thing only binds this disparate group together: they are attracted to computers not by their intrinsic interest in the machines themselves but by externalities, by a vision of what the machines can do, whether for them or for others. The two groups often seem — and believe themselves to be — at odds with one another, but in truth they need each other. Witness the dynamic duo of Woz and Jobs that built the Apple II and got it to the masses. Or witness Ken and Roberta Williams, the power couple of 1980s adventure gaming.

Ken and Roberta married in 1972. He was just 18 at the time; she was 19. He was attending California Polytechnic Pomona University as a physics major, and failing; she was living at home and not doing much of anything. Contrary to what you might be thinking, there was no shotgun involved. He simply wanted Roberta in his life and was determined to have her there, permanently. Steven Levy writes that his words to her were simply, “We’re getting married, and that’s it.” She “didn’t fight it.” Right there you learn a lot about their two personalities.

Within a year or so of their marriage Ken, a restless, driven, somewhat aggressive young man with no real respect for or interest in higher education with its hierarchical structure and its abstract theorizing, could see he wasn’t going to make it as a physics major, much less a physicist. Roberta, meanwhile, was now pregnant. Ken needed a career, and he needed one quick.

In the early 1970s the institutional computer industry was nearing its peak, supplying mainframes and minicomputers by the thousands to businesses, universities, public and private schools, branches of government, and research installations. We’ve met several of the prominent companies already (IBM, DEC, HP), each serving their own core sectors of this huge market while competing with one another on the margins. Another was Control Data Corporation. Founded in 1957 by a group of refugees from an even earlier company, Sperry, CDC had by the early 1970s carved out a reputation for itself as a manufacturer of prestigious and expensive supercomputers of the type used for some of the most intensive scientific computing. The supercomputer market was, however, a small one, and so the bulk of CDC’s business was courtesy of its line of more plebeian mainframes that competed directly with IBM for corporate business. To carve out a place for itself against the larger company, CDC tried to stick to a “10% rule”: to make sure each of its systems was always 10% faster and 10% cheaper than the nearest equivalent IBM model. For a number of years this approach was very good to CDC, sufficiently so that the company opened a little trade school all its own to train future custodians of its systems. Armed with a $1500 student loan co-signed by a very concerned father-in-law, Ken entered Control Data Institute. In doing so he was conforming to a stereotype that remains with the computer industry to this day: the pure hackers go to universities and get computer-science degrees; the careerists go to trade schools and get certificates in something “practical.”

Indeed, the atmosphere at CDI promised nothing like the free-wheeling intellectual exploration of the computer-science labs at MIT or Berkley. The emphasis was on pounding in the rote tasks and procedures needed to maintain and run the big, batch-processing mainframes of CDC at the banks and other large bureaucratic entities that housed them. And that suited Ken, hungry for career in business, just fine. Where an MIT hacker might have seen intolerable drudgery, he saw money to be made. When he turned out to be pretty good at this computer stuff — even within limits to enjoy it — that just increased the earning potential.

After finishing at CDI, Ken spent the rest of the 1970s living a life that we more typically associate with the following decade, bouncing from company to company in search of ever better salaries while generally also juggling two or three independent consulting gigs on the side. With computers still mysterious, almost occult objects to most people, a fast-talking, energetic, and ambitious young man like Ken could go far with just the modicum of knowledge he had gained at CDI. As that knowledge increased and he became an ever better programmer and problem solver courtesy of the best teacher of all, experience, he seemed even more of a miracle worker, and found himself even more in demand. Ken, in other words, was becoming a pretty damn good hacker almost in spite of himself. But he always wanted more — a new hot tub, a bigger house, a nicer car, a place in the country — even as he dreamed of retiring young and bequeathing a fortune to his children. (These things would in fact happen, although not in the way Ken thought they would in the 1970s.) Ken made no apologies for his materialism. “I guess greed,” he later told Levy, “would summarize me better than anything. I always want more.”

When the first kit computers that one could build in one’s home appeared in 1975, Ken barely noticed. There was no real money to be made in them, he believed, unlike his big, boring mainframes. When the trinity of 1977 marked the arrival of a PC you didn’t need a soldering iron to assemble, he likewise paid no attention. It was not until a couple of years later that the beginning of a real, paying market in professional business software, exemplified by pioneering applications like VisiCalc and WordStar, made Ken begin to pay attention to the little “toy” machines. When he finally bought an Apple II in January of 1980, it was for a very specific purpose.

At the time there were only two real language possibilities for Apple programmers: they could use BASIC, which was easy to learn and get started with but quickly became a nightmare when trying to structure large, complex programs; or assembly language, which gave the ultimate in precise control over the hardware but was well-nigh impenetrable for the uninitiated, tedious in the micro-management it required, and just as bereft of structure. Ken saw an opportunity for a more sophisticated high-level language, one designed to be used by serious programmers creating complex software. Specifically, he wanted to bring FORTRAN, as it happens the implementation language of the original Adventure (not that Ken likely knew this or cared), to the little Apple II. With that purpose in mind, he registered a company of his own, choosing to call it On-Line Systems, a name fairly typical of the vaguely futuristic, vaguely compound, but essentially meaningless names (Microsoft, anyone?) that were so common in the era.

And what was Roberta doing during these years? Well, she was raising the Williams’ two children and happily (at least to external observers) playing the role of housewife and homemaker. She had always been a painfully shy, passive personality who by her own admission “could hardly make a phone call.” If Ken seemed to already be living in the frenetic 1980s rather than the mellow 1970s, Roberta seemed a better match for the 1950s, the doting wife who took care of the children, made sure everyone in the family had a good breakfast, lunch, and dinner, and meekly entrusted the big decisions and the earning of a living to the man of the house. That makes what happened next doubly surprising.

Shortly before Ken bought that first Apple, and while the Williams’ second son was just eight months old, Ken happened to have a remote terminal at the house for one of his gigs. The mainframe to which it could connect had on it a copy of Adventure, which by now had been ported to a variety of other platforms beyond the PDP-10. Ken called Roberta over to have a look at what he regarded as nothing more than a curiosity. Roberta, however, was immediately transfixed. “I started playing and kept playing it. I had a baby at the time, Chris was eight months old; I totally ignored him. I didn’t want to be bothered. I didn’t want to stop and make dinner.” As Ken wondered what had become of his dutiful wife, Roberta stayed up most of the night playing, then lay awake in bed working through the puzzles in her mind. It was no doubt a relief to everyone when she finally finished the game after a month of effort.

But the respite didn’t last long. After Ken brought the Apple II into the home, it didn’t take Roberta long to learn about the works of Scott Adams. Soon she was back to obsessively playing again. But then another thought began to crowd out the conundrums of the games: what if she could make a text adventure of her own? She was turning the most inspirational corner I know, imagining herself as a creator rather than a passive consumer. Inspired mostly by Agatha Christie’s novel Ten Little Indians and the board game Clue, she began to sketch ideas for a text adventure as cozy murder mystery, a genre that the form had not yet tackled. When she was pretty far along, she took a deep breath and laid out her ideas to Ken.

The story concept was certainly innovative, but it wasn’t the sort of innovation that would immediately appeal to a guy like Ken, with little interest in game design in the abstract. He was rather interested in products he could sell, operating intuitively by a rule he would later, for better and perhaps sometimes for worse, codify and articulate regularly: “Games have to have ‘WOW-value.’ If you don’t say ‘wow’ when someone describes the game to you, or you see it from 10 feet away, there’s no reason to market the game.” At first, caught up in his FORTRAN software and his prior experience of computers only as serious tools of business, he was dismissive of Roberta’s little project. But as she persisted, and as he perhaps began to notice that companies like Adventure International were growing rapidly and making real money just like the “serious” software houses, he began to reconsider. Still, he needed something special, needed an angle to help their little game stand out from the likes of the established line of Scott Adams games.

He began to think about the Apple II, with its comparatively cavernous 48 K of RAM, its fast and reliable disk drives, and its bitmap graphics capability. What if he designed their game around the unique capabilities of that machine, instead of taking the portable lowest-common-denominator approach of Adams? And then came the brainstorm: he could use the Apple’s hi-res mode to include pictures with the text. That would certainly make their game stand out. Pretty soon FORTRAN was forgotten, and work on Mystery House (the first of a whole line of On-Line Systems “Hi-Res Adventures”) had begun in earnest. The husband-and-wife team were not that far removed from Woz and Jobs. Here, Roberta designed the thing out of her inherit fascination with the thing itself, while Ken enabled her efforts, providing the tools and support she needed to bring her vision to life and, soon enough, finding ways to sell that vision to the masses.

 

Understanding Dijkstra's Algorithm

aos.github.io - Comments

When I first started learning algorithms and data structures, every resource I came across would mention Dijkstra’s algorithm in a sort of mystical, this-is-beyond-your-lowly-understanding manner. I disagree with that approach (in fact, I disagree with that approach for just about everything). Now that I’ve actually invested some time into learning it, it really isn’t as frightful as I was told to believe. I will (to the best of my ability) elucidate Dijkstra’s algorithm here.

Dijkstra’s algorithm is used to solve the single source shortest path. What does that mean? Starting at a single point, what is the shortest path to another point, say X. The best example would be using Google maps to get from your location to a destination. Using an algorithm (not necessarily Dijkstra’s) we can compute the shortest path by space and/or time. Let’s say you are given the diagram below, each node depicts a certain location and the number in blue a “distance” between to the two node. What is the shortest path from node S to H?

Dijkstra Graph Ex 1

It is not so obvious at first glance. Indeed, this is also only a simple example. Imagine the entirety of Manhattan mapped out. The complexity of the diagram would increase by a million-fold! Having said that, we will come back to this diagram to solve our problem. For now we require some additional background knowledge before we go about implementing Dijkstra’s algorithm.

At its heart, Dijkstra’s algorithm is really only a modified bread-first search. Breadth-first search (BFS) is a graph traversal algorithm which works by exploring all neighboring nodes first, before moving on to the next level of neighbors (neighbors’ neighbors). Let’s see how this would play out using the figure above (forgetting about the numbers for a minute) starting at S:

BFS

We see that S has 2 neighbors, A and F. We add these on our todo list to explore (marked in gray). We start with A which was at the top of our todo list. A likewise has 2 neighbors, E and B – so we add those to our todo list. We’ve now fully explored A (black) so we move on to the next item on our todo list, F. We explore F and see that it has two neighbors as well, E and G. We again add those two to our todo list. We are now done exploring F and see that the top of our list has E, so we start exploring that. Lucky for us, E only has one neighbor we have yet to explore, and that is D.

I won’t belabor you by writing out every step, but you can see the pattern here: we explore a node, add its neighbors to our todo list, explore those neighbors. A caveat that I should mention is that we never explore the same node twice. In our diagram above, both A and F have E as their neighbor. When we explore E we make sure to mark it as explored so that we don’t explore it again. Eventually, we’ll have explored all the nodes! But where is the logic coming from for this magical “todo list”? That is our next topic…

I hand-waved our addition of the nodes to a todo list, but it is a legitimate data structure with a name: a queue. Breadth-first search uses a queue to keep track of nodes not yet explored. A queue is a first-in-first-out (FIFO) data structure. This means that the first item to be added, will be the first item to be removed. Think of a queue as a waiting line at a grocery store checkout station. There might be people ahead of you, in which case you will have to wait your turn until all the ones who came before you have been checked out. The queue’s most important operations are enqueue which adds an item to the end, and dequeue which removes the first item in line.

Now that we know about a queue, let’s build a simple implementation of breadth-first search. I won’t get into the specifics of graph representation in code, but it is typically represented as an “adjacency list” – a hash object with the keys as nodes, and the values as neighbors:

# Simple graph representation like adjacency list
# using an array to represent our list
graph = {
  'A': ['B', 'C'],
  'B': ['A', 'E'],
  'C': ['A', 'D'],
  'D': ['C', 'E']
  'E': ['B', 'D']
}

And this is how this graph would look like:

SG

Now let’s implement BFS:

def bfs(graph, source)
  q = Queue.new
  # Keep track of what we've explored
  explored = Hash.new

  # Start at source node
  q.enqueue(source)

  # While our queue is not empty,
  # keep going through nodes
  while !q.empty?
    node = q.dequeue
    explored[node] = true
    # Get the neighbor nodes
    neighbors = graph[node]

    # For each neighbor node, add it to the queue
    # unless we've previously explored it
    neighbors.each do |v|
      q.enqueue(v) unless explored[v]
    end
  end
end

This will traverse our graph layer-by-layer, node-by-node. This is as barebones as it gets of course but it gives us an idea of how a BFS algorithm would look like and how we can modify it to fit our needs.

One of Dijkstra’s algorithm modifications on breadth-first search is its use of a priority queue instead of a normal queue. With a priority queue, each task added to the queue has a “priority” and will slot in accordingly into the queue based on its priority level.

PQ

The figure above shows a queue consisting of items with rankings based on priority. In this case, the item with priority 2 would be dequeued next. If we were to enqueue the blue item with priority 4 next, it would go before the item with priority 5 but after the one ranked 3. This is known as a minimum priority queue – the smallest priority will be highest ranked and will be processed first.

We’ve now been introduced to all of the pieces of the puzzle – let’s fit them together. We will use the priority queue in place of the normal queue in our modified BFS – with the priorities being the distances between nodes. How will this work out? Let’s go step by step:

  1. Initialize source node with a distance of 0 since we are starting here
  2. Initialize all other nodes with an infinite distance
  3. Start the queue loop after inserting our source node into it
  4. For each neighbor of our node, calculate the tentative distance between the current node’s distance and the distance to its neighbor
    • If the tentative distance is less than the distance of the neighbor, set that neighbor node’s distance to the tentative distance and enqueue it with that distance as its priority
      (This is known as Dijkstra’s greedy score)
  5. Continue through the priority queue until we’ve calculated all nodes’ shortest path to source
    • Optionally, we can provide a target node and as soon as that node is dequeued, we can stop our search

Let’s see these steps in action using the graph we started with:

DAGIF

Based on the animation above, our shortest path from S to H is S->F->G->H and has a distance of 14. Another thing to note is that our algorithm calculated the shortest distance for every node from the source. The animation above “dropped” paths that did not satisfy Dijkstra’s greedy score. In actuality the algorithm will just not enqueue any node that does not satisfy the greedy score.

Let’s see the algorithm implementation:

def dijkstra(graph, source)
  pq = PriorityQueue.new
  # Initialize distances for all nodes at infinity
  dist = Hash.new(Float::INFINITY)
  # Get edge lengths
  lengths = graph.get_lengths

  # Initialize source node and insert into queue
  dist[source] = 0
  # Priority queue holds nodes as key/value pairs
  pq.enqueue(source, dist[source])

  while !pq.empty?
    node = pq.dequeue[:key]
    neighbors = graph[node]

    neighbors.each do |neighbor|
      # Calculate tentative distance 
      # (Dijkstra's greedy score)
      tent_dist = dist[node] + lengths[node][neighbor]
      
      if tent_dist < dist[neighbor]
        # Set neighbor's new distance as it is shorter
        dist[neighbor] = tent_dist
        # Enqueue using new distance as priority
        pq.enqueue(neighbor, dist[neighbor])
      end
    end
  end
  return dist # Or whatever else you want
end

This algorithm looks very similar to BFS! We just added the calculation of the greedy score which determined how to prioritize insertion into our priority queue. One thing to note is I glanced over getting edge lengths as it is just a detail of implementation. The usual way to do this is to construct a 2-dimensional matrix between nodes and fill out lengths. It can also be done by creating a hash.

As it stands, the running time of our algorithm is not too great. If we assume a naive implementation of a priority queue (enqueue will scan entire structure to find place to insert), then our algorithm is running in quadratic time O(mn).

Can we do better? Of course! We can implement our priority queue using a heap. A heap will provide the same API to us as a priority queue (enqueue/insert, dequeue/extract) but it does insertion in logarithmic time. That’s great news! That means we effectively go from O(mn) to O(m log n) time by using a heap.

The implementation above does not map out the paths. The algorithm can easily be augmented to accomodate that by creating a predecessor hash which can be added to as we are enqueue-ing. We can also stop the algorithm earlier if we provide to it a target node: as we are dequeue-ing, we can just check if that is the node we are looking for. If so, we return from the algorithm with the distance to that node. Our current implementation goes through every node.

A caveat I forgot to mention earlier is that this algorithm requires positive edge lengths and will break with negative edge lengths. There is Bellman-Ford algorithm for that situation.

This has been a fun dive into a famous algorithm. It is elegant, and surprisingly simple once we spent a little time into it. I hope you enjoyed reading this as much as I enjoyed creating it!

A Programmable Programming Language

cacm.acm.org - Comments

A Programmable Programming Language, illustration

Credit: Chris Labrooy

In the ideal world, software developers would analyze each problem in the language of its domain and then articulate solutions in matching terms. They could thus easily communicate with domain experts and separate problem-specific ideas from the details of general-purpose languages and specific program design decisions.

Back to Top

Key Insights

ins01.gif

In the real world, however, programmers use a mainstream programming language someone else picked for them. To address this conflict, they resort to—and on occasion build their own—domain-specific languages embedded in the chosen language (embedded domain-specific languages, or eDSLs). For example, JavaScript programmers employ jQuery for interacting with the Document Object Model and React for dealing with events and concurrency. As developers solve their problems in appropriate eDSLs, they compose these solutions into one system; that is, they effectively write multilingual software in a common host language.a

Sadly, multilingual eDSL programming is done today on an ad hoc basis and is rather cumbersome. To create and deploy a language, programmers usually must step outside the chosen language to set up configuration files and run compilation tools and link-in the resulting object-code files. Worse, the host languages fail to support the proper and sound integration of components in different eDSLs. Moreover, most available integrated development environments (IDEs) do not even understand eDSLs or perceive the presence of code written in eDSLs.

The goal of the Racket project is to explore this emerging idea of language-oriented programming, or LOP, at two different levels. At the practical level, the goal is to build a programming language that enables language-oriented software design. This language must facilitate easy creation of eDSLs, immediate development of components in these newly created languages, and integration of components in distinct eDSLs; Racket is available at http://racket-lang.org/

At the conceptual level, the case for LOP is analogous to the ones for object-oriented programming and for concurrency-oriented programming.3 The former arose from making the creation and manipulation of objects syntactically simple and dynamically cheap, the latter from Erlang's inexpensive process creation and message passing. Both innovations enabled new ways to develop software and triggered research projects. The question is how our discipline will realize LOP and how it will affect the world of software.

Our decision to develop a new language—Racket—is partly an historical artifact and partly due to our desire to free ourselves from any unnecessary constraints of industrial mainstream languages as we investigate LOP. The next section spells out how Racket got started, how we honed in on LOP, and what the idea of LOP implies.

Back to Top

Principles of Racket

The Racket project dates to January 1995 when we started it as a language for experimenting with pedagogic programming languages.15 Working on them quickly taught us that a language itself is a problem-solving tool. We soon found ourselves developing different languages for different parts of the project: a (meta-) language for expressing many pedagogic languages, another for specializing the DrRacket IDE,15 and a third for managing configurations. In the end, the software was a multilingual system, as outlined earlier.

Racket's guiding principle reflects the insight we gained: empower programmers to create new programming languages easily and add them with a friction-free process to a codebase. By "language," we mean a new syntax, a static semantics, and a dynamic semantics that usually maps the new syntax to elements of the host language and possibly external languages via a foreign-function interface (FFI). For a concrete example, see Figure 1 for a diagram of the architecture of a recently developed pair of scripting languages for video editing2,b designed to assist people who turn recordings of conference presentations into YouTube videos and channels. Most of that work is repetitive—adding preludes and post-ludes, concatenating playlists, and superimposing audio—with few steps demanding manual intervention. This task calls for a domain-specific scripting language; video is a declarative eDSL that meets this need.

f1.jpg
Figure 1. Small language-oriented programming example.

The typed/video language adds a type system to video. Clearly, the domain of type systems comes with its own language of expertise, and typed/video's implementation thus uses turnstile,6 an eDSL created for expressing type systems. Likewise, the implementation of video's rendering facility calls for bindings to a multimedia framework. Ours separates the binding definitions from the repetitive details of FFI calls, yielding two parts: an eDSL for multimedia FFIs, dubbed video/ffi, and a single program in the eDSL. Finally, in support of creating all these eDSLs, Racket comes with the syntax parse eDSL,7 which targets eDSL creation.

The LOP principle implies two subsidiary guidelines:

Enable creators of a language to enforce its invariants. A programming language is an abstraction, and abstractions are about integrity. Java, for example, comes with memory safety and type soundness. When a program consists of pieces in different languages, values flow from one context into another and need protection from operations that might violate their integrity, as we discuss later; and


Most notably, Racket eliminates the hard boundary between library and language, overcoming a seemingly intractable conflict.


Turn extra-linguistic mechanisms into linguistic constructs. A LOP programmer who resorts to extra-linguistic mechanisms effectively acknowledges that the chosen language lacks expressive power.13,c The numerous external languages required to deal with Java projects—a configuration language, a project description language, and a makefile language—represent symptoms of this problem. We treat such gaps as challenges later in the article.

They have been developed in a feedback loop that includes DrRacket15 plus typed,36 lazy,4 and pedagogical languages.15

Back to Top

Libraries and Languages Reconciled

Racket is an heir of Lisp and Scheme. Unlike these ancestors, however, Racket emphasizes functional over imperative programming without enforcing an ideology. Racket is agnostic when it comes to surface syntax, accommodating even conventional variants (such as Algol 60).d Like many languages, Racket comes with "batteries included."

Most notably, Racket eliminates the hard boundary between library and language, overcoming a seemingly intractable conflict. In practice, this means new linguistic constructs are as seamlessly imported as functions and classes from libraries and packages. For example, Racket's class system and for loops are imports from plain libraries, yet most programmers use these constructs without ever noticing their nature as user-defined concepts.

Racket's key innovation is a modular syntax system,17,26 an improvement over Scheme's macro system,11,24,25 which in turn improved on Lisp's tree-transformation system. A Racket module provides such services as functions, classes, and linguistic constructs. To implement them, a module may require the services of other modules. In this world of modules, creating a new language means simply creating a module that provides the services for a language. Such a module may subtract linguistic constructs from a base language, reinterpret others, and add a few new ones. A language is rarely built from scratch.

Like Unix shell scripts, which specify their dialect on the first line, every Racket module specifies its language on the first line, too. This language specification refers to a file that contains a language-defining module. Creating this file is all it takes to install a language built in Racket. Practically speaking, a programmer may develop a language in one tab of the IDE, while another tab may be a module written in the language of the first. Without ever leaving the IDE to run compilers, linkers, or other tools, the developer can modify the language implementation in the first tab and immediately experience the modification in the second; that is, language development is a friction-free process in Racket.

In the world of shell scripts, the first-line convention eventually opened the door to a slew of alternatives to shells, including Perl, Python, and Ruby. The Racket world today reflects a similar phenomenon, with language libraries proliferating within its ecosystem: racket/base, the Racket core language; racket, the "batteries included" variant; and typed/racket, a typed variant. Some lesser-known examples are datalog and a web-server language.27,30 When precision is needed, we use the lowercase name of the language in typewriter font; otherwise we use just "Racket."

Figure 2 is an illustrative module. Its first line—pronounced "hash lang racket base"—says it is written in racket/base. The module provides a single function, walk-simplex. The accompanying line comments—introduced with semicolons—informally state a type definition and a function signature in terms of this type definition; later, we show how developers can use typed/racket to replace such comments with statically checked types, as in Figure 5. To implement this function, the module imports functionality from the constraints module outlined in Figure 3. The last three lines of Figure 2 sketch the definition of the walk-simplex function, which refers to the maximizer function imported from constraints.

f2.jpg
Figure 2. A plain Racket module.

f3.jpg
Figure 3. A module for describing a simplex shape.

The "constraints" module in Figure 3 expresses the implementation of its only service in a domain-specific language because it deals with simplexes, which are naturally expressed through a system of inequalities. The module's simplex language inherits the line-comment syntax from racket/base but uses infix syntax otherwise. As the comments state, the module exports a single function, maximizer, which consumes two optional keyword parameters. When called as (maximizer #:xn), as in Figure 2, it produces the maximal y value of the system of constraints. As in the lower half of Figure 3, these constraints are specified with conventional syntax.


In general, cooperating multilingual components must respect the invariants established by each participating language.


In support of this kind of programming, Racket's modular syntax system benefits from several key innovations. A particularly illustrative one is the ability to incrementally redefine the meaning of existing language constructs via the module system. It allows eDSL creators to ease their users into a new language by reusing familiar syntax, but reinterpreted.

Consider lambda expressions, for example. Suppose a developer wishes to equip a scripting language (such as video) with functions that check whether their arguments satisfy specified predicates. Figure 4 shows the basic idea:

f4.jpg
Figure 4. Lambda, redefined.

line 01 The module uses the racket language.

line 03 It exports a defined compile-time function, new-lambda, under the name cacm6103_i.gif, which is over-lined in the code to mark its origin as this module.

line 05 Here, the module imports tools from a library for creating robust compile-time functions conveniently.7

line 07 The comment says a function on syntax trees follows.

line 08 While (define (f x) . . . ) introduces an ordinary function f of x, (define-syntax (c stx) . . . ) creates the compile-time function c with a single argument, stx.

line 09 As with many functional languages, Racket comes with pattern-matching constructs. This one uses syntax-parse from the library mentioned earlier. Its first piece specifies the to-be-matched tree (stx); the remainder specifies a series of pattern-responses clauses.

line 10 This pattern matches any syntax tree with first token as new-lambda followed by a parameter specification and a body. The annotation :id demands that the pattern variables x and predicate match only identifiers in the respective positions. Likewise, :expr allows only expressions to match the body pattern variable.

line 11 A compile-time function synthesizes new trees with syntax.

line 12 The generated syntax tree is a lambda expression. Specifically, the function generates an expression that uses lambda. The underline in the code marks its origin as the ambient language, here racket.

other lines Wherever the syntax system encounters the pattern variables x, predicate, and body, it inserts the respective subtrees that match x, predicate, and body.

When another module uses "new-lam" as its language, the compiler elaborates the surface syntax into the core language like this

(lambda (x :: integer?) (+ x 1))
-elaborates to→
cacm6103_i.gif (x :: integer?) (+ x 1))
-elaborates to→
(new-lambda (x :: integer?) (+ x 1))
-elaborates to→
(lambda (x)
(unless (integer? x)
<elided error reporting>)
(+ x 1))

The first elaboration step resolves lambda to its imported meaning,18 or lambda. The second reverses the "rename on export" instruction. Finally, the new-lambda compile-time function translates the given syntax tree into a racket function.

Inessence, Figure 4 implements a simplistic precondition system for one-argument functions. Next, the language developer might wish to introduce multiargument lambda expressions, add a position for specifying the post-condition, or make the annotations optional. Naturally, the compile-time functions could then be modified to check some or all of these annotations statically, eventually resulting in a language that resembles typed/racket.

Back to Top

Sound Cooperation Between Languages

A LOP-based software system consists of multiple cooperating components, each written in domain-specific languages. Cooperation means the components exchange values, while "multiple languages" implies these values are created in distinct languages. In this setting, things can easily go wrong, as demonstrated in Figure 5 with a toy scenario. On the left, a module written in typed/racket exports a numeric differentiation function. On the right, a module written in racket imports this function and applies it in three different ways, all illegal. If such illegal uses of the function were to go undiscovered, developers would not be able to rely on type information for designing functions or for debugging, nor could compilers rely on them for optimizations. In general, cooperating multilingual components must respect the invariants established by each participating language.

f5.jpg
Figure 5. Protecting invariants.

In the real world, programming languages satisfy a spectrum of guarantees about invariants. For example, C++ is unsound. A running C++ program may apply any operation to any bit pattern and, as long as the hardware does not object, program execution continues. The program may even terminate "normally," printing all kinds of output after the misinterpretation of the bits. In contrast, Java does not allow the misrepresentation of bits but is only somewhat more sound than C++.1 ML improves on Java again and is completely sound, with no value ever manipulated by an inappropriate operation.

Racket aims to mirror this spectrum of soundness at two levels: language implementation itself and cooperation between two components written in different embedded languages. First consider the soundness of languages. As the literature on domain-specific languages suggests,20 such languages normally evolve in a particular manner, as is true for the Racket world, as in Figure 6. A first implementation is often a thin veneer over an efficient C-level API. Racket developers create such a veneer with a foreign interface that allows parenthesized C-level programming.5 Programmers can refer to a C library, import functions and data structures, and wrap these imports in Racket values. Figure 7 illustrates the idea with a sketch of a module; video's initial implementation consisted of just such a set of bindings to a video-rendering framework. When a racket/base module imports the ffi/unsafe library, the language of the module is unsound.

f6.jpg
Figure 6. Hardening a module.

f7.jpg
Figure 7. A Racket module using the foreign-function interface.

A language developer who starts with an unsound eDSL is likely to make it sound as the immediate next step. To this end, the language is equipped with runtime checks similar to those found in dynamically typed scripting languages to prevent the flow of bad values to unsound primitives. Unfortunately, such protection is ad hoc, and, unless developers are hypersensitive, the error messages may originate from inside the library, thus blaming some racket/base primitive operation for the error. To address this problem, Racket comes with higher-order contracts16 with which a language developer might uniformly protect the API of a library from bad values. For example, the video/ffi language provides language constructs for making the bindings to the video-rendering framework safe. In addition to plain logical assertions, Racket's developers are also experimenting with contracts for checking protocols, especially temporal ones.9 The built-in blame mechanism of the contract library ensures sound blame assignment.10

Finally, a language developer may wish to check some logical invariants before the programs run. Checking simple types is one example, though other forms of static checking are also possible. The typed/video language illustrates this point with a type system that checks the input and output types of functions that may include numeric constraints on the integer arguments; as a result, no script can possibly render a video of negative length. Likewise, typed/racket is a typed variant of (most of) racket.

Now consider the soundness of cooperating languages. It is again up to the language developer to anticipate how programs in this language interact with others. For example, the creator of typed/video provides no protection for its programs. In contrast, the creators of typed/racket intended the language to be used in a multilingual context; typed/racket thus compiles the types of exported functions into the higher-order contracts mentioned. When, for example, an exported function must always be applied to integer values, the generated contract inserts a check that ensures the "integerness" of the argument at every application site for this function; there is no need to insert such a check for the function's return points because the function is statically type checked. For a function that consumes an integer-valued function, the contract must ensure the function argument always returns an integer. In general, a contract wraps exported values with a proxy31 that controls access to the value. The idea is due to Matthews and Findler,29 while Tobin-Hochstadt's and Felleisen's Blame Theorem35 showed that if something goes wrong with such a mixed system, the runtime exception points to two faulty components and their boundary as the source of the problem.10 In general, Racket supplies a range of protection mechanisms, and a language creator can use them to implement a range of soundness guarantees for cooperating eDSLs.

Back to Top

Universality vs. Expressiveness

Just because a general-purpose language can compute all partial-recursive functions, programmers cannot necessarily express all their ideas about programs in this language.13 This point is best illustrated through an example. So, imagine the challenge of building an IDE for a new programming language in the very same language. Like any modern IDE, it is supposed to enable users to compile and run their code. If the code goes into an infinite loop, the user must be able to terminate it with a simple mouse click. To implement this capability in a naturale manner, the language must internalize the idea of a controllable process, a thread. If it does not internalize such a notion, the implementer of the IDE must step outside the language and somehow re-use processes from the underlying operating system.

For a programming language researcher, "stepping outside the language" signals failure. Or, as Ingalls21 said, "[an] operating system is a collection of things that don't fit into a language[; t]here shouldn't be one." We, Racket creators, have sought to identify services Racket borrows from the surrounding operating system and assimilate them into the language itself.19 Here are three sample constructs for which programmers used to step outside of Racket but no longer need to:

Sandboxes. That restrict access to resources;

Inspectors. That control reflective capabilities; and

Custodians. That manage resources (such as threads and sockets).

To understand how inclusion of such services helps language designers, consider a 2014 example, the shill language.32 Roughly speaking, shill is a secure scripting language in Racket's ecosystem. With shill, a developer articulates fine-grain security and resource policies—along with, say, what files a function may access or what binaries the script may run—and the language ensures these constraints are satisfied. To make this concrete, consider a homework server to which students can submit their programs. The instructor might wish to run an autograde process for all submissions. Using a shill script, the homework server can execute student programs that cannot successfully attack the server, poke around in the file system for solutions, or access external connections to steal other students' solutions. Naturally, shill's implementation makes extensive use of Racket's means of running code in sandboxes and harvesting resources via custodians.

Back to Top

State of Affairs

The preceding sections explained how Racket enables programmers to do the following:

Create languages. Create by way of linguistic reuse for specific tasks and aspects of a problem;

Equip with soundness. Equip a language with almost any conventional level of soundness, as found in ordinary language implementations; and

Exploit services. Exploit a variety of internalized operating system services for constructing runtime libraries for these embedded languages.

What makes such language-oriented programming work is "incrementality," or the ability to develop languages in small pieces, step by step. If conventional syntax is not a concern, developers can create new languages from old ones, one construct at a time. Likewise, they do not have to deliver a sound and secure product all at once; they can thus create a new language as a wrapper around, say, an existing C-level library, gradually tease out more of the language from the interface, and make the language as sound or secure as time permits or a growing user base demands.


Racket borrows from the surrounding operating system and assimilates such extra-linguistic mechanisms into the language itself.


Moreover, the entire process takes place within the Racket ecosystem. A developer creates a language as a Racket module and installs it by "importing" it into another module. This tight coupling has two implications: the development tools of the ecosystem can be used for creating language modules and their clients; and the language becomes available for creating more languages. Large projects often employ a tower involving a few dozen languages, all helping manage the daunting complexity in modern software systems.

Sony's Naughty Dog game studio has created just such a large project, actually a framework for creating projects. Roughly speaking, Sony's Racket-based architecture provides languages for describing scenes, transitions between scenes, scores for scenes, and more. Domain specialists use the languages to describe aspects of the game. The Racket implementation composes these domain-specific programs, then compiles them into dynamically linked libraries for a C-based game engine; Figure 8 sketches the arrangement graphically.

f8.jpg
Figure 8. A sketch of an industrial example of language-oriented programming.

Racket's approach to language-oriented programming is by no means perfect. To start with, recognizing when a library should become a language requires a discriminating judgment call. The next steps require good choices in terms of linguistic constructs, syntax, and runtime primitives.

As for concrete syntax, Racket currently has strong support for typical, incremental Lisp-style syntax development, including traditional support for conventional syntax, or generating lexers and parsers. While traditional parsing introduces the natural separation between surface syntax and meaning mentioned earlier, it also means the development process is no longer incremental. The proper solution would be to inject Racket ideas into a context where conventional syntax is the default.f

As for static checking, Racket forces language designers to develop such checkers wholesale, not incrementally. The type checker for typed/racket looks like, for example, the type checker for any conventionally typed language; it is a complete recursive-descent algorithm that traverses the module's representation and algebraically checks types. What Racket developers really want is a way to attach type-checking rules to linguistic constructs, so such algorithms can be synthesized as needed.

Chang et al.6 probably took a first step toward a solution for this problem and have thus far demonstrated how their approach can equip a DSL with any structural type system in an incremental and modular manner. A fully general solution must also cope with substructural type systems (such as the Rust programming language) and static program analyses (such as those found in most compilers).

As for dynamic checking, Racket suffers from two notable limitations: On one hand, it provides the building blocks for making language cooperation sound, but developers must create the necessary soundness harnesses on an ad hoc basis. To facilitate the composition of components in different languages, Racket developers need both a theoretical framework and abstractions for the partial automation of this task. On the other hand, the available spectrum of soundness mechanisms lacks power at both ends, and how to integrate these powers seamlessly is unclear. To achieve full control over its context, Racket probably needs access to assembly languages on all possible platforms, from hardware to browsers. To realize the full power of types, typed/racket will have to be equipped with dependent types. For example, when a Racket program uses vectors, its corresponding typed variant type-checks what goes into them and what comes out, but like ML or Haskell, indexing is left to a (contractual) check in the runtime system. Tobin-Hochstadt and his Typed Racket group are working on first steps in this direction, focusing on numeric constraints,23 similar to Xi's and Pfenning's research.37


To achieve full control over its context, Racket probably needs access to assembly languages on all possible platforms, from hardware to browsers.


As for security, the Racket project is still looking for a significant break-through. While the shill team was able to construct the language inside the Racket ecosystem, its work exposed serious gaps between Racket's principle of language-oriented programming and its approach to enforcing security policies. It thus had to alter many of Racket's security mechanisms and invent new ones. Racket must clearly make this step much easier, meaning more research is needed to turn security into an integral part of language creation.

Finally, LOP also poses brand-new challenges for tool builders. An IDE typically provides tools for a single programming language or a family of related languages, including debuggers, tracers, and profilers. Good tools communicate with developers in terms of the source language. Due to its very nature, LOP calls for customization of such tools to many languages, along with their abstractions and invariants. We have partially succeeded in building a tool for debugging programs in the syntax language,8 have the foundations of a debugging framework,28 and started to explore how to infer scoping rules and high-level semantics for newly introduced, language-level abstractions.33,34 Customizing these tools automatically to newly created (combinations of) languages remains an open challenge.

Back to Top

Conclusion

Programming language research is short of its ultimate goal—provide software developers tools for formulating solutions in the languages of problem domains. Racket is one attempt to continue the search for proper linguistic abstractions. While it has achieved remarkable success in this direction, it also shows that programming-language research has many problems to address before the vision of language-oriented programming becomes reality.

Back to Top

Acknowledgments

We thank Claire Alvis, Robert Cart-wright, Ryan Culpepper, John Clements, Stephen Chang, Richard Cob-be, Greg Cooper, Christos Dimoulas, Bruce Duba, Carl Eastlund, Burke Fetscher, Cormac Flanagan, Kathi Fisler, Dan Friedman, Tony Garnock Jones, Paul Graunke, Dan Grossman, Kathy Gray, Casey Klein, Eugene Kohlbecker, Guillaume Marceau, Jacob Matthews, Scott Owens, Greg Pettyjohn, Jon Rafkind, Vincent St-Amour, Paul Steckler, Stevie Strickland, James Swaine, Asumu Takikawa, Kevin Tew, Neil Toronto, and Adam Wick for their contributions.

A preliminary version of this article appeared in the Proceedings of the First Summit on Advances in Programming Languages conference in 2015.14 In addition to its reviewers, Sam Caldwell, Eduardo Cavazos, John Clements, Byron Davies, Ben Greenman, Greg Hendershott, Manos Renieris, Marc Smith, Vincent St-Amour, and Asumu Takikawa suggested improvements to the presentation of this material. The anonymous Communications reviewers challenged several aspects of our original submission and thus forced us to greatly improve the exposition.

Since the mid-1990s, this work has been generously supported by our host institutions—Rice University, University of Utah, Brown University, University of Chicago, Northeastern University, Northwestern University, Brigham Young University, University of Massachusetts Lowell, and Indiana University—as well as a number of funding agencies, foundations, and companies, including the Air Force Office of Scientific Research, Cisco Systems Inc., the Center for Occupational Research and Development, the Defense Advanced Research Projects Agency, the U.S. Department of Education's Fund for the Improvement of Postsecondary Education, the ExxonMobil Foundation, Microsoft, the Mozilla Foundation, the National Science Foundation, and the Texas Advanced Technology Program.

uf1.jpg
Figure. Watch the authors discuss their work in this exclusive Communications video. https://cacm.acm.org/videos/a-programmable-programming-language

Back to Top

References

1. Amin, N. and Tate, R. Java and Scala's type systems are unsound: The existential crisis of null pointers. In Proceedings of ACM SIGPLAN conference on Object-Oriented Programming Systems, Languages & Applications, 2016, 838–848.

2. Andersen, L., Chang, S., and Felleisen, M. Super 8 languages for making movies. In Proceedings of the ACM SIGPLAN International Conference on Functional Programming, 2017, 1–29.

3. Armstrong, J. Concurrency-oriented programming. In Frühjahrsfachgespräch der German Unix User Group, 2003; http://guug.de/veranstaltungen/ffg2003/papers/

4. Barzilay, E. and Clements, J. Laziness without all the hard work. In Proceedings of the Workshop on Functional and Declarative Programming in Education, 2005, 9–13.

5. Barzilay, E. and Orlovsky, D. Foreign interface for PLT Scheme. In Proceedings of the Ninth ACM SIGPLAN Workshop on Scheme and Functional Programming, 2004, 63–74.

6. Chang, S., Knauth, A., and Greenman, B. Type systems as macros. In Proceedings of the 44th ACM SIGPLAN Principles of Programming Languages, 2017, 694–705.

7. Culpepper, R. Fortifying macros. Journal of Functional Programming 22, 4–5 (Aug. 2012), 439–476.

8. Culpepper, R. and Felleisen, M. Debugging macros. Science of Computer Programming 75, 7 (July 2010), 496–515.

9. Dimoulas, C., New, M., Findler, R., and Felleisen, M. Oh Lord, please don't let contracts be misunderstood. In Proceedings of the ACM SIGPLAN International Conference on Functional Programming, 2016, 117–131.

10. Dimoulas, C., Tobin-Hochstadt, S., and Felleisen, M. Complete monitors for behavioral contracts. In Proceedings of the European Symposium on Programming, 2012, 214–233.

11. Dybvig, R., Hieb, R., and Bruggeman, C. Syntactic abstraction in Scheme. Lisp and Symbolic Computation 5, 4 (Dec. 1993), 295–326.

12. Erdweg, S., van der Storm, T., Vlter, M., Tratt, L., Bosman, R., Cook, W.R., Gerritsen, A., Hulshout, A., Kelly, S., Loh, A., Konat, G., Molina, P.J., Palatnik, M., Pohjonen, R., Schindler, E., Schindler, K., Solmi, R., Vergu, V., Visser, E., van der Vlist, K., Wachsmuth, G., and van derWoning, J. Evaluating and comparing language workbenches: Existing results and benchmarks for the future. Computer Languages, Systems and Structures 44, Part A (Dec. 2015), 24–47.

13. Felleisen, M. On the expressive power of programming languages. Science of Computer Programming 17, 1–3 (Dec. 1991), 35–75.

14. Felleisen, M., Findler, R.B., Flatt, M., Krishnamurthi, S., Barzilay, E., McCarthy, J., and Tobin-Hochstadt, S. The Racket Manifesto. In Proceedings of the First Summit on Advances in Programming Languages, T. Ball, R. Bodik, S. Krishnamurthi, B.S. Lerner, and G. Morrisett, Eds. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 2015, 113–128.

15. Findler, R., Clements, J., Flanagan, C., Flatt, M., Krishnamurthi, S., Steckler, P., and Felleisen, M. DrScheme: A programming environment for Scheme. Journal of Functional Programming 12, 2 (Mar. 2002), 159–182.

16. Findler, R.B. and Felleisen, M. Contracts for higher-order functions. In Proceedings of the Seventh ACM SIGPLAN International Conference on Functional Programming, 2002, 48–59.

17. Flatt, M. Composable and compilable macros: You want it when? In Proceedings of the Seventh ACM SIGPLAN International Conference on Functional Programming, 2002, 72–83.

18. Flatt, M. Bindings as sets of scopes. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 2016, 705–717.

19. Flatt, M., Findler, R.B., Krishnamurthi, S., and Felleisen, M. Programming languages as operating systems (or revenge of the son of the Lisp machine). In Proceedings of the International Conference on Functional Programming, 1999, 138–147.

20. Fowler, M. and Parsons, R. Domain-Specific Languages. Addison-Wesley, Boston, MA, 2010.

21. Ingalls, D.H. Design principles behind Smalltalk. Byte Magazine 6, 8 (Aug. 1981), 286–298.

22. Kats, L.C.L. and Visser, E. The Spoofax language workbench. In Proceedings of the Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages & Applications, 2010, 444–463.

23. Kent, A.M., Kempe, D., and Tobin-Hochstadt, S. Occurrence typing modulo theories. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, 2016, 296–309.

24. Kohlbecker, E.E., Friedman, D.P., Felleisen, M., and Duba, B.F. Hygienic macro expansion. In Proceedings of the ACM Conference on Lisp and Functional Programming, 1986, 151–161.

25. Kohlbecker, E.E. and Wand, M. Macros-by-example: Deriving syntactic transformations from their specifications. In Proceedings of the 14th Annual ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, 1987, 77–84.

26. Krishnamurthi, S. Linguistic Reuse. Ph.D. Thesis, Rice University, Houston, TX, 2001; https://www2.ccs.neu.edu/racket/pubs/#thesis-shriram

27. Krishnamurthi, S., Hopkins, P.W., McCarthy, J., Graunke, P.T., Pettyjohn, G., and Felleisen, M. Implementation and use of the PLT Scheme Web server. Higher-Order and Symbolic Computation 20, 4 (Apr. 2007), 431–460.

28. Marceau, G., Cooper, G.H., Spiro, J.P., Krishnamurthi, S., and Reiss, S.P. The design and implementation of a dataflow language for scriptable debugging. In Proceedings of the Annual ACM SIGCSE Technical Symposium on Computer Science Education, 2007, 59–86.

29. Matthews, J. and Findler, R.B. Operational semantics for Multilanguage programs. ACM Transactions on Programming Languages and Systems 31, 3 (Apr. 2009), 1–44.

30. McCarthy, J. The two-state solution. In Proceedings of the Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages & Applications, 2010, 567–582.

31. Miller, M.S. Robust Composition: Towards a United Approach to Access Control and Concurrency Control. Ph.D. Thesis, Johns Hopkins University, Baltimore, MD, May 2006; http://www.erights.org/talks/thesis/

32. Moore, S., Dimoulas, C., King, D., and Chong, S. Shill: A secure shell scripting language. In Proceedings of the Conference on Operating Systems Design and Implementation, 2014, 183–199.

33. Pombrio, J. and Krishnamurthi, S. Resugaring: Lifting evaluation sequences through syntactic sugar. In Proceedings of the Conference on Programming Language Design and Implementation, 2014, 361–371.

34. Pombrio, J., Krishnamurthi, S., and Wand, M. Inferring scope through syntactic sugar. In Proceedings of the ACM SIGPLAN International Conference on Functional Programming, 2017, 1–28.

35. Tobin-Hochstadt, S. and Felleisen, M. Interlanguage migration: From scripts to programs. In Proceedings of the ACM SIGPLAN Dynamic Language Symposium, 2006, 964–974.

36. Tobin-Hochstadt, S. and Felleisen, M. The design and implementation of Typed Scheme. In Proceedings of the 35th Annual ACM SIGPLAN-SIGACT Conference on the Principles of Programming Languages, 2008, 395–406.

37. Xi, H. and Pfenning, F. Eliminating array bound checking through dependent types. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, 1998, 249–257.

Back to Top

Authors

Matthias Felleisen ([email protected]) is a Trustee Professor in the College of Computer Science at Northeastern University, Boston, MA, USA.

Robert Bruce Findler ([email protected]) is a professor of computer science at Northwestern University, Evanston, IL, USA.

Matthew Flatt ([email protected]) is a professor of computer science at the University of Utah, Salt Lake City, UT, USA.

Shriram Krishnamurthi ([email protected]) is a professor of computer science at Brown University, Providence, RI, USA.

Eli Barzilay ([email protected]) is a research scientist at Microsoft Research, Cambridge, MA, USA.

Jay McCarthy ([email protected]) is an associate professor of computer science at the University of Massachusetts, Lowell, MA, USA.

Sam Tobin-Hochstadt ([email protected]) is an assistant professor of computer science at Indiana University, Bloomington, IN, USA.

Back to Top

Footnotes

a. The numerous language-like libraries in scripting languages (such as JavaScript, Python, and Ruby), books (such as Fowler and Parson),20 and websites (such as Federico Tomassetti's, https://tomassetti.me/resources-create-programming-languages/) are evidence of the desire by programmers to use and develop eDSLs.

b. The video language, including an overview of the implementation, is available as a use-case artifact at https://www2.ccs.neu.edu/racket/pubs/#icfp17-acf

c. Like many programming-language researchers, we subscribe to a weak form of the Sapir-Whorf hypothesis; see http://docs.racket-lang.org/algol60/ and https://www.hashcollision.org/brainfudge/ showing how Racket copes with obscure syntax.

d. See http://docs.racket-lang.org/algol60/, as well as well as https://www.hashcollision.org/brainfudge/, which shows how Racket copes with obscure syntax.

e. An alternative is to rewrite the entire program before handing it to the given compiler, exactly what distinguishes "expressiveness" from "universality."

f. Language workbenches (such as Spoofax22) deal with conventional syntax for DSLs but do not support the incremental modification of existing languages. A 2015 report12 suggests, however, these tool chains are also converging toward the idea of language creation as language modification. We conjecture that, given sufficient time, development of Racket and language workbenches will converge on similar designs.


©2018 ACM  0001-0782/18/03

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from [email protected] or fax (212) 869-0481.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2018 ACM, Inc.


Comments


Eric Nieuwland

In a way this reminds me of the work done by Paul Klint c.s. at CWI in The Netherlands.
See http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.92.6665&rep=rep1&type=pdf

Quite a bit of the effort from over 25 years ago seems to have been repeated.


Displaying 1 comment

Ian Knot

www.fieggen.com - Comments

Ian Knot animation

I tie my shoelaces with my own "Ian Knot", the World's Fastest Shoelace Knot (yes – I am the inventor): Make a loop with both ends and simultaneously pull them through each other to form an almost instant knot. Don't confuse this with the very similar looking Two Loop Shoelace Knot – the technique is quite different and much faster. It's a truly revolutionary way to tie your shoelaces!

Please don't be put off by these detailed instructions – even the Standard Shoelace Knot looks tricky when illustrated! Give it a try – you'll find it's easier than it looks.

Step 1:

Ian Knot diagram 1 Ian Knot picture 1

Tie a Left-over-Right Starting Knot as shown, then hold the laces as pictured. The right (blue) lace is held between the right thumb and forefinger while the left (yellow) lace is held around the left thumb and forefinger, using the other fingers of the left hand to hold the lace taut.

Step 2:

Ian Knot diagram 2 Ian Knot picture 2

This move creates two loops, one with the loose end behind, the other with the loose end in front. Use the middle finger of the right hand to push the loose end of the right lace behind, while the left hand simply rotates forwards to swing its loop across to the right.

Step 3:

Ian Knot diagram 3 Ian Knot picture 3

This next move crosses the two loops over each other. Use the left thumb to push its loose end over to the right, while the right middle finger continues to push its loose end all the way between the left thumb and forefinger to end up inside the left loop.

Note that the diagram at left shows somewhat exaggerated crossed loops. They are really more "alongside" each other, which is difficult to illustrate.

Step 4:

Ian Knot diagram 4 Ian Knot picture 4

This tricky move requires each hand to use the two fingers inside its own loop to grab the loose end of the other hand's loop. Use the left thumb and forefinger to grab the loose right end, then the right thumb and middle finger can grab the loose left end.

Step 5:

Ian Knot diagram 5 Ian Knot picture 5

This move sees each hand releasing its own loop and pulling the loose end of the opposite loop through its own. Take care not to pull the ends all the way through, as this will form a "knot" instead of a "bow". In fact, this is a quick way to tie a starting knot (though the finger movements must be reversed left/right to balance the finished knot).

Step 6:

Ian Knot diagram 6 Ian Knot picture 6

This final step simply completes the knot by pulling the loops tight. With practice, I can now tie my laces in about one third of the time of a conventional knot!

NOTE: You do not have to follow my instructions to the letter. So long as you retain the "core" of the technique, you are welcome to use whatever finger movements are most comfortable for you.

Finished Knot

Finished Ian Knot picture

The finished "Ian Knot" is identical to either the Standard Shoelace Knot or the Two Loop Shoelace Knot. Because it was tied much more quickly and symmetrically, the laces suffer less wear and tear and thus last longer.

NOTE: If your finished knot comes out crooked, it's probably because you tie your Starting Knot the opposite way to mine. This will result in an un-balanced "Granny Knot", which both sits crooked and comes undone more easily. See my Granny Knot page for more information.

Rate This Shoelace Knot

Please only vote once – multiple votes are removed daily

Ruby 2.6.0-preview1 Released

www.ruby-lang.org - Comments

We are pleased to announce the release of Ruby 2.6.0-preview1.

Ruby 2.6.0-preview1 is the first preview toward Ruby 2.6.0. This preview1 is released earlier than usual because it includes an important new feature, JIT.

JIT

Ruby 2.6 introduces an initial implementation of JIT (Just-in-time) compiler.

JIT compiler aims to improve performance of any Ruby program execution. Unlike ordinary JIT compilers for other languages, Ruby’s JIT compiler does JIT compilation in a unique way, which prints C code to a disk and spawns common C compiler process to generate native code. See also: MJIT organization by Vladimir Makarov.

How to use: Just specify --jit in command line or $RUBYOPT environment variable. Specifying --jit-verbose=1 allows to print basic information of ongoing JIT compilation. See ruby --help for other options.

The main purpose of this JIT release is to provide a chance to check if it works for your platform and to find out security risks before the 2.6 release. Currently JIT compiler is supported only when Ruby is built by gcc or clang and the compiler is available on runtime. Otherwise you can’t use it for now.

As of 2.6.0-preview1, we’re just preparing infrastructure for JIT and very few optimizations are implemented. You can measure some of potential improvements in micro benchmarks with this release, but it is NOT ready for benchmarking final performance of Ruby’s JIT compiler, especially for large programs like Rails applications.

We’re going to implement method inlining in JIT compiler, which is expected to increase Ruby’s performance significantly.

Also, we’re planning to increase the supported platforms, and the next plan is to support Visual Studio.

Stay tuned for the new age of Ruby’s performance.

New Features

  • Add Random.bytes. [Feature #4938]
  • Add Binding#source_location. [Feature #14230]

    This method returns the source location of binding, a 2-element array of __FILE__ and __LINE__. Traditionally, the same information could be retrieved by eval("[__FILE__, __LINE__]", binding), but we are planning to change this behavior so that Kernel#eval ignores binding’s source location [Bug #4352]. So, users should use this newly-introduced method instead of Kernel#eval.

  • Add :exception option to let Kernel.#system raise error instead of returning false. [Feature #14386]

Performance improvements

  • Speedup Proc#call because we don’t need to care about $SAFE any more. [Feature #14318]

    With lc_fizzbuzz benchmark which uses Proc#call so many times we can measure x1.4 improvements [Bug #10212].

  • Speedup block.call where block is passed block parameter. [Feature #14330]

    Ruby 2.5 improves block passing performance. [Feature #14045] Additionally, Ruby 2.6 improves the performance of passed block calling. With micro-benchmark we can observe 2.6x improvemnt.

Other notable changes since 2.5

  • $SAFE is a process global state and we can set 0 again. [Feature #14250]

  • Passing safe_level to ERB.new is deprecated. trim_mode and eoutvar arguments are changed to keyword arguments. [Feature #14256]

  • Merged RubyGems 2.7.6

See NEWS or commit logs for details.

With those changes, 1115 files changed, 23023 insertions(+), 14748 deletions(-) since Ruby 2.5.0!

Today, Feburary 24th, is Ruby’s 25th birthday. Happy birthday Ruby, and enjoy programming with Ruby 2.6.0-preview1!

Download

  • https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.0-preview1.tar.gz

    SIZE:   16082501 bytes
    SHA1:   94b4a2f5f992dc9855364284e9c64316bf129c90
    SHA256: 2023c42676d9237481e1a97157d5e2ecc10db5e320d5b9cf872ec1d293265d61
    SHA512: 004696c4f087333ba7cb2285418dcce70f399966ae8fed817aab9759fd2d75beb088c4aeb294fcd4260112e8422f490cd4dbdfce402d73f96bb679b8bb3e1607
    
  • https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.0-preview1.zip

    SIZE:   19807007 bytes
    SHA1:   24d76f67bf913348eca783a2ecf6f3faf37299ae
    SHA256: 6c883927e80430cf07f2d90728d6c2c71164223f378a48ebf964d3b66319f623
    SHA512: 1e7f318cec1b7875fd9891df671078de7585b556695a2a85708483ddcacfd6e0b63b70ec2535e92ff981b4f72063721ed552df49688e066666fcd7ae520ae667
    
  • https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.0-preview1.tar.bz2

    SIZE:   14104578 bytes
    SHA1:   9f0fb79643a4673a839b0c8496eccc6e1dbd022d
    SHA256: 8bd6c373df6ee009441270a8b4f86413d101b8f88e8051c55ef62abffadce462
    SHA512: d9cb270529a97670d54f43a0236fab072714e715c39277dab70b7a1843ec818e6700e47e1384c7256f9e0ae41ab2c0b768a0de38a5ecf4f4fff5da6ef5ad4944
    
  • https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.0-preview1.tar.xz

    SIZE:   11423984 bytes
    SHA1:   bbbc89d760cdaadbca3cbff587295864edeff0af
    SHA256: 1d99139116e4e245ce543edb137b2a8873c26e9f0bde88d8cee6789617cc8d0e
    SHA512: d12ff29778d8d940344619881b4d8247c2fb6b44ac2b2dbddb7078828e893cfac9a5a95b5588f0afdbed52bdb6dea95cff1b9ce3ad47dfa62209e97dab8810b6
    

Trump administration cracks down on H-1B visa abuse

money.cnn.com - Comments

H-1B visas by the numbers

The Trump administration is cracking down on companies that get visas for foreign workers and farm them out to employers.

Some staffing agencies seek hard-to-get H-1B visas for high-skilled workers, only to contract them out to other companies. There's nothing inherently illegal about contracting out visa recipients, but the workers are supposed to maintain a relationship with their employers, among other requirements.

In some cases, outsourcing firms flood the system with applicants. The U.S. Citizenship and Immigration Services agency said in a new policy memo released Thursday it will require more information about H-1B workers' employment to ensure the workers are doing what they were hired for.

Companies will have to provide specific work assignments, including dates and locations, to verify the "employer-employee" relationship between the company applying for an H-1B and its visa recipient.

H-1B visas are valid for three years and can be renewed for another three years. It is a visa that is near and dear to the tech community, with many engineers vying for one of the program's 85,000 visas each year. (20,000 of that quota are reserved for advanced degree holders.) Demand for the visa often exceeds the supply -- in that case, a lottery system is activated.

"Since there is a limited number of H-1B visas it is important that those visa workers go where they are legitimately needed," attorney Sara Blackwell told CNN. Blackwell advocates for American workers replaced by foreigner visa holders.

The government's crackdown is in line with Trump's direction to federal agencies to implement a "Buy American, Hire American" strategy. The administration proposed new rules and guidance for preventing fraud and abuse of work visas -- particularly the H-1B program.

The USCIS says it may limit the length of the visa to shorter than three years based the information an employer provides. For example, if an employer can't prove the H-1B holder is "more likely than not" needed for the full three years, the government might issue the visa for fewer than three years.

The memo also says the administration wants to prevent employee "benching." That's when firms bring on H-1B visa holders but don't give them work and don't pay them the required wages while they wait for jobs. Most projects don't need foreign workers for the full term, according to Monty Hamilton, CEO of IT contractor Rural Sourcing.

Although most agree that some employers abuse H-1B visas, how pervasive the abuse -- and how to prevent it -- remains a sensitive and divisive issue.

Robert Cormier, a retired criminal investigator for federal law enforcement, says that asking for more information from third party companies could help the government crack down on bad actors. The threat of prosecution if companies are caught lying could be enough to deter fraud.

"That could change the game completely," he told CNN.

Others say it could have an unintended consequence: Hurting those who are using the H-1B properly, according to Betsy Lawrence, the American Immigration Lawyers Association's director of government relations.

Immigration attorney Tahmina Watson said the government has already been much stricter about H-1B enforcement since Trump took office.

"Much of what is said in the memo has been carried out in the last 12 months leading to record number of denials," Watson said.

Indian outsourcing firms will be the hardest hit. Indian workers receive more than 70% of all H-1B visas.

Companies and immigration lawyers are preparing for the new H-1B lottery season. Applications must be filed on April 1st.

CNNMoney (New York) First published February 23, 2018: 1:06 PM ET

Keep your Identity Small (2009)

www.paulgraham.com - Comments

Keep Your Identity Small

February 2009

I finally realized today why politics and religion yield such uniquely useless discussions.

As a rule, any mention of religion on an online forum degenerates into a religious argument. Why? Why does this happen with religion and not with Javascript or baking or other topics people talk about on forums?

What's different about religion is that people don't feel they need to have any particular expertise to have opinions about it. All they need is strongly held beliefs, and anyone can have those. No thread about Javascript will grow as fast as one about religion, because people feel they have to be over some threshold of expertise to post comments about that. But on religion everyone's an expert.

Then it struck me: this is the problem with politics too. Politics, like religion, is a topic where there's no threshold of expertise for expressing an opinion. All you need is strong convictions.

Do religion and politics have something in common that explains this similarity? One possible explanation is that they deal with questions that have no definite answers, so there's no back pressure on people's opinions. Since no one can be proven wrong, every opinion is equally valid, and sensing this, everyone lets fly with theirs.

But this isn't true. There are certainly some political questions that have definite answers, like how much a new government policy will cost. But the more precise political questions suffer the same fate as the vaguer ones.

I think what religion and politics have in common is that they become part of people's identity, and people can never have a fruitful argument about something that's part of their identity. By definition they're partisan.

Which topics engage people's identity depends on the people, not the topic. For example, a discussion about a battle that included citizens of one or more of the countries involved would probably degenerate into a political argument. But a discussion today about a battle that took place in the Bronze Age probably wouldn't. No one would know what side to be on. So it's not politics that's the source of the trouble, but identity. When people say a discussion has degenerated into a religious war, what they really mean is that it has started to be driven mostly by people's identities. [1]

Because the point at which this happens depends on the people rather than the topic, it's a mistake to conclude that because a question tends to provoke religious wars, it must have no answer. For example, the question of the relative merits of programming languages often degenerates into a religious war, because so many programmers identify as X programmers or Y programmers. This sometimes leads people to conclude the question must be unanswerable—that all languages are equally good. Obviously that's false: anything else people make can be well or badly designed; why should this be uniquely impossible for programming languages? And indeed, you can have a fruitful discussion about the relative merits of programming languages, so long as you exclude people who respond from identity.

More generally, you can have a fruitful discussion about a topic only if it doesn't engage the identities of any of the participants. What makes politics and religion such minefields is that they engage so many people's identities. But you could in principle have a useful conversation about them with some people. And there are other topics that might seem harmless, like the relative merits of Ford and Chevy pickup trucks, that you couldn't safely talk about with others.

The most intriguing thing about this theory, if it's right, is that it explains not merely which kinds of discussions to avoid, but how to have better ideas. If people can't think clearly about anything that has become part of their identity, then all other things being equal, the best plan is to let as few things into your identity as possible. [2]

Most people reading this will already be fairly tolerant. But there is a step beyond thinking of yourself as x but tolerating y: not even to consider yourself an x. The more labels you have for yourself, the dumber they make you.

Notes

[1] When that happens, it tends to happen fast, like a core going critical. The threshold for participating goes down to zero, which brings in more people. And they tend to say incendiary things, which draw more and angrier counterarguments.

[2] There may be some things it's a net win to include in your identity. For example, being a scientist. But arguably that is more of a placeholder than an actual label—like putting NMI on a form that asks for your middle initial—because it doesn't commit you to believing anything in particular. A scientist isn't committed to believing in natural selection in the same way a bibilical literalist is committed to rejecting it. All he's committed to is following the evidence wherever it leads.

Considering yourself a scientist is equivalent to putting a sign in a cupboard saying "this cupboard must be kept empty." Yes, strictly speaking, you're putting something in the cupboard, but not in the ordinary sense.

Thanks to Sam Altman, Trevor Blackwell, Paul Buchheit, and Robert Morris for reading drafts of this.

The quantum internet has arrived (and it hasn’t)

www.nature.com - Comments

Before she became a theoretical physicist, Stephanie Wehner was a hacker. Like most people in that arena, she taught herself from an early age. At 15, she spent her savings on her first dial-up modem, to use at her parents’ home in Würzburg, Germany. And by 20, she had gained enough street cred to land a job in Amsterdam, at a Dutch Internet provider started by fellow hackers.

A few years later, while working as a network-security specialist, Wehner went to university. There, she learnt that quantum mechanics offers something that today’s networks are sorely lacking — the potential for unhackable communications. Now she is turning her old obsession towards a new aspiration. She wants to reinvent the Internet.

The ability of quantum particles to live in undefined states — like Schrödinger’s proverbial cat, both alive and dead — has been used for years to enhance data encryption. But Wehner, now at Delft University of Technology in the Netherlands, and other researchers argue that they could use quantum mechanics to do much more, by harnessing nature’s uncanny ability to link, or entangle, distant objects, and teleporting information between them. At first, it all sounded very theoretical, Wehner says. Now, “one has the hope of realizing it”.

Proponents say that such a quantum internet could open up a whole universe of applications that are not possible with classical communications, including connecting quantum computers together; building ultra-sharp telescopes using widely separated observatories; and even establishing new ways of detecting gravitational waves. Some see it as one day displacing the Internet in its current form. “I’m personally of the opinion that in the future, most — if not all — communications will be quantum,” says physicist Anton Zeilinger at the University of Vienna, who led one of the first experiments on quantum teleportation1, in 1997.

A team at Delft has already started to build the first genuine quantum network, which will link four cities in the Netherlands. The project, set to be finished in 2020, could be the quantum version of ARPANET, a communications network developed by the US military in the late 1960s that paved the way for today’s Internet.

Wehner, who is involved in the effort, is also coordinating a larger European project, called the Quantum Internet Alliance, which aims to expand the Dutch experiment to a continental scale. As part of that process, she and others are trying to bring computer scientists, engineers and network-security experts together to help design the future quantum internet.

Many technical details still need to be sorted out, and some researchers caution that it is too early to say exactly how much a quantum internet might deliver. But by thinking about security early, Wehner says that she hopes to avoid the vulnerabilities that the Internet inherited from ARPANET. “Maybe we have a chance to do it all right from the start.”

Quantum keys

The first proposals for quantum modes of communication date back to around the 1970s. Stephen Wiesner, then a young physicist at Columbia University in New York City, saw potential in one of the most basic principles of quantum mechanics: that it is impossible to measure a property of a system without changing it.

Wiesner suggested that information could be encoded in the states of objects such as isolated atoms, whose ‘spins’ can point up or down — like the 0 and 1 of classical bits — but can also be in both states simultaneously. Such units of quantum information are now commonly called quantum bits, or qubits. Wiesner pointed out that because the properties of a qubit can’t be measured without changing its state, it is also impossible to make exact copies or ‘clones’ of one2. Otherwise, someone could extract information about the state of the original qubit without affecting it, simply by measuring its clone. This prohibition later became known as quantum no-cloning, and it turns out to be a boon for security, because a hacker cannot extract quantum information without leaving a trace.

Inspired by Wiesner, in 1984, Charles Bennett, a computer scientist at IBM in Yorktown Heights, New York, and his collaborator Gilles Brassard, at the University of Montreal in Canada, came up with an ingenious scheme by which two users could generate an unbreakable encryption key that only they know3. The scheme depends on the fact that light can be polarized, so that the electromagnetic waves oscillate either in a horizontal or a vertical plane. One user converts a random sequence of 1s and 0s into a quantum key encoded in those two polarization states and sends it streaming to another person. In a sequence of steps, the recipient measures the key and establishes that the transmission was not disturbed by the measurements of an eavesdropper. Confident in the security of the key, the two parties can then scramble any message made up of classical bits — an image, for example — and send it just as they would any other encrypted message over the conventional Internet, or any other channel.

In 1989, Bennett led the team that first demonstrated this ‘quantum key distribution’ (QKD) experimentally4. Today, QKD devices that use similar schemes are commercially available and typically sold to financial or government organizations. ID Quantique, for example, a company founded in 2001 in Geneva, Switzerland, built a quantum link that has been protecting the results of Swiss elections for more than ten years.

Last year, China’s Micius satellite, the brainchild of physicist Pan Jianwei of the University of Science and Technology of China in Hefei, made some of the flashiest demonstrations of the approach. Using a variant of Bennett and Brassard’s protocol, the spacecraft created two keys, then sent one to a ground station in Beijing and another to Vienna as it passed overhead. An on-board computer then combined the two secret keys to create a new one, which it beamed down classically. Armed with their private keys, the Vienna and Beijing teams could unscramble that combined key by essentially subtracting their own, and so learn the other’s secret key. With both keys, one team could decrypt a transmission that the other team encrypted with its key. Last September, Pan and Zeilinger used this approach to set up the first intercontinental video chat to be secured in part with a quantum key5.

Satellites such as Micius could help to address one of the main challenges in making today’s quantum communications secure: distance. The photons needed to create an encryption key can get absorbed by the atmosphere or — in the case of ground networks — by an optical fibre, which renders quantum transmission impractical after several tens of kilometres.

A prototype of a "Nitrogen vacancy center" to test protocols for quantum internet.

Part of an experiment to investigate diamond-based systems as quantum-internet nodes at Delft University of Technology in the Netherlands.Credit: Marcel Wogram for Nature

Because quantum states cannot be copied, it is not an option to send multiple copies of a qubit in the hope that at least one will arrive. So, at the moment, creating long-distance QKD links requires building ‘trusted nodes’ to act as intermediaries. If a person were to hack into a trusted node, which handles keys in both their quantum and classical forms, they would be able to copy the keys without being detected — and so, of course, could the government or company operating the node. This is true both for trusted nodes on the ground and for Micius. “The satellite knows everything,” Pan says. But passing satellites could cut down on the number of trusted nodes that are needed to connect distant points.

Pan says that trusted nodes are already a step forward for some applications, because they reduce the number of spots where a network is vulnerable to attack. He has also led the creation of the extensive Beijing–Shanghai quantum-communication backbone. Launched in September, this connects 4 cities with 32 trusted nodes using more than 2,000 kilometres of optical fibre, and is being tested for banking and commercial communications, such as linking up the data centres of Internet-shopping giant Alibaba, Pan says.

Quantum connections

But networks that involve trusted nodes are only partly quantum. Quantum physics plays a part only in how the nodes create the encryption key; the subsequent encryption and transmission of information is entirely classical. A true quantum network would be able to harness entanglement and teleportation to transmit quantum information over long distances, without the need for vulnerable trusted nodes.

One of the main motivations for building such networks is to enable quantum computers to talk to each other, both between countries and across a single room. The number of qubits that can be packed into any one computing system may be limited, so networking the systems together could help physicists to scale them up. “At this point, it’s fair to say that probably you’ll be able to build a quantum computer with maybe a couple of hundred qubits,” says Mikhail Lukin, a physicist at Harvard University in Cambridge, Massachusetts. “But beyond that, the only way to do this is use this modular approach, involving quantum communications.”

On a larger scale, researchers envision a quantum-computing cloud, with a few highly sophisticated machines that are accessible through a quantum internet from most university labs. “The extra cool thing is that such cloud quantum computing is also secure,” says Ronald Hanson, an experimental physicist at Delft. “People at the server are unable to know what kind of program you’re running and the data you have.”

Researchers have come up with a plethora of other proposals for Internet applications — such as auctions, elections, contract negotiations and speed trading — that could exploit quantum phenomena to be faster or more secure than their classical counterparts.

But the biggest impact of a quantum internet could be on science itself. Synchronizing clocks using entanglement could improve the precision of Global Positioning System-like navigation networks from metres to millimetres, some researchers say. And Lukin and others have proposed using entanglement to combine distant atomic clocks into a single clock with vastly improved precision, which he says could lead to new ways of detecting gravitational waves, for example. In astronomy, quantum networks might link distant optical telescopes across thousands of kilometres, to effectively give them the resolution of one dish spanning the same distance. This process, called very long baseline interferometry, is applied routinely in radio astronomy, but operating in optical frequencies requires timing precision that is currently out of reach.

Spooky security

In the past decade or so, experiments pioneered by Christopher Monroe6, a physicist at the University of Maryland in College Park, and others have demonstrated some of the fundamentals needed to build a truly quantum network, such as teleporting information encoded in qubits from one place to another (see ‘Creating a quantum internet’).

Nik Spencer/Nature

To see how teleportation (also proposed by Bennett and Brassard7) works, imagine two users: Alice and Bob. Alice holds a qubit, which could be a trapped ion or some other quantum system, and wants to transfer the information stored in it to Bob. As luck would have it, Alice and Bob come into possession of two ‘proxy’ particles — also qubits — that are entangled with each other. If Alice can entangle her qubit and proxy particle, the qubit will, by extension, also become entangled with Bob’s particle. To do so, Alice performs a particular kind of joint measurement on her two particles. She then shares the results of that measurement (which are ordinary, classical data) with Bob. To complete the teleportation process, Bob then uses that information to manipulate his particle so that it ends up in the same state as Alice’s qubit originally was.

For practical purposes, it doesn’t matter how Alice and Bob obtain the entangled proxy particles. They could be individual atoms delivered in a briefcase, say, or photons beamed to the pair by a third party. (One of Micius’s experiments last year sent entangled pairs of photons to two ground stations in China over a record distance of more than 1,200 kilometres.) Alice and Bob could also entangle the qubits they hold, by sending photons out to interact at a third location.

The beauty of quantum teleportation is that the quantum information does not technically travel along the network. The photons that do travel are just used to establish a link between Alice and Bob so that quantum information can then be transferred. If one pair of entangled photons fails to establish a connection, another pair will. This means that the quantum information is not lost if photons are.

Link and repeat

A quantum internet would be able to produce entanglement on demand between any two users. Researchers think that this will involve sending photons through both fibre-optic networks and satellite links. But connecting distant users will require a technology that can extend the reach of entanglement — relaying it from user to user and along intermediate points.

One way in which such a quantum repeater could work was proposed in 2001 by Lukin and his collaborators8. In their scheme, small quantum computers that can store qubits and do simple operations on them are used to entangle a qubit at an upstream station with one downstream. Repeated application of this ‘entanglement swapping’ process along a path in a network would eventually produce entanglement between any two users.

In 2015, Hanson and his collaborators showed how to build one leg of a network when they linked two qubits built from single-atom impurities in diamond crystals and separated by 1.3 kilometres9. Photons emitted by the two qubits travelled towards an intermediate station, where they then interacted, establishing entanglement. “It shows that one can really establish entanglement — strong, reliable entanglement — between two distant quantum-information processors,” says Seth Lloyd, a physicist at the Massachusetts Institute of Technology in Cambridge.

Researchers are investigating other ways to construct and manipulate qubits, including using individual ions suspended in a vacuum — pioneered by Monroe and others — as well as systems that pair up atoms and photons bouncing between two mirrors inside a cavity.

Like Hanson’s diamond system, these qubits could be used to build both quantum repeaters and quantum computers. Fortunately for people hoping to ramp up quantum communications, the requirements for a repeater may be less demanding than those for a fully fledged quantum computer. Iordanis Kerenidis, a quantum-computation researcher at the University of Paris Diderot, made this argument at a workshop on quantum repeaters in Seefeld, Austria, last September. “If you tell experimentalists that you need 1,000 qubits, they are going to laugh,” he said. “If you tell them you need ten — well, they laugh less.”

The prospect of creating a quantum internet is now becoming a problem of systems engineering. “From an experimental point of view, people have demonstrated various building blocks” for quantum networks, says Tracy Northup, a physicist at Austria’s University of Innsbruck whose team works on cavity qubits and is part of Wehner’s pan-European Quantum Internet Alliance. “But putting them together in one place — we all see how challenging it is,” Northup says.

For the moment, Wehner’s alliance is still at an early stage and looking for public funding as well as corporate partners. In the meantime, the Dutch demonstration network — which Wehner co-leads with Hanson and Erwin van Zwet, a joint systems engineer at the Dutch research organization TNO — has been moving forward. Hanson and his colleagues have been improving the speed of their systems, which in the 2015 experiment entangled just 245 qubit pairs over the equivalent of about 9 days. Another crucial challenge has been to reliably convert photons from the visible wavelengths that come out of the diamond qubits to longer, infrared ones that can travel well along optical fibres; this is tricky because the new photon still has to carry the quantum information of the old one, but without the possibility of cloning it. Earlier this year, Hanson and his colleagues achieved this by making photons interact with a laser beam of longer wavelength10. That technique would enable qubits to be linked over distances of tens of kilometres over fibre.

Hanson’s team is now building a link between Delft and The Hague, a good 10 kilometres away. By 2020, the researchers hope to have connected up four Dutch cities, with a station at each site acting as a quantum repeater. If successful, the project would be the world’s first genuine quantum-teleportation network. The group aims to open it up to other teams interested in performing quantum-communications experiments remotely, much like IBM’s Quantum Experience, which allows remote users access to a rudimentary quantum computer.

The network could be a test bed for researchers hoping to fix some of the Internet’s flaws, not least the ease with which users can forge or steal identities. “The idea that you could join a network without establishing identity is a problem from early on,” said Robert Broberg, a network engineer from the telecommunications equipment giant CISCO, at the Seefeld meeting. Wehner and others have proposed quantum techniques that would allow users to prove their identity by certifying that they own the correct secret code (a series of classical bits) without ever transmitting it. Instead, the user and the server use the code to create a sequence of qubits and send them to a ‘black box’ in between. The black box — which could be, say, a cash machine — can then compare the two sequences to see whether they match, without ever knowing the underlying code.

But some researchers caution against overselling the potential reach of the technology. “Today’s Internet will never be entirely quantum, no more than computers will ever be all-quantum,” says Nicolas Gisin, a physicist at the University of Geneva in Switzerland and a co-founder of ID Quantique. And it could be that many of the things people hope to achieve with quantum networks could be done with more conventional technologies. “Sometimes, something looks like a great idea at first, and then it turns out to be easily achievable without a quantum effect,” says Norbert Lütkenhaus, a physicist at the University of Waterloo in Canada who is helping to develop standards for the future quantum internet.

Time will tell whether the promise of the quantum internet will materialize. As far as we know, teleportation is a phenomenon that, although physically possible, does not occur in nature, Zeilinger says. “So this is really new for humanity. It might take some time.”

Wehner’s familiarity with both physics and network security has made her a point of reference for people in the field. And after having done much work on hard-core quantum theory, she is relishing the opportunity to shape these future networks. “For me,” she says, “this is really full circle.”

Apple moves to store iCloud keys in China, raising human rights fears

www.reuters.com - Comments

SAN FRANCISCO/BEIJING (Reuters) - When Apple Inc begins hosting Chinese users’ iCloud accounts in a new Chinese data center at the end of this month to comply with new laws there, Chinese authorities will have far easier access to text messages, email and other data stored in the cloud.

That’s because of a change to how the company handles the cryptographic keys needed to unlock an iCloud account. Until now, such keys have always been stored in the United States, meaning that any government or law enforcement authority seeking access to a Chinese iCloud account needed to go through the U.S. legal system.

Now, according to Apple, for the first time the company will store the keys for Chinese iCloud accounts in China itself. That means Chinese authorities will no longer have to use the U.S. courts to seek information on iCloud users and can instead use their own legal system to ask Apple to hand over iCloud data for Chinese users, legal experts said.

Human rights activists say they fear the authorities could use that power to track down dissidents, citing cases from more than a decade ago in which Yahoo Inc handed over user data that led to arrests and prison sentences for two democracy advocates.  Jing Zhao, a human rights activist and Apple shareholder, said he could envisage worse human rights issues arising from Apple handing over iCloud data than occurred in the Yahoo case.

In a statement, Apple said it had to comply with recently introduced Chinese laws that require cloud services offered to Chinese citizens be operated by Chinese companies and that the data be stored in China. It said that while the company’s values don’t change in different parts of the world, it is subject to each country’s laws.

“While we advocated against iCloud being subject to these laws, we were ultimately unsuccessful,” it said. Apple said it decided it was better to offer iCloud under the new system because discontinuing it would lead to a bad user experience and actually lead to less data privacy and security for its Chinese customers.

As a result, Apple has established a data center for Chinese users in a joint venture with state-owned firm Guizhou - Cloud Big Data Industry Co Ltd. The firm was set up and funded by the provincial government in the relatively poor southwestern Chinese province of Guizhou in 2014. The Guizhou company has close ties to the Chinese government and the Chinese Communist Party.

The Apple decision highlights a difficult reality for many U.S. technology companies operating in China. If they don’t accept demands to partner with Chinese companies and store data in China then they risk losing access to the lucrative Chinese market, despite fears about trade secret theft and the rights of Chinese customers.

     BROAD POWERS

Apple says the joint venture does not mean that China has any kind of “backdoor” into user data and that Apple alone – not its Chinese partner – will control the encryption keys.  But Chinese customers will notice some differences from the start: their iCloud accounts will now be co-branded with the name of the local partner, a first for Apple.

And even though Chinese iPhones will retain the security features that can make it all but impossible for anyone, even Apple, to get access to the phone itself, that will not apply to the iCloud accounts. Any information in the iCloud account could be accessible to Chinese authorities who can present Apple with a legal order.

Apple said it will only respond to valid legal requests in China, but China’s domestic legal process is very different than that in the U.S., lacking anything quite like an American “warrant” reviewed by an independent court, Chinese legal experts said. Court approval isn’t required under Chinese law and police can issue and execute warrants.

“Even very early in a criminal investigation, police have broad powers to collect evidence,” said Jeremy Daum, an attorney and research fellow at Yale Law School’s Paul Tsai China Center in Beijing.  “(They are) authorized by internal police procedures rather than independent court review, and the public has an obligation to cooperate.”

    Guizhou - Cloud Big Data and China’s cyber and industry regulators did not immediately respond to requests for comment. The Guizhou provincial government said it had no specific comment.

There are few penalties for breaking what rules do exist around obtaining warrants in China. And while China does have data privacy laws, there are broad exceptions when authorities investigate criminal acts, which can include undermining communist values, “picking quarrels” online, or even using a virtual private network to browse the Internet privately.

Apple says the cryptographic keys stored in China will be specific to the data of Chinese customers, meaning Chinese authorities can’t ask Apple to use them to decrypt data in other countries like the United States.

Privacy lawyers say the changes represent a big downgrade in protections for Chinese customers.

    “The U.S. standard, when it’s a warrant and when it’s properly executed, is the most privacy-protecting standard,” said Camille Fischer of the Electronic Frontier Foundation.

WARNED CUSTOMERS

Apple has given its Chinese users notifications about the Feb. 28 switchover data to the Chinese data center in the form of emailed warnings and so-called push alerts, reminding users that they can chose to opt out of iCloud and store information solely on their device. The change only affects users who set China as their country on Apple devices and doesn’t affect users who select Hong Kong, Macau or Taiwan.

The default settings on the iPhone will automatically create an iCloud back-up when a phone is activated. Apple declined to comment on whether it would change its default settings to make iCloud an opt-in service, rather than opt-out, for Chinese users.

Apple said it will not switch customers’ accounts to the Chinese data center until they agree to new terms of service and that more than 99.9 percent of current users have already done so.

Until now, Apple appears to have handed over very little data about Chinese users. From mid-2013 to mid-2017, Apple said it did not give customer account content to Chinese authorities, despite having received 176 requests, according to transparency reports published by the company. By contrast, Apple has given the United States customer account content in response to 2,366 out of 8,475 government requests.

Those figures are from before the Chinese cyber security laws took effect and also don’t include special national security requests in which U.S. officials might have requested data about Chinese nationals. Apple, along with other companies, is prevented by law from disclosing the targets of those requests.

Apple said requests for data from the new Chinese datacentre will be reflected in its transparency reports and that it won’t respond to “bulk” data requests.

Human rights activists say they are also concerned about such a close relationship with a state-controlled entity like Guizhou-Cloud Big Data.

Sharon Hom, executive director of Human Rights in China, said the Chinese Communist Party could also pressure Apple through a committee of members it will have within the company. These committees have been pushing for more influence over decision making within foreign-invested companies in the past couple of years.

   

Reporting by Stephen NellisEditing by Jonathan Weber and Martin Howell

Learn Physics by Programming in Haskell

arxiv.org - Comments

Authors: Scott N. Walck (Lebanon Valley College, Annville, Pennsylvania, USA)

(Submitted on 16 Dec 2014)

Abstract: We describe a method for deepening a student's understanding of basic physics by asking the student to express physical ideas in a functional programming language. The method is implemented in a second-year course in computational physics at Lebanon Valley College. We argue that the structure of Newtonian mechanics is clarified by its expression in a language (Haskell) that supports higher-order functions, types, and type classes. In electromagnetic theory, the type signatures of functions that calculate electric and magnetic fields clearly express the functional dependency on the charge and current distributions that produce the fields. Many of the ideas in basic physics are well-captured by a type or a function.

Submission history

From: EPTCS [view email]
[v1] Tue, 16 Dec 2014 05:14:30 GMT (15kb)

On the Work of Edward Witten (1990) [pdf]

bohr.physics.berkeley.edu - Comments