LINQ Expressions and Reflection.Emit: an uncomfortable union

The other day I needed a component for a project of mine, which would rewrite methods into state machines like the CTP C# compiler does for async methods. The rewriter would accept a callback to identify the sites where a continuation must be created, and another one to emit glue code for the site using two primitives: SAVE_STATE() which returns the continuation delegate and RESTORE_STATE() which returns the value passed to the continuation. This approach permits the users of the rewriter to avoid the overhead of saving and restoring state when the continuation turns out to be unnecessary (in async terms, when the awaited thing is already complete). The new CTP compiler implements this optimization. My component would eventually have to work with IL methods either via Reflection.Emit or Cecil, but I thought it would be interesting to make it work with LINQ Expressions first. Besides, the DLR has a similar rewriter for yield which I could scavenge for useful hints. The DLR rewriter uses a nested lambda to create the 'environment', shifting the work to the LINQ Expression compiler (EC). Probably because of permission issues, EC does not create new closure classes and instead uses a thinly veiled object[] to store closure locals. I did not want this, I wanted to generate a proper closure class. After all, EC can compile a lambda expression into a MethodBuilder!

Although I more-or-less made it work, I must report that LINQ Expressions don't work all that well with Reflection.Emit:

1. A LambdaExpression cannot have parameters of unbaked type. EC appears to have no problems with this, but Expression.CreateLambda uses generic lambda factories and an unbaked type cannot serve as a type argument. One must have the lambda accept a suitable base class or object.

2. It is impossible to generate a call to any method which has not been 'baked' because LINQ Expression constructors insist on validating a method's parameters and call MethodBase.GetParameters(). This extends to object construction as Expression.New validates constructor parameters. These two problems are very obnoxious and make any serious use of LINQ Expressions with Reflection.Emit impossible.

   As a side note, I find it puzzling and inconvenient that MethodBuilder does not expose its parameter list.

3. EC has no problems emitting constants referring to a TypeBuilder or a MethodBuilder, but one must tell Expression.Constant the correct types, viz. Type and MethodInfo. Otherwise EC happily emits casts to TypeBuilder which blow up at runtime. Just a minor gotcha, but still.

4. There is no DelegateCreationExpression. This is strictly speaking not related to Reflection.Emit, but it is annoying to have to generate a call to CreateDelegate, complete with casts and stuff, instead of a ldftn-new combo.

5. EC cannot compile lambda expressions into member methods even if the method signature is compatible. The target method has to be static, period.

This was a useful excercise and working directly with IL should not be much more complicated.

Bakunin fulminates

"Etatism and anarchy" (1873)

Let us respect [social] scientists for their achievements, but for the sake of their reason and moral integrity we must not give them any privileges and only recognize their right, common to all, to freely preach their convictions, thoughts and knowledge. We should give power neither to them nor to anybody else, for whoever has power becomes, by an immutable law of sociology, an oppressor and exploiter of the society.

Woe unto mankind should theoretical speculation become the only source of guidance for society, should science alone take charge of all social administration. Life would wither, and human society would turn into a voiceless and servile herd. The domination of life by science can have no other result than the brutalization of mankind.

February link clearance

One-paragraph

 A white female LHC engineer bemoans the lack of black PhDs in physics in a rather over-the-top way — 'threatens the viability of scientific research'? Give me a break. But the statistic cannot be simply brushed aside. With all the Affirmative Action programs, powerful Equal Opportunities commissions etc. everyone who wants to go for a PhD in physics should have the opportunity, so what's up? Could it be that they just don't want it? Scientific research is boring much of the time, needs hard work and a particular mindset*, and it does not pay any too well in either money or status, so why should they? Especially given all the Xyz Studies departments. Check out White Studies, by the way.

 BBC article about robot carers in Japan contains very condescending remarks aimed at Japan's anti-immigration policy. It appears that many Anglophone Brahmins resent Japan for getting along tolerably well without mass immigration and want it to open up (this is noticeable even in Lonely Planet Japan), while condescension towards Japan is a regular feature of both articles and comments; sometimes these produce poisonous outbursts which then get moderated. Comment section to the above-mentioned article contains interesting discussion of job markets etc.

One-liners

 Average female faces by country/region. Source: The Postnational Monitor. Also great mod text.

 Mark Thoma's lectures on economics theory, econometrics etc.

 Problems with bank executives' incentive structures — the breakdown in corporate governance [...] is complete.

 E. M. Forster's 1909 (!) short story "The Machine Stops" — much of the internet bears an uncanny resemblance to the story's tele-lecturers, who spend lives 'growing spiritually' by exchanging tenth-hand regurgitated information.

 It is more efficient to read books pursuing some idea than expecting to get ideas, even if your original idea turns out to be worthless later it serves as a scaffold for acquired knowledge. Very good!

 A very large epidemiological study of a 1946 British cohort.

 The Enron Corpus — 600000 emails generated by 158 Enron employees, went public during the Enron trial. Used to train systems for e-discovery, communication analysis etc.

 Journal of Macroeconomics article on economic growth in the Gilded Age US got cowened.

 Predictions on how Germans can't fix Eurozone's financial troubles — very realistic thinking and understanding of how European financial sector really works.

 The Onion makes fun of Tyler Cowen's call for more status for scientists.

 Excellent article about MMORPG design — class system, levels, balancing etc.

 People who have trouble focusing attention may be more creative, especially those who have high IQ.

Book report: Ivan Ovsinsky, "A new system of agriculture"

Овсинский И.Е. "Новая система земледелия" (Киев, 1899)

Ivan Ovsinsky "A new system of agriculture"

“A careful consideration of the tilling and fertilization recipes leaves one amazed as to how illogical and expensive they are. Happily, a significant proportion of farmers is ignorant of Liebich's theory and continues farming the land in the same way their forefathers did. Otherwise only the lucky few able to afford hitching 3 pairs of oxen to a German plow and to sprinkle their fields with powders (fertilizer) would continue farming.

Ovsinsky was an early proponent of no-till, no-fertilizer agriculture. His fields in the south of Ukraine amazed visitors with rye 10 feet high and wheat green and fresh in the middle of droughts, to say nothing of his yields (stable 4-5t/ha, double the average of that time). In this short book, he presented his system for the general public. His main ideas are as follows.

1. Plants balance expenditures on seeds with expenditures on vegetative growth. To shift this balance towards seeds, the farmer has to apply moderate environmental pressure and force plants to struggle for existence. For grains, where pruning, pitching and suckering are not an option, the prescription is to sow more densely than usual, only avoiding clumping-up of seeds, and to put more space between rows. This makes plants compete and induces them to produce heavier seed to occupy this vacant space.

2. Soil and air together contain massively more nutrients than is removed with the harvest. Healthy soil makes this available to plants naturally. Farmers are only forced to provide plants with easily soluble nutrients by applying artificial fertilizer because deep tilling destroys soil health. Ovsinsky quotes Deherain as saying, at the end of a listing of "horrible" quantities of mineral nutrients present in the soil, "This brings us to the untenable conclusion that fertilizers are useless and not necessary."

Healthy soil has a porous top layer and a developed system of hollow channels (created both by the decay of old roots and by action of worms) though which air and water circulate. Plants can also reuse these channels to grow deeper roots than would otherwise be the case. Simultaneously the soil retains its capillary properties. The top layer protects the soil from drying out and heating up. The temperature differential between air and subsoil helps condense moisture and dew, which also contains in itself more nitrogen (as ammonia) than is removed with the harvest. Soil biochemistry uses abundant air and moisture to break down organic remains and oxidize ammonia to nitrate in the topsoil and mobilize phosphate and other minerals from rock particles in the subsoil.

Every part and factor in this complex arrangement works together and is essential for the whole. Deep tilling, originally conceived to bring more nutrients to the surface, upsets all this completely. In particular, irrigation becomes necessary because compacted soil cannot capture and retain moisture properly. Remedies and modifications applied afterwards may restore one or two factors, but the lack of balance means that the results are not robust and uninspiring. Deep tilling has the pernicious property, which seems common to primitive technological solutions to complex problems, that, once begun, more and more of it is required simply to keep up.

Accordingly Ovsinsky forswears deep tilling (i.e. deeper than 2 inches, the thickness of the top layer) although he does not eliminate tilling entirely, using it for weed control. He adjusts topsoil porosity in the spring or even in autumn to make the spring sun heat up soil faster. He knows about "green manure", but apparently does not apply it as systematically as Fukuoka did. Although he mentions that crops suppress the vegetation of weeds once they are mature enough, he does not use clover and mulch for weed control, relying instead on shallow tilling. Neither does he appear to think in terms of sustainable land use — small wonder given that he writes at the turn of the XX century.

Ovsinsky, even if his methods are not always consistent, represents a large step in the right direction. Modern farming is carried out by the 'lucky few', and this has been a great boon for all other sectors of the economy. Ovsinsky shows us that we don't have to lay waste to our soil for it.

Russian text is available online free of charge. 90pp.
HT: Novaya Gazeta

January link clearance

One-paragraph

 How can Ed Glaeser write about 'treating the seller of kidneys with respect' and 'capable of choosing for himself or herself even in difficult circumstances' 150 years after Marx explained all about economic coercion? Would he treat the choice of this girl (scroll to end) with respect, too? It is all very well to value any contract when the valuer has never had to 'enter into an inequitable arrangement out of fear of starvation, or economic ruin', or at least if the inequitableness always stayed small enough. Also check out Eric's comment: The point is that while economics may be in theory non-normative, it often doesn't stay that way in practice.
Two asides:

1. Relying on charity, including government charity, does not count as an out from economic coercion if the charity is the Dickensian kind, as is very often the case.
2. In contrast to XVII-XIXc. western Europe and Britain, economic coercion was much less acute in America because of easy availability of land to farm, prairie to ranch and forest to fell. This might have blinded American economists to economic coercion.

 Why astronomers should not get sucked too much into Dark Energy-related projects:

1. won't advance astrophysics on a broad front, instruments not likely to be useful for much else (experiment vs observatory);
2. large collaboration culture of fundamental physics experiments will scare away young talent looking to make an original contribution;
3. negative impact on astronomy's image as "ambassador of physics" because subject too abstract and removed from everyday experience.

Interesting statistics on bibliographical changes in astrophysics papres over 30 years: citations/article x4, authors/article x2, partly because of 'the use of citations as measure of performance'. Note: the dynamic is similar to the degradation of programmer performance measures, only the timescale is much longer.

One-liners

 How to set up authentication in HttpListener

 Exact requirements for an IAsyncResult implementation — could this be expressed in code?

 Что редактор советского "Огонька" Коротич писал про Америку до и после перестройки

 iPad imagined in 1988, but comes 10 years late (rus)

 Steve Jobs vs Bill Gates — hippie vs serious boy (rus)

 French gentleman's mansion, locked for 100 years, opens as a museum

 Swedish anti-immigration party politician tries hard to avoid saying that Muslim immigrants are more prone to commit crimes

 IQ negatively associated with criminality at individual level (review) and at county level. Association "not confounded by a measure of concentrated disadvantage that captures the effects of race, poverty, and other social disadvantages of the county."

 IQ positively associated with attractiveness (full text), attractiveness measure is binary but appears legit

 GxE in cognitive ability development (!)

 People believe they have more free will and are more in control of their actions than others

 Ruins of Detroit — check out the clueless comments

 Kan's speech at Davos — Japan's society is probably less atomized than any Western one, but listen to him talk about strengthening social bonds.

 This has to be part of the theory of the modern state: a government cutting off communications from fear of public protests produces a strong and public signal that makes protests more likely to occur and to succeed by overcoming protesters' communication problems for them at a stroke.

How I wasted New Year's Eve

Apparently I am blessed with the kind of brain susceptible to nerd sniping (by the way, the answer to that problem is $4/\pi-1/2$; see [Cserti], an elegant approach using discrete Green functions). On New Year's Eve, an ill-considered click brought this problem before my eyes:

Both ends of a thin flexible rod are joined to the same pivot. What is the angle between the rod's ends?

I had not used much of the mathematical technique learned in the university for ten years. I don't like writing. I have other, arguably more important things to do. Oh well.

This problem seems made for a demonstration of variational calculus. Accordingly, I need an expression for the potential energy of a bent rod to vary. The rod being thin, I can apply simple bending theory, which says that the flexural energy of a piece of rod is proportional to the square of its curvature. The shape the rod makes does not depend on the rod's material or length as long as the assumptions of simple bending theory hold. Also the shape will obviously be symmetrical about the line bisecting my angle of interest (I will make this line my $y$ axis). A final observation is that the rod's curvature at the pivot is zero, because the pivot rotates freely and nonzero curvature generates momentum. At this point physics ends and mathematics, as applied by physicists, begins.

Let $\theta$ be the angle between the tangent to the curve and the $x$ axis, and $s$ the natural parameter (i.e. arc length). Using the Frenet's formula $\dot{\bf{t}}=\kappa\bf{n}$ ($\bf{t}$ the tangent unit vector, $\bf{n}$ the normal unit vector, $\kappa$ the curvature), I obtain that $\kappa=\dot{\theta}$, so the main term to vary is $\dot\theta^2$. In addition, I have to constrain the curve's ends to meet; the curve being symmetrical about the $y$ axis, I worry only about the separation of the curve's ends along the $x$ axis: $\int\cos\theta ds=0$. Accordingly I shall vary $$\tag{*}{(*)}\int_{-1}^{1}\dot\phi^2+½\mu^2\cos2\phi ds$$ where I have introduced $\phi=\theta/2$, arbitrarily made the curve's length equal $2$ and where $\mu$ is the Lagrange multiplier. Equating the variation of (*) to zero yields the ODE for $\phi$: $$\dot\phi^2=\mu^2k^{-2}(1-k^2\sin^2\phi)$$ with $k$ a constant of integration. The curve's symmetry and my choice of axes give me the boundary condition $\phi(0)=0$, and I denote $\phi(-1)=\alpha$. The solution to this ODE is the elliptic integral of the first kind: $\mu s=k\,\mathrm{F}(\phi,k)$. Since the curvature at $s=\pm1$ is zero, $k=1/\sin\alpha$. Now I bring into play the end-meeting constraint: $$0=\int_{-1}^{1}\cos2\phi ds=(\mu/k)\int_{-\mu/k}^{\mu/k}2\mathrm{cn}^2u-1 du,$$ whence G&R 5.134.2 produces $$\mathrm{E}(\alpha,k)=(1-½k^2)\,\mathrm{F}(\alpha,k).$$ At this point I have enough equations to solve for $\alpha$ and need not bother about $\mu$ — not surprising considering that $\mu$ is really a dimensional quantity which arises because I fixed the length of the curve. The solution is implicitly determined by $$2\sin^2\alpha\,\mathrm{E}(\alpha,1/\sin\alpha)=(2\sin^2\alpha-1)\,\mathrm{F}(\alpha,1/\sin\alpha).$$ Mathematica fails to find $\alpha$ numerically from this equation, however, so I solved the two equations together to find $\alpha\approx65°21'$. $\alpha$ being one half of the tangent angle to one end of the rod, the angle between the rod's ends is $4\alpha-\pi$, approximately $81°25'$. ■